index

package
v3.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 17, 2024 License: AGPL-3.0 Imports: 19 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// MagicIndex 4 bytes at the head of an index file.
	MagicIndex = 0xBAAAD700
	// HeaderLen represents number of bytes reserved of index for header.
	HeaderLen = 5

	// FormatV1 represents 1 version of index.
	FormatV1 = 1
	// FormatV2 represents 2 version of index.
	FormatV2 = 2

	TenantLabel = "__loki_tenant__"
)

Variables

View Source 
var AllPostingsKey = labels.Label{}

Functions

func EncodePostingsRaw 

func EncodePostingsRaw(e *encoding.Encbuf, offs []uint32) error

EncodePostingsRaw uses the "basic" postings list encoding format with no compression: <BE uint32 len X><BE uint32 0><BE uint32 1>...<BE uint32 X-1>.

func ReadPostingsOffsetTable 

func ReadPostingsOffsetTable(bs ByteSlice, off uint64, f func(name, value []byte, postingsOffset uint64, labelOffset int) error) error

ReadPostingsOffsetTable reads the postings offset table and at the given position calls f for each found entry. The name and value parameters passed to f reuse the backing memory of the underlying byte slice, so they shouldn't be persisted without previously copying them. If f returns an error it stops decoding and returns the received error.

Types

type BufferWriter 

type BufferWriter struct {
	// contains filtered or unexported fields
}

func NewBufferWriter 

func NewBufferWriter() *BufferWriter

NewBufferWriter returns a new BufferWriter. todo: pooling memory

func (*BufferWriter) AddPadding 

func (fw *BufferWriter) AddPadding(size int) error

func (*BufferWriter) Buffer 

func (fw *BufferWriter) Buffer() ([]byte, io.Closer, error)

func (*BufferWriter) Close 

func (fw *BufferWriter) Close() error

func (*BufferWriter) Flush 

func (fw *BufferWriter) Flush() error

func (*BufferWriter) Pos 

func (fw *BufferWriter) Pos() uint64

func (*BufferWriter) Read 

func (fw *BufferWriter) Read(buf []byte) (int, error)

func (*BufferWriter) ReadFrom 

func (fw *BufferWriter) ReadFrom(r io.Reader) (int64, error)

func (*BufferWriter) Remove 

func (fw *BufferWriter) Remove() error

func (*BufferWriter) Reset 

func (fw *BufferWriter) Reset()

func (*BufferWriter) Write 

func (fw *BufferWriter) Write(bufs ...[]byte) error

func (*BufferWriter) WriteAt 

func (fw *BufferWriter) WriteAt(buf []byte, pos uint64) error

type ByteSlice 

type ByteSlice interface {
	Len() int
	Range(start, end int) []byte
}

ByteSlice abstracts a byte slice.

type Decoder 

type Decoder struct {
	LookupSymbol func(context.Context, uint32) (string, error)
}

Decoder provides decoding methods for the v1 and v2 index file format.

It currently does not contain decoding methods for all entry types but can be extended by them if there's demand.

func (*Decoder) LabelNamesOffsetsFor 

func (dec *Decoder) LabelNamesOffsetsFor(b []byte) ([]uint32, error)

LabelNamesOffsetsFor decodes the offsets of the name symbols for a given series. They are returned in the same order they're stored, which should be sorted lexicographically.

func (*Decoder) LabelValueFor 

func (dec *Decoder) LabelValueFor(ctx context.Context, b []byte, label string) (string, error)

LabelValueFor decodes a label for a given series.

func (*Decoder) Postings 

func (dec *Decoder) Postings(b []byte) (int, index.Postings, error)

Postings returns a postings list for b and its number of elements.

func (*Decoder) PostingsFromDecbuf 

func (dec *Decoder) PostingsFromDecbuf(d encoding.Decbuf) (int, index.Postings, error)

PostingsFromDecbuf returns a postings list for d and its number of elements.

func (*Decoder) Series 

func (dec *Decoder) Series(b []byte, builder *labels.ScratchBuilder, chks *[]chunks.Meta) error

Series decodes a series entry from the given byte slice into builder and chks. Previous contents of builder can be overwritten - make sure you copy before retaining. Skips reading chunks metadata if chks is nil.

type PostingsEncoder 

type PostingsEncoder func(*encoding.Encbuf, []uint32) error

type Range 

type Range struct {
	Start, End int64
}

Range marks a byte range.

type Reader 

type Reader struct {
	// contains filtered or unexported fields
}

func NewReader 

func NewReader(b ByteSlice) (*Reader, error)

NewReader returns a new index reader on the given byte slice. It automatically handles different format versions.

func (*Reader) Close 

func (r *Reader) Close() error

Close the reader and its underlying resources.

func (*Reader) LabelNames 

func (r *Reader) LabelNames(_ context.Context, matchers ...*labels.Matcher) ([]string, error)

LabelNames returns all the unique label names present in the index. TODO(twilkie) implement support for matchers.

func (*Reader) LabelNamesFor 

func (r *Reader) LabelNamesFor(ctx context.Context, ids ...storage.SeriesRef) ([]string, error)

LabelNamesFor returns all the label names for the series referred to by IDs. The names returned are sorted.

func (*Reader) LabelValueFor 

func (r *Reader) LabelValueFor(ctx context.Context, id storage.SeriesRef, label string) (string, error)

LabelValueFor returns label value for the given label name in the series referred to by ID.

func (*Reader) LabelValues 

func (r *Reader) LabelValues(ctx context.Context, name string, matchers ...*labels.Matcher) ([]string, error)

LabelValues returns value tuples that exist for the given label name. It is not safe to use the return value beyond the lifetime of the byte slice passed into the Reader. TODO(replay): Support filtering by matchers.

func (*Reader) Postings 

func (r *Reader) Postings(ctx context.Context, name string, values ...string) (index.Postings, error)

func (*Reader) PostingsForLabelMatching 

func (r *Reader) PostingsForLabelMatching(ctx context.Context, name string, match func(string) bool) index.Postings

func (*Reader) PostingsForMatchers 

func (r *Reader) PostingsForMatchers(ctx context.Context, ms ...*labels.Matcher) (index.Postings, error)

PostingsForMatchers assembles a single postings iterator against the index reader based on the given matchers. The resulting postings are not ordered by series.

func (*Reader) PostingsRanges 

func (r *Reader) PostingsRanges() (map[labels.Label]Range, error)

PostingsRanges returns a new map of byte range in the underlying index file for all postings lists.

func (*Reader) Series 

func (r *Reader) Series(id storage.SeriesRef, builder *labels.ScratchBuilder, chks *[]chunks.Meta) error

Series reads the series with the given ID and writes its labels and chunks into builder and chks.

func (*Reader) ShardedPostings 

func (r *Reader) ShardedPostings(p index.Postings, shardIndex, shardCount uint64) index.Postings

ShardedPostings returns a postings list filtered by the provided shardIndex out of shardCount.

func (*Reader) Size 

func (r *Reader) Size() int64

Size returns the size of an index file.

func (*Reader) SortedLabelValues 

func (r *Reader) SortedLabelValues(ctx context.Context, name string, matchers ...*labels.Matcher) ([]string, error)

SortedLabelValues returns value tuples that exist for the given label name. It is not safe to use the return value beyond the lifetime of the byte slice passed into the Reader.

func (*Reader) SortedPostings 

func (r *Reader) SortedPostings(p index.Postings) index.Postings

SortedPostings returns the given postings list reordered so that the backing series are sorted.

func (*Reader) SymbolTableSize 

func (r *Reader) SymbolTableSize() uint64

SymbolTableSize returns the symbol table size in bytes.

func (*Reader) Symbols 

func (r *Reader) Symbols() StringIter

Symbols returns an iterator over the symbols that exist within the index.

func (*Reader) Version 

func (r *Reader) Version() int

Version returns the file format version of the underlying index.

type RealByteSlice 

type RealByteSlice []byte

func (RealByteSlice) Len 

func (b RealByteSlice) Len() int

func (RealByteSlice) Range 

func (b RealByteSlice) Range(start, end int) []byte

func (RealByteSlice) Sub 

func (b RealByteSlice) Sub(start, end int) ByteSlice

type StringIter 

type StringIter interface {
	// Next advances the iterator and returns true if another value was found.
	Next() bool

	// At returns the value at the current iterator position.
	At() string

	// Err returns the last error of the iterator.
	Err() error
}

StringIter iterates over a sorted list of strings.

type Symbols 

type Symbols struct {
	// contains filtered or unexported fields
}

func NewSymbols 

func NewSymbols(bs ByteSlice, version, off int) (*Symbols, error)

NewSymbols returns a Symbols object for symbol lookups.

func (Symbols) Iter 

func (s Symbols) Iter() StringIter

func (Symbols) Lookup 

func (s Symbols) Lookup(o uint32) (string, error)

func (Symbols) ReverseLookup 

func (s Symbols) ReverseLookup(sym string) (uint32, error)

func (Symbols) Size 

func (s Symbols) Size() int

type TOC 

type TOC struct {
	Symbols           uint64
	Series            uint64
	LabelIndices      uint64
	LabelIndicesTable uint64
	Postings          uint64
	PostingsTable     uint64
}

TOC represents the index Table Of Contents that states where each section of the index starts.

func NewTOCFromByteSlice 

func NewTOCFromByteSlice(bs ByteSlice) (*TOC, error)

NewTOCFromByteSlice returns a parsed TOC from the given index byte slice.

type Writer 

type Writer struct {
	Version int
	// contains filtered or unexported fields
}

Writer implements the IndexWriter interface for the standard serialization format.

func NewWriter 

func NewWriter() (*Writer, error)

NewWriter creates a new index writer using the default encoder. See NewWriterWithEncoder.

func NewWriterWithEncoder 

func NewWriterWithEncoder(encoder PostingsEncoder) (*Writer, error)

NewWriter returns a new Writer to the given filename. It serializes data in format version 2. It uses the given encoder to encode each postings list.

func (*Writer) AddSeries 

func (w *Writer) AddSeries(ref storage.SeriesRef, lset labels.Labels, chunks ...chunks.Meta) error

AddSeries adds the series one at a time along with its chunks.

func (*Writer) AddSymbol 

func (w *Writer) AddSymbol(sym string) error

func (*Writer) Buffer 

func (w *Writer) Buffer() ([]byte, io.Closer, error)

func (*Writer) Close 

func (w *Writer) Close() error

func (*Writer) Reset 

func (w *Writer) Reset() error

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.