encoding

package
v1.3.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 23, 2022 License: AGPL-3.0 Imports: 20 Imported by: 1

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CopyBlock added in v0.7.0 

func CopyBlock(ctx context.Context, meta *backend.BlockMeta, src backend.Reader, dest backend.Writer) error

CopyBlock copies a block from one backend to another. It is done at a low level, all encoding/formatting is preserved.

func ValidateConfig added in v0.6.0 

func ValidateConfig(b *BlockConfig) error

ValidateConfig returns true if the config is valid

Types

type Appender 

type Appender interface {
	Append(common.ID, []byte) error
	Complete() error
	Records() []common.Record
	RecordsForID(common.ID) []common.Record
	Length() int
	DataLength() uint64
}

Appender is capable of tracking objects and ids that are added to it

func NewAppender 

func NewAppender(dataWriter common.DataWriter) Appender

NewAppender returns an appender. This appender simply appends new objects 

to the provided dataWriter.

func NewBufferedAppender 

func NewBufferedAppender(writer common.DataWriter, indexDownsample int, totalObjectsEstimate int) (Appender, error)

NewBufferedAppender returns an bufferedAppender. This appender builds a writes to 

the provided writer and also builds a downsampled records slice.

func NewRecordAppender added in v1.0.0 

func NewRecordAppender(records []common.Record) Appender

NewRecordAppender returns an appender that stores records only.

type BackendBlock added in v0.5.0 

type BackendBlock struct {
	// contains filtered or unexported fields
}

BackendBlock represents a block already in the backend.

func NewBackendBlock added in v0.5.0 

func NewBackendBlock(meta *backend.BlockMeta, r backend.Reader) (*BackendBlock, error)

NewBackendBlock returns a BackendBlock for the given backend.BlockMeta 

It is version aware.

func (*BackendBlock) BlockMeta added in v0.7.0 

func (b *BackendBlock) BlockMeta() *backend.BlockMeta

func (*BackendBlock) Find added in v0.5.0 

func (b *BackendBlock) Find(ctx context.Context, id common.ID) ([]byte, error)

Find searches a block for the ID and returns an object if found.

func (*BackendBlock) Iterator added in v0.6.0 

func (b *BackendBlock) Iterator(chunkSizeBytes uint32) (Iterator, error)

Iterator returns an Iterator that iterates over the objects in the block from the backend

func (*BackendBlock) NewIndexReader added in v0.7.0 

func (b *BackendBlock) NewIndexReader() (common.IndexReader, error)

func (*BackendBlock) PartialIterator added in v1.3.0 

func (b *BackendBlock) PartialIterator(chunkSizeBytes uint32, startPage int, totalPages int) (Iterator, error)

PartialIterator returns an Iterator that iterates over the a subset of pages in the block from the backend

type BlockConfig added in v0.6.0 

type BlockConfig struct {
	IndexDownsampleBytes int              `yaml:"index_downsample_bytes"`
	IndexPageSizeBytes   int              `yaml:"index_page_size_bytes"`
	BloomFP              float64          `yaml:"bloom_filter_false_positive"`
	BloomShardSizeBytes  int              `yaml:"bloom_filter_shard_size_bytes"`
	Encoding             backend.Encoding `yaml:"encoding"`
	SearchEncoding       backend.Encoding `yaml:"search_encoding"`
	SearchPageSizeBytes  int              `yaml:"search_page_size_bytes"`
}

BlockConfig holds configuration options for newly created blocks

type BufferedAppenderGeneric added in v1.2.0 

type BufferedAppenderGeneric struct {
	// contains filtered or unexported fields
}

bufferedAppender buffers objects into pages and builds a downsampled index

func NewBufferedAppenderGeneric added in v1.2.0 

func NewBufferedAppenderGeneric(writer common.DataWriterGeneric, maxPageSize int) *BufferedAppenderGeneric

NewBufferedAppender returns an bufferedAppender. This appender builds a writes to 

the provided writer and also builds a downsampled records slice.

func (*BufferedAppenderGeneric) Append added in v1.2.0 

func (a *BufferedAppenderGeneric) Append(ctx context.Context, id common.ID, i interface{}) error

Append appends the id/object to the writer. Note that the caller is giving up ownership of the two byte arrays backing the slices. 

Copies should be made and passed in if this is a problem

func (*BufferedAppenderGeneric) Complete added in v1.2.0 

func (a *BufferedAppenderGeneric) Complete(ctx context.Context) error

Complete flushes all buffers and releases resources

func (*BufferedAppenderGeneric) Records added in v1.2.0 

func (a *BufferedAppenderGeneric) Records() []common.Record

Records returns a slice of the current records

type Finder 

type Finder interface {
	Find(context.Context, common.ID) ([]byte, error)
}

Finder is capable of finding the requested ID

func NewPagedFinder added in v0.7.0 

func NewPagedFinder(index common.IndexReader, r common.DataReader, combiner common.ObjectCombiner, objectRW common.ObjectReaderWriter, dataEncoding string) Finder

NewPagedFinder returns a paged. This finder is used for searching 

a set of records and returning an object. If a set of consecutive records has
matching ids they will be combined using the ObjectCombiner.

type Iterator 

type Iterator interface {
	Next(context.Context) (common.ID, []byte, error)
	Close()
}

Iterator is capable of iterating through a set of objects

func NewDedupingIterator 

func NewDedupingIterator(iter Iterator, combiner common.ObjectCombiner, dataEncoding string) (Iterator, error)

NewDedupingIterator returns a dedupingIterator. This iterator is used to wrap another 

iterator.  It will dedupe consecutive objects with the same id using the ObjectCombiner.

func NewIterator 

func NewIterator(reader io.Reader, o common.ObjectReaderWriter) Iterator

NewIterator returns the most basic iterator. It iterates over raw objects.

func NewMultiblockIterator added in v1.1.0 

func NewMultiblockIterator(ctx context.Context, inputs []Iterator, bufferSize int, combiner common.ObjectCombiner, dataEncoding string, logger log.Logger) Iterator

NewMultiblockIterator Creates a new multiblock iterator. Iterates concurrently in a separate goroutine and results are buffered. Traces are deduped and combined using the object combiner.

func NewPrefetchIterator added in v1.2.0 

func NewPrefetchIterator(ctx context.Context, iter Iterator, bufferSize int) Iterator

NewPrefetchIterator Creates a new multiblock iterator. Iterates concurrently in a separate goroutine and results are buffered.

func NewRecordIterator 

func NewRecordIterator(r []common.Record, dataR common.DataReader, objectRW common.ObjectReaderWriter) Iterator

NewRecordIterator returns a recordIterator. This iterator is used for iterating through 

a series of objects by reading them one at a time from Records.

type StreamingBlock added in v0.7.0 

type StreamingBlock struct {
	// contains filtered or unexported fields
}

func NewStreamingBlock added in v0.7.0 

func NewStreamingBlock(cfg *BlockConfig, id uuid.UUID, tenantID string, metas []*backend.BlockMeta, estimatedObjects int) (*StreamingBlock, error)

NewStreamingBlock creates a ... new streaming block. Objects are appended one at a time to the backend.

func (*StreamingBlock) AddObject added in v0.7.0 

func (c *StreamingBlock) AddObject(id common.ID, object []byte) error

func (*StreamingBlock) BlockMeta added in v0.7.0 

func (c *StreamingBlock) BlockMeta() *backend.BlockMeta

func (*StreamingBlock) Complete added in v0.7.0 

func (c *StreamingBlock) Complete(ctx context.Context, tracker backend.AppendTracker, w backend.Writer) (int, error)

Complete finishes writes the compactor metadata and closes all buffers and appenders

func (*StreamingBlock) CurrentBufferLength added in v0.7.0 

func (c *StreamingBlock) CurrentBufferLength() int

func (*StreamingBlock) CurrentBufferedObjects added in v0.7.0 

func (c *StreamingBlock) CurrentBufferedObjects() int

func (*StreamingBlock) FlushBuffer added in v0.7.0 

func (c *StreamingBlock) FlushBuffer(ctx context.Context, tracker backend.AppendTracker, w backend.Writer) (backend.AppendTracker, int, error)

FlushBuffer flushes any existing objects to the backend

func (*StreamingBlock) Length added in v0.7.0 

func (c *StreamingBlock) Length() int

type VersionedEncoding added in v0.7.0 

type VersionedEncoding interface {
	Version() string

	NewDataWriter(writer io.Writer, encoding backend.Encoding) (common.DataWriter, error)
	NewIndexWriter(pageSizeBytes int) common.IndexWriter

	NewDataReader(ra backend.ContextReader, encoding backend.Encoding) (common.DataReader, error)
	NewIndexReader(ra backend.ContextReader, pageSizeBytes int, totalPages int) (common.IndexReader, error)

	NewObjectReaderWriter() common.ObjectReaderWriter
	NewRecordReaderWriter() common.RecordReaderWriter
}

VersionedEncoding has a whole bunch of versioned functionality. This is 

currently quite sloppy and could easily be tightened up to just a few methods
but it is what it is for now!

func FromVersion added in v0.7.0 

func FromVersion(v string) (VersionedEncoding, error)

FromVersion returns a versioned encoding for the provided string

func LatestEncoding added in v0.7.0 

func LatestEncoding() VersionedEncoding

LatestEncoding is used by Compactor and Complete block

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.