series

package
v0.15.17-hotfix.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 28, 2020 License: Apache-2.0 Imports: 27 Imported by: 16

README

series

Series related documentation.

Series flush lifecycle

Warm/cold writes end up in versioned buckets based on write type (ColdWrite or WarmWrite). When flushes occur, we fetch in-mem data from all write type specific buckets to persist.

For warm flushes, we write all warm written buckets to disk and mark the state of the block as WarmRetrievable.

For cold flushes, we merge this data w/ data that's already on disk in fs/merger.go and write to disk. Once finished, we then update the ColdVersionRetrievable to the cold version we just wrote to disk.

Data is only evicted from mem during a Tick(). This evicts either cold buckets up until flush state ColdVersionRetrievable or warm buckets that are marked as WarmRetrievable (or warm blocks that we have already warm flushed to disk).

Snapshotting/Bootstrap

Snapshots work by merging all buckets for a series buffer regardless of write type into streams and persisting to disk. Snapshots are in the commitlog bootstrapper and snapshotted series data are loaded into BufferBucket.loadedBlocks. Attempts to call series.LoadBlock() for WarmWrite blocks will return an error if it already exists on disk.

Series snapshots persist writes in both warm & cold buckets. During a flush, we persist snapshot files w/ a commit log ID. This ID is later used during the async cleanup process to deleted rotated commit logs.

Repair

Shard repairs load data as cold writes into series buffer buckets.

Documentation

Overview

Package series is a generated GoMock package.

Package series is a generated GoMock package.

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrSeriesAllDatapointsExpired is returned on tick when all datapoints are expired
	ErrSeriesAllDatapointsExpired = errors.New("series datapoints are all expired")
)

Functions

func ValidateCachePolicy 

func ValidateCachePolicy(v CachePolicy) error

ValidateCachePolicy validates a cache policy.

Types

type BlockState added in v0.8.2 

type BlockState struct {
	WarmRetrievable bool
	ColdVersion     int
}

BlockState contains the state of a block.

type BootstrappedBlockStateSnapshot added in v0.11.0 

type BootstrappedBlockStateSnapshot struct {
	Snapshot map[xtime.UnixNano]BlockState
}

BootstrappedBlockStateSnapshot represents a bootstrapped shard block state snapshot.

type BufferBucket added in v0.8.2 

type BufferBucket struct {
	// contains filtered or unexported fields
}

BufferBucket is a specific version of a bucket of encoders, which is where writes are ultimately stored before they are persisted to disk as a fileset. See comment for BufferBucketVersions for more detail on bucket versions.

type BufferBucketPool added in v0.8.2 

type BufferBucketPool struct {
	// contains filtered or unexported fields
}

BufferBucketPool provides a pool for BufferBuckets.

func NewBufferBucketPool added in v0.8.2 

func NewBufferBucketPool(opts pool.ObjectPoolOptions) *BufferBucketPool

NewBufferBucketPool creates a new BufferBucketPool.

func (*BufferBucketPool) Get added in v0.8.2 

func (p *BufferBucketPool) Get() *BufferBucket

Get gets a BufferBucket from the pool.

func (*BufferBucketPool) Put added in v0.8.2 

func (p *BufferBucketPool) Put(bucket *BufferBucket)

Put puts a BufferBucket back into the pool.

type BufferBucketVersions added in v0.8.2 

type BufferBucketVersions struct {
	// contains filtered or unexported fields
}

BufferBucketVersions is a container for different versions of buffer buckets. Bucket versions are how the buffer separates writes that have been written to disk as a fileset and writes that have not. The bucket with a version of `writableBucketVersion` is the bucket that all writes go into (as thus is the bucket version that have not yet been persisted). After a bucket gets persisted, its version gets set to a version that the shard passes down to it (since the shard knows what has been fully persisted to disk).

type BufferBucketVersionsPool added in v0.8.2 

type BufferBucketVersionsPool struct {
	// contains filtered or unexported fields
}

BufferBucketVersionsPool provides a pool for BufferBucketVersions.

func NewBufferBucketVersionsPool added in v0.8.2 

func NewBufferBucketVersionsPool(opts pool.ObjectPoolOptions) *BufferBucketVersionsPool

NewBufferBucketVersionsPool creates a new BufferBucketVersionsPool.

func (*BufferBucketVersionsPool) Get added in v0.8.2 

func (p *BufferBucketVersionsPool) Get() *BufferBucketVersions

Get gets a BufferBucketVersions from the pool.

func (*BufferBucketVersionsPool) Put added in v0.8.2 

func (p *BufferBucketVersionsPool) Put(buckets *BufferBucketVersions)

Put puts a BufferBucketVersions back into the pool.

type CachePolicy 

type CachePolicy uint

CachePolicy is the series cache policy. 

const (
	// CacheNone specifies that no series will be cached by default.
	CacheNone CachePolicy = iota
	// CacheAll specifies that all series must be cached at all times
	// which requires loading all into cache on bootstrap and never
	// expiring series from memory until expired from retention.
	CacheAll
	// CacheRecentlyRead specifies that series that are recently read
	// must be cached, configurable by the namespace block expiry after
	// not accessed period.
	CacheRecentlyRead
	// CacheLRU specifies that series that are read will be cached
	// using an LRU of fixed capacity. Series that are least recently
	// used will be evicted first.
	CacheLRU

	// DefaultCachePolicy is the default cache policy.
	DefaultCachePolicy = CacheLRU
)

func ParseCachePolicy 

func ParseCachePolicy(str string) (CachePolicy, error)

ParseCachePolicy parses a CachePolicy from a string.

func ValidCachePolicies 

func ValidCachePolicies() []CachePolicy

ValidCachePolicies returns the valid series cache policies.

func (CachePolicy) String 

func (p CachePolicy) String() string

func (*CachePolicy) UnmarshalYAML 

func (p *CachePolicy) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML unmarshals an CachePolicy into a valid type from string.

type DatabaseSeries 

type DatabaseSeries interface {
	block.OnRetrieveBlock
	block.OnEvictedFromWiredList

	// ID returns the ID of the series.
	ID() ident.ID

	// Metadata returns the metadata of the series.
	Metadata() doc.Document

	// UniqueIndex is the unique index for the series (for this current
	// process, unless the time series expires).
	UniqueIndex() uint64

	// Tick executes async updates
	Tick(blockStates ShardBlockStateSnapshot, nsCtx namespace.Context) (TickResult, error)

	// Write writes a new value.
	Write(
		ctx context.Context,
		timestamp time.Time,
		value float64,
		unit xtime.Unit,
		annotation []byte,
		wOpts WriteOptions,
	) (bool, WriteType, error)

	// ReadEncoded reads encoded blocks.
	ReadEncoded(
		ctx context.Context,
		start, end time.Time,
		nsCtx namespace.Context,
	) ([][]xio.BlockReader, error)

	// FetchBlocks returns data blocks given a list of block start times.
	FetchBlocks(
		ctx context.Context,
		starts []time.Time,
		nsCtx namespace.Context,
	) ([]block.FetchBlockResult, error)

	// FetchBlocksForColdFlush fetches blocks for a cold flush. This function
	// informs the series and the buffer that a cold flush for the specified
	// block start is occurring so that it knows to update bucket versions.
	FetchBlocksForColdFlush(
		ctx context.Context,
		start time.Time,
		version int,
		nsCtx namespace.Context,
	) (block.FetchBlockResult, error)

	// FetchBlocksMetadata returns the blocks metadata.
	FetchBlocksMetadata(
		ctx context.Context,
		start, end time.Time,
		opts FetchBlocksMetadataOptions,
	) (block.FetchBlocksMetadataResult, error)

	// IsEmpty returns whether series is empty (includes both cached blocks and in-mem buffer data).
	IsEmpty() bool

	// IsBufferEmptyAtBlockStart returns whether the series buffer is empty at block start
	// (only checks for in-mem buffer data).
	IsBufferEmptyAtBlockStart(time.Time) bool

	// NumActiveBlocks returns the number of active blocks the series currently holds.
	NumActiveBlocks() int

	/// LoadBlock loads a single block into the series.
	LoadBlock(
		block block.DatabaseBlock,
		writeType WriteType,
	) error

	// WarmFlush flushes the WarmWrites of this series for a given start time.
	WarmFlush(
		ctx context.Context,
		blockStart time.Time,
		persistFn persist.DataFn,
		nsCtx namespace.Context,
	) (FlushOutcome, error)

	// Snapshot snapshots the buffer buckets of this series for any data that has
	// not been rotated into a block yet.
	Snapshot(
		ctx context.Context,
		blockStart time.Time,
		persistFn persist.DataFn,
		nsCtx namespace.Context,
	) (SnapshotResult, error)

	// ColdFlushBlockStarts returns the block starts that need cold flushes.
	ColdFlushBlockStarts(blockStates BootstrappedBlockStateSnapshot) OptimizedTimes

	// Bootstrap will moved any bootstrapped data to buffer so series
	// is ready for reading.
	Bootstrap(nsCtx namespace.Context) error

	// Close will close the series and if pooled returned to the pool.
	Close()

	// Reset resets the series for reuse.
	Reset(opts DatabaseSeriesOptions)
}

DatabaseSeries is a series in the database.

func NewDatabaseSeries 

func NewDatabaseSeries(opts DatabaseSeriesOptions) DatabaseSeries

NewDatabaseSeries creates a new database series.

type DatabaseSeriesAllocate 

type DatabaseSeriesAllocate func() DatabaseSeries

DatabaseSeriesAllocate allocates a database series for a pool.

type DatabaseSeriesOptions added in v0.15.0 

type DatabaseSeriesOptions struct {
	ID                     ident.ID
	Metadata               doc.Document
	UniqueIndex            uint64
	BlockRetriever         QueryableBlockRetriever
	OnRetrieveBlock        block.OnRetrieveBlock
	OnEvictedFromWiredList block.OnEvictedFromWiredList
	Options                Options
}

DatabaseSeriesOptions is a set of options for creating a database series.

type DatabaseSeriesPool 

type DatabaseSeriesPool interface {
	// Get provides a database series from the pool.
	Get() DatabaseSeries

	// Put returns a database series to the pool.
	Put(block DatabaseSeries)
}

DatabaseSeriesPool provides a pool for database series.

func NewDatabaseSeriesPool 

func NewDatabaseSeriesPool(opts pool.ObjectPoolOptions) DatabaseSeriesPool

NewDatabaseSeriesPool creates a new database series pool

type DecodedTestValue added in v0.15.0 

type DecodedTestValue struct {
	// Timestamp is the data point timestamp.
	Timestamp time.Time
	// Value is the data point value.
	Value float64
	// Unit is the data point unit.
	Unit xtime.Unit
	// Annotation is the data point annotation.
	Annotation []byte
}

DecodedTestValue is a decoded datapoint.

func DecodeSegmentValues added in v0.15.0 

func DecodeSegmentValues(
	results []xio.SegmentReader,
	iter encoding.MultiReaderIterator,
	schema namespace.SchemaDescr,
) ([]DecodedTestValue, error)

DecodeSegmentValues is a test utility to read through a slice of SegmentReaders.

type FetchBlocksMetadataOptions 

type FetchBlocksMetadataOptions struct {
	block.FetchBlocksMetadataOptions
}

FetchBlocksMetadataOptions encapsulates block fetch metadata options and specifies a few series specific options too.

type FlushOutcome 

type FlushOutcome int

FlushOutcome is an enum that provides more context about the outcome of series.WarmFlush() to the caller. 

const (
	// FlushOutcomeErr is just a default value that can be returned when we're
	// also returning an error.
	FlushOutcomeErr FlushOutcome = iota
	// FlushOutcomeBlockDoesNotExist indicates that the series did not have a
	// block for the specified flush blockStart.
	FlushOutcomeBlockDoesNotExist
	// FlushOutcomeFlushedToDisk indicates that a block existed and was flushed
	// to disk successfully.
	FlushOutcomeFlushedToDisk
)

type MockDatabaseSeries 

type MockDatabaseSeries struct {
	// contains filtered or unexported fields
}

MockDatabaseSeries is a mock of DatabaseSeries interface

func NewMockDatabaseSeries 

func NewMockDatabaseSeries(ctrl *gomock.Controller) *MockDatabaseSeries

NewMockDatabaseSeries creates a new mock instance

func (*MockDatabaseSeries) Bootstrap 

func (m *MockDatabaseSeries) Bootstrap(arg0 namespace.Context) error

Bootstrap mocks base method

func (*MockDatabaseSeries) Close 

func (m *MockDatabaseSeries) Close()

Close mocks base method

func (*MockDatabaseSeries) ColdFlushBlockStarts added in v0.10.0 

func (m *MockDatabaseSeries) ColdFlushBlockStarts(arg0 BootstrappedBlockStateSnapshot) OptimizedTimes

ColdFlushBlockStarts mocks base method

func (*MockDatabaseSeries) EXPECT 

func (m *MockDatabaseSeries) EXPECT() *MockDatabaseSeriesMockRecorder

EXPECT returns an object that allows the caller to indicate expected use

func (*MockDatabaseSeries) FetchBlocks 

func (m *MockDatabaseSeries) FetchBlocks(arg0 context.Context, arg1 []time.Time, arg2 namespace.Context) ([]block.FetchBlockResult, error)

FetchBlocks mocks base method

func (*MockDatabaseSeries) FetchBlocksForColdFlush added in v0.10.0 

func (m *MockDatabaseSeries) FetchBlocksForColdFlush(arg0 context.Context, arg1 time.Time, arg2 int, arg3 namespace.Context) (block.FetchBlockResult, error)

FetchBlocksForColdFlush mocks base method

func (*MockDatabaseSeries) FetchBlocksMetadata 

func (m *MockDatabaseSeries) FetchBlocksMetadata(arg0 context.Context, arg1, arg2 time.Time, arg3 FetchBlocksMetadataOptions) (block.FetchBlocksMetadataResult, error)

FetchBlocksMetadata mocks base method

func (*MockDatabaseSeries) ID 

func (m *MockDatabaseSeries) ID() ident.ID

ID mocks base method

func (*MockDatabaseSeries) IsBufferEmptyAtBlockStart added in v0.15.7 

func (m *MockDatabaseSeries) IsBufferEmptyAtBlockStart(arg0 time.Time) bool

IsBufferEmptyAtBlockStart mocks base method

func (*MockDatabaseSeries) IsEmpty 

func (m *MockDatabaseSeries) IsEmpty() bool

IsEmpty mocks base method

func (*MockDatabaseSeries) LoadBlock added in v0.15.0 

func (m *MockDatabaseSeries) LoadBlock(arg0 block.DatabaseBlock, arg1 WriteType) error

LoadBlock mocks base method

func (*MockDatabaseSeries) Metadata added in v0.15.4 

func (m *MockDatabaseSeries) Metadata() doc.Document

Metadata mocks base method

func (*MockDatabaseSeries) NumActiveBlocks 

func (m *MockDatabaseSeries) NumActiveBlocks() int

NumActiveBlocks mocks base method

func (*MockDatabaseSeries) OnEvictedFromWiredList 

func (m *MockDatabaseSeries) OnEvictedFromWiredList(arg0 ident.ID, arg1 time.Time)

OnEvictedFromWiredList mocks base method

func (*MockDatabaseSeries) OnRetrieveBlock 

func (m *MockDatabaseSeries) OnRetrieveBlock(arg0 ident.ID, arg1 ident.TagIterator, arg2 time.Time, arg3 ts.Segment, arg4 namespace.Context)

OnRetrieveBlock mocks base method

func (*MockDatabaseSeries) ReadEncoded 

func (m *MockDatabaseSeries) ReadEncoded(arg0 context.Context, arg1, arg2 time.Time, arg3 namespace.Context) ([][]xio.BlockReader, error)

ReadEncoded mocks base method

func (*MockDatabaseSeries) Reset 

func (m *MockDatabaseSeries) Reset(arg0 DatabaseSeriesOptions)

Reset mocks base method

func (*MockDatabaseSeries) Snapshot 

func (m *MockDatabaseSeries) Snapshot(arg0 context.Context, arg1 time.Time, arg2 persist.DataFn, arg3 namespace.Context) (SnapshotResult, error)

Snapshot mocks base method

func (*MockDatabaseSeries) Tick 

func (m *MockDatabaseSeries) Tick(arg0 ShardBlockStateSnapshot, arg1 namespace.Context) (TickResult, error)

Tick mocks base method

func (*MockDatabaseSeries) UniqueIndex added in v0.15.0 

func (m *MockDatabaseSeries) UniqueIndex() uint64

UniqueIndex mocks base method

func (*MockDatabaseSeries) WarmFlush added in v0.10.0 

func (m *MockDatabaseSeries) WarmFlush(arg0 context.Context, arg1 time.Time, arg2 persist.DataFn, arg3 namespace.Context) (FlushOutcome, error)

WarmFlush mocks base method

func (*MockDatabaseSeries) Write 

func (m *MockDatabaseSeries) Write(arg0 context.Context, arg1 time.Time, arg2 float64, arg3 time0.Unit, arg4 []byte, arg5 WriteOptions) (bool, WriteType, error)

Write mocks base method

type MockDatabaseSeriesMockRecorder 

type MockDatabaseSeriesMockRecorder struct {
	// contains filtered or unexported fields
}

MockDatabaseSeriesMockRecorder is the mock recorder for MockDatabaseSeries

func (*MockDatabaseSeriesMockRecorder) Bootstrap 

func (mr *MockDatabaseSeriesMockRecorder) Bootstrap(arg0 interface{}) *gomock.Call

Bootstrap indicates an expected call of Bootstrap

func (*MockDatabaseSeriesMockRecorder) Close 

func (mr *MockDatabaseSeriesMockRecorder) Close() *gomock.Call

Close indicates an expected call of Close

func (*MockDatabaseSeriesMockRecorder) ColdFlushBlockStarts added in v0.10.0 

func (mr *MockDatabaseSeriesMockRecorder) ColdFlushBlockStarts(arg0 interface{}) *gomock.Call

ColdFlushBlockStarts indicates an expected call of ColdFlushBlockStarts

func (*MockDatabaseSeriesMockRecorder) FetchBlocks 

func (mr *MockDatabaseSeriesMockRecorder) FetchBlocks(arg0, arg1, arg2 interface{}) *gomock.Call

FetchBlocks indicates an expected call of FetchBlocks

func (*MockDatabaseSeriesMockRecorder) FetchBlocksForColdFlush added in v0.10.0 

func (mr *MockDatabaseSeriesMockRecorder) FetchBlocksForColdFlush(arg0, arg1, arg2, arg3 interface{}) *gomock.Call

FetchBlocksForColdFlush indicates an expected call of FetchBlocksForColdFlush

func (*MockDatabaseSeriesMockRecorder) FetchBlocksMetadata 

func (mr *MockDatabaseSeriesMockRecorder) FetchBlocksMetadata(arg0, arg1, arg2, arg3 interface{}) *gomock.Call

FetchBlocksMetadata indicates an expected call of FetchBlocksMetadata

func (*MockDatabaseSeriesMockRecorder) ID 

func (mr *MockDatabaseSeriesMockRecorder) ID() *gomock.Call

ID indicates an expected call of ID

func (*MockDatabaseSeriesMockRecorder) IsBufferEmptyAtBlockStart added in v0.15.7 

func (mr *MockDatabaseSeriesMockRecorder) IsBufferEmptyAtBlockStart(arg0 interface{}) *gomock.Call

IsBufferEmptyAtBlockStart indicates an expected call of IsBufferEmptyAtBlockStart

func (*MockDatabaseSeriesMockRecorder) IsEmpty 

func (mr *MockDatabaseSeriesMockRecorder) IsEmpty() *gomock.Call

IsEmpty indicates an expected call of IsEmpty

func (*MockDatabaseSeriesMockRecorder) LoadBlock added in v0.15.0 

func (mr *MockDatabaseSeriesMockRecorder) LoadBlock(arg0, arg1 interface{}) *gomock.Call

LoadBlock indicates an expected call of LoadBlock

func (*MockDatabaseSeriesMockRecorder) Metadata added in v0.15.4 

func (mr *MockDatabaseSeriesMockRecorder) Metadata() *gomock.Call

Metadata indicates an expected call of Metadata

func (*MockDatabaseSeriesMockRecorder) NumActiveBlocks 

func (mr *MockDatabaseSeriesMockRecorder) NumActiveBlocks() *gomock.Call

NumActiveBlocks indicates an expected call of NumActiveBlocks

func (*MockDatabaseSeriesMockRecorder) OnEvictedFromWiredList 

func (mr *MockDatabaseSeriesMockRecorder) OnEvictedFromWiredList(arg0, arg1 interface{}) *gomock.Call

OnEvictedFromWiredList indicates an expected call of OnEvictedFromWiredList

func (*MockDatabaseSeriesMockRecorder) OnRetrieveBlock 

func (mr *MockDatabaseSeriesMockRecorder) OnRetrieveBlock(arg0, arg1, arg2, arg3, arg4 interface{}) *gomock.Call

OnRetrieveBlock indicates an expected call of OnRetrieveBlock

func (*MockDatabaseSeriesMockRecorder) ReadEncoded 

func (mr *MockDatabaseSeriesMockRecorder) ReadEncoded(arg0, arg1, arg2, arg3 interface{}) *gomock.Call

ReadEncoded indicates an expected call of ReadEncoded

func (*MockDatabaseSeriesMockRecorder) Reset 

func (mr *MockDatabaseSeriesMockRecorder) Reset(arg0 interface{}) *gomock.Call

Reset indicates an expected call of Reset

func (*MockDatabaseSeriesMockRecorder) Snapshot 

func (mr *MockDatabaseSeriesMockRecorder) Snapshot(arg0, arg1, arg2, arg3 interface{}) *gomock.Call

Snapshot indicates an expected call of Snapshot

func (*MockDatabaseSeriesMockRecorder) Tick 

func (mr *MockDatabaseSeriesMockRecorder) Tick(arg0, arg1 interface{}) *gomock.Call

Tick indicates an expected call of Tick

func (*MockDatabaseSeriesMockRecorder) UniqueIndex added in v0.15.0 

func (mr *MockDatabaseSeriesMockRecorder) UniqueIndex() *gomock.Call

UniqueIndex indicates an expected call of UniqueIndex

func (*MockDatabaseSeriesMockRecorder) WarmFlush added in v0.10.0 

func (mr *MockDatabaseSeriesMockRecorder) WarmFlush(arg0, arg1, arg2, arg3 interface{}) *gomock.Call

WarmFlush indicates an expected call of WarmFlush

func (*MockDatabaseSeriesMockRecorder) Write 

func (mr *MockDatabaseSeriesMockRecorder) Write(arg0, arg1, arg2, arg3, arg4, arg5 interface{}) *gomock.Call

Write indicates an expected call of Write

type MockQueryableBlockRetriever 

type MockQueryableBlockRetriever struct {
	// contains filtered or unexported fields
}

MockQueryableBlockRetriever is a mock of QueryableBlockRetriever interface

func NewMockQueryableBlockRetriever 

func NewMockQueryableBlockRetriever(ctrl *gomock.Controller) *MockQueryableBlockRetriever

NewMockQueryableBlockRetriever creates a new mock instance

func (*MockQueryableBlockRetriever) BlockStatesSnapshot added in v0.8.2 

func (m *MockQueryableBlockRetriever) BlockStatesSnapshot() ShardBlockStateSnapshot

BlockStatesSnapshot mocks base method

func (*MockQueryableBlockRetriever) EXPECT 

func (m *MockQueryableBlockRetriever) EXPECT() *MockQueryableBlockRetrieverMockRecorder

EXPECT returns an object that allows the caller to indicate expected use

func (*MockQueryableBlockRetriever) IsBlockRetrievable 

func (m *MockQueryableBlockRetriever) IsBlockRetrievable(arg0 time.Time) (bool, error)

IsBlockRetrievable mocks base method

func (*MockQueryableBlockRetriever) RetrievableBlockColdVersion added in v0.10.0 

func (m *MockQueryableBlockRetriever) RetrievableBlockColdVersion(arg0 time.Time) (int, error)

RetrievableBlockColdVersion mocks base method

func (*MockQueryableBlockRetriever) Stream 

func (m *MockQueryableBlockRetriever) Stream(arg0 context.Context, arg1 ident.ID, arg2 time.Time, arg3 block.OnRetrieveBlock, arg4 namespace.Context) (xio.BlockReader, error)

Stream mocks base method

type MockQueryableBlockRetrieverMockRecorder 

type MockQueryableBlockRetrieverMockRecorder struct {
	// contains filtered or unexported fields
}

MockQueryableBlockRetrieverMockRecorder is the mock recorder for MockQueryableBlockRetriever

func (*MockQueryableBlockRetrieverMockRecorder) BlockStatesSnapshot added in v0.8.2 

func (mr *MockQueryableBlockRetrieverMockRecorder) BlockStatesSnapshot() *gomock.Call

BlockStatesSnapshot indicates an expected call of BlockStatesSnapshot

func (*MockQueryableBlockRetrieverMockRecorder) IsBlockRetrievable 

func (mr *MockQueryableBlockRetrieverMockRecorder) IsBlockRetrievable(arg0 interface{}) *gomock.Call

IsBlockRetrievable indicates an expected call of IsBlockRetrievable

func (*MockQueryableBlockRetrieverMockRecorder) RetrievableBlockColdVersion added in v0.10.0 

func (mr *MockQueryableBlockRetrieverMockRecorder) RetrievableBlockColdVersion(arg0 interface{}) *gomock.Call

RetrievableBlockColdVersion indicates an expected call of RetrievableBlockColdVersion

func (*MockQueryableBlockRetrieverMockRecorder) Stream 

func (mr *MockQueryableBlockRetrieverMockRecorder) Stream(arg0, arg1, arg2, arg3, arg4 interface{}) *gomock.Call

Stream indicates an expected call of Stream

type MockdatabaseBuffer 

type MockdatabaseBuffer struct {
	// contains filtered or unexported fields
}

MockdatabaseBuffer is a mock of databaseBuffer interface

func NewMockdatabaseBuffer 

func NewMockdatabaseBuffer(ctrl *gomock.Controller) *MockdatabaseBuffer

NewMockdatabaseBuffer creates a new mock instance

func (*MockdatabaseBuffer) ColdFlushBlockStarts added in v0.10.0 

func (m *MockdatabaseBuffer) ColdFlushBlockStarts(blockStates map[time0.UnixNano]BlockState) OptimizedTimes

ColdFlushBlockStarts mocks base method

func (*MockdatabaseBuffer) EXPECT 

func (m *MockdatabaseBuffer) EXPECT() *MockdatabaseBufferMockRecorder

EXPECT returns an object that allows the caller to indicate expected use

func (*MockdatabaseBuffer) FetchBlocks 

func (m *MockdatabaseBuffer) FetchBlocks(ctx context.Context, starts []time.Time, nsCtx namespace.Context) []block.FetchBlockResult

FetchBlocks mocks base method

func (*MockdatabaseBuffer) FetchBlocksForColdFlush added in v0.10.0 

func (m *MockdatabaseBuffer) FetchBlocksForColdFlush(ctx context.Context, start time.Time, version int, nsCtx namespace.Context) (block.FetchBlockResult, error)

FetchBlocksForColdFlush mocks base method

func (*MockdatabaseBuffer) FetchBlocksMetadata 

func (m *MockdatabaseBuffer) FetchBlocksMetadata(ctx context.Context, start, end time.Time, opts FetchBlocksMetadataOptions) (block.FetchBlockMetadataResults, error)

FetchBlocksMetadata mocks base method

func (*MockdatabaseBuffer) IsEmpty 

func (m *MockdatabaseBuffer) IsEmpty() bool

IsEmpty mocks base method

func (*MockdatabaseBuffer) IsEmptyAtBlockStart added in v0.15.7 

func (m *MockdatabaseBuffer) IsEmptyAtBlockStart(arg0 time.Time) bool

IsEmptyAtBlockStart mocks base method

func (*MockdatabaseBuffer) Load added in v0.11.0 

func (m *MockdatabaseBuffer) Load(bl block.DatabaseBlock, writeType WriteType)

Load mocks base method

func (*MockdatabaseBuffer) MoveTo added in v0.15.6 

func (m *MockdatabaseBuffer) MoveTo(buffer databaseBuffer, nsCtx namespace.Context) error

MoveTo mocks base method

func (*MockdatabaseBuffer) ReadEncoded 

func (m *MockdatabaseBuffer) ReadEncoded(ctx context.Context, start, end time.Time, nsCtx namespace.Context) ([][]xio.BlockReader, error)

ReadEncoded mocks base method

func (*MockdatabaseBuffer) Reset 

func (m *MockdatabaseBuffer) Reset(opts databaseBufferResetOptions)

Reset mocks base method

func (*MockdatabaseBuffer) Snapshot 

func (m *MockdatabaseBuffer) Snapshot(ctx context.Context, blockStart time.Time, metadata persist.Metadata, persistFn persist.DataFn, nsCtx namespace.Context) (SnapshotResult, error)

Snapshot mocks base method

func (*MockdatabaseBuffer) Stats 

func (m *MockdatabaseBuffer) Stats() bufferStats

Stats mocks base method

func (*MockdatabaseBuffer) Tick 

func (m *MockdatabaseBuffer) Tick(versions ShardBlockStateSnapshot, nsCtx namespace.Context) bufferTickResult

Tick mocks base method

func (*MockdatabaseBuffer) WarmFlush added in v0.10.0 

func (m *MockdatabaseBuffer) WarmFlush(ctx context.Context, blockStart time.Time, metadata persist.Metadata, persistFn persist.DataFn, nsCtx namespace.Context) (FlushOutcome, error)

WarmFlush mocks base method

func (*MockdatabaseBuffer) Write 

func (m *MockdatabaseBuffer) Write(ctx context.Context, id ident.ID, timestamp time.Time, value float64, unit time0.Unit, annotation []byte, wOpts WriteOptions) (bool, WriteType, error)

Write mocks base method

type MockdatabaseBufferMockRecorder 

type MockdatabaseBufferMockRecorder struct {
	// contains filtered or unexported fields
}

MockdatabaseBufferMockRecorder is the mock recorder for MockdatabaseBuffer

func (*MockdatabaseBufferMockRecorder) ColdFlushBlockStarts added in v0.10.0 

func (mr *MockdatabaseBufferMockRecorder) ColdFlushBlockStarts(blockStates interface{}) *gomock.Call

ColdFlushBlockStarts indicates an expected call of ColdFlushBlockStarts

func (*MockdatabaseBufferMockRecorder) FetchBlocks 

func (mr *MockdatabaseBufferMockRecorder) FetchBlocks(ctx, starts, nsCtx interface{}) *gomock.Call

FetchBlocks indicates an expected call of FetchBlocks

func (*MockdatabaseBufferMockRecorder) FetchBlocksForColdFlush added in v0.10.0 

func (mr *MockdatabaseBufferMockRecorder) FetchBlocksForColdFlush(ctx, start, version, nsCtx interface{}) *gomock.Call

FetchBlocksForColdFlush indicates an expected call of FetchBlocksForColdFlush

func (*MockdatabaseBufferMockRecorder) FetchBlocksMetadata 

func (mr *MockdatabaseBufferMockRecorder) FetchBlocksMetadata(ctx, start, end, opts interface{}) *gomock.Call

FetchBlocksMetadata indicates an expected call of FetchBlocksMetadata

func (*MockdatabaseBufferMockRecorder) IsEmpty 

func (mr *MockdatabaseBufferMockRecorder) IsEmpty() *gomock.Call

IsEmpty indicates an expected call of IsEmpty

func (*MockdatabaseBufferMockRecorder) IsEmptyAtBlockStart added in v0.15.7 

func (mr *MockdatabaseBufferMockRecorder) IsEmptyAtBlockStart(arg0 interface{}) *gomock.Call

IsEmptyAtBlockStart indicates an expected call of IsEmptyAtBlockStart

func (*MockdatabaseBufferMockRecorder) Load added in v0.11.0 

func (mr *MockdatabaseBufferMockRecorder) Load(bl, writeType interface{}) *gomock.Call

Load indicates an expected call of Load

func (*MockdatabaseBufferMockRecorder) MoveTo added in v0.15.6 

func (mr *MockdatabaseBufferMockRecorder) MoveTo(buffer, nsCtx interface{}) *gomock.Call

MoveTo indicates an expected call of MoveTo

func (*MockdatabaseBufferMockRecorder) ReadEncoded 

func (mr *MockdatabaseBufferMockRecorder) ReadEncoded(ctx, start, end, nsCtx interface{}) *gomock.Call

ReadEncoded indicates an expected call of ReadEncoded

func (*MockdatabaseBufferMockRecorder) Reset 

func (mr *MockdatabaseBufferMockRecorder) Reset(opts interface{}) *gomock.Call

Reset indicates an expected call of Reset

func (*MockdatabaseBufferMockRecorder) Snapshot 

func (mr *MockdatabaseBufferMockRecorder) Snapshot(ctx, blockStart, metadata, persistFn, nsCtx interface{}) *gomock.Call

Snapshot indicates an expected call of Snapshot

func (*MockdatabaseBufferMockRecorder) Stats 

func (mr *MockdatabaseBufferMockRecorder) Stats() *gomock.Call

Stats indicates an expected call of Stats

func (*MockdatabaseBufferMockRecorder) Tick 

func (mr *MockdatabaseBufferMockRecorder) Tick(versions, nsCtx interface{}) *gomock.Call

Tick indicates an expected call of Tick

func (*MockdatabaseBufferMockRecorder) WarmFlush added in v0.10.0 

func (mr *MockdatabaseBufferMockRecorder) WarmFlush(ctx, blockStart, metadata, persistFn, nsCtx interface{}) *gomock.Call

WarmFlush indicates an expected call of WarmFlush

func (*MockdatabaseBufferMockRecorder) Write 

func (mr *MockdatabaseBufferMockRecorder) Write(ctx, id, timestamp, value, unit, annotation, wOpts interface{}) *gomock.Call

Write indicates an expected call of Write

type OptimizedTimes added in v0.10.0 

type OptimizedTimes struct {
	// contains filtered or unexported fields
}

OptimizedTimes is a struct that holds an unknown number of times. This is used to avoid heap allocations as much as possible by trying to not allocate a slice of times. To do this, `optimizedTimesArraySize` needs to be strategically sized such that for the vast majority of the time, the internal array can hold all the times required so that `slice` is nil.

OptimizedTimes should only be interacted with via its helper functions - its fields should never be accessed or modified directly, which could cause an invalid state.

func (*OptimizedTimes) Add added in v0.10.0 

func (t *OptimizedTimes) Add(newTime xtime.UnixNano)

Add adds a time to this OptimizedTimes.

func (*OptimizedTimes) Contains added in v0.10.0 

func (t *OptimizedTimes) Contains(target xtime.UnixNano) bool

Contains returns whether the target time is in this OptimizedTimes.

func (*OptimizedTimes) ForEach added in v0.10.0 

func (t *OptimizedTimes) ForEach(fn func(t xtime.UnixNano))

ForEach runs the given function for each time in this OptimizedTimes.

func (*OptimizedTimes) Len added in v0.10.0 

func (t *OptimizedTimes) Len() int

Len returns the number of times in this OptimizedTimes.

type Options 

type Options interface {
	// Validate validates the options
	Validate() error

	// SetClockOptions sets the clock options
	SetClockOptions(value clock.Options) Options

	// ClockOptions returns the clock options
	ClockOptions() clock.Options

	// SetInstrumentOptions sets the instrumentation options
	SetInstrumentOptions(value instrument.Options) Options

	// InstrumentOptions returns the instrumentation options
	InstrumentOptions() instrument.Options

	// SetRetentionOptions sets the retention options
	SetRetentionOptions(value retention.Options) Options

	// RetentionOptions returns the retention options
	RetentionOptions() retention.Options

	// SetDatabaseBlockOptions sets the database block options
	SetDatabaseBlockOptions(value block.Options) Options

	// DatabaseBlockOptions returns the database block options
	DatabaseBlockOptions() block.Options

	// SetCachePolicy sets the series cache policy
	SetCachePolicy(value CachePolicy) Options

	// CachePolicy returns the series cache policy
	CachePolicy() CachePolicy

	// SetContextPool sets the contextPool
	SetContextPool(value context.Pool) Options

	// ContextPool returns the contextPool
	ContextPool() context.Pool

	// SetEncoderPool sets the contextPool
	SetEncoderPool(value encoding.EncoderPool) Options

	// EncoderPool returns the contextPool
	EncoderPool() encoding.EncoderPool

	// SetMultiReaderIteratorPool sets the multiReaderIteratorPool
	SetMultiReaderIteratorPool(value encoding.MultiReaderIteratorPool) Options

	// MultiReaderIteratorPool returns the multiReaderIteratorPool
	MultiReaderIteratorPool() encoding.MultiReaderIteratorPool

	// SetFetchBlockMetadataResultsPool sets the fetchBlockMetadataResultsPool
	SetFetchBlockMetadataResultsPool(value block.FetchBlockMetadataResultsPool) Options

	// FetchBlockMetadataResultsPool returns the fetchBlockMetadataResultsPool
	FetchBlockMetadataResultsPool() block.FetchBlockMetadataResultsPool

	// SetIdentifierPool sets the identifierPool
	SetIdentifierPool(value ident.Pool) Options

	// IdentifierPool returns the identifierPool
	IdentifierPool() ident.Pool

	// SetStats sets the configured Stats.
	SetStats(value Stats) Options

	// Stats returns the configured Stats.
	Stats() Stats

	// SetColdWritesEnabled sets whether cold writes are enabled.
	SetColdWritesEnabled(value bool) Options

	// ColdWritesEnabled returns whether cold writes are enabled.
	ColdWritesEnabled() bool

	// SetBufferBucketVersionsPool sets the BufferBucketVersionsPool.
	SetBufferBucketVersionsPool(value *BufferBucketVersionsPool) Options

	// BufferBucketVersionsPool returns the BufferBucketVersionsPool.
	BufferBucketVersionsPool() *BufferBucketVersionsPool

	// SetBufferBucketPool sets the BufferBucketPool.
	SetBufferBucketPool(value *BufferBucketPool) Options

	// BufferBucketPool returns the BufferBucketPool.
	BufferBucketPool() *BufferBucketPool

	// SetRuntimeOptionsManager sets the runtime options manager.
	SetRuntimeOptionsManager(value runtime.OptionsManager) Options

	// RuntimeOptionsManager returns the runtime options manager.
	RuntimeOptionsManager() runtime.OptionsManager
}

Options represents the options for series

func NewOptions 

func NewOptions() Options

NewOptions creates new database series options

type QueryableBlockRetriever 

type QueryableBlockRetriever interface {
	block.DatabaseShardBlockRetriever

	// IsBlockRetrievable returns whether a block is retrievable
	// for a given block start time.
	IsBlockRetrievable(blockStart time.Time) (bool, error)

	// RetrievableBlockColdVersion returns the cold version that was
	// successfully persisted.
	RetrievableBlockColdVersion(blockStart time.Time) (int, error)

	// BlockStatesSnapshot returns a snapshot of the whether blocks are
	// retrievable and their flush versions for each block start. This is used
	// to reduce lock contention of acquiring flush state.
	//
	// Flushes may occur and change the actual block state while iterating
	// through this snapshot, so any logic using this function should take this
	// into account.
	BlockStatesSnapshot() ShardBlockStateSnapshot
}

QueryableBlockRetriever is a block retriever that can tell if a block is retrievable or not for a given start time.

type Reader 

type Reader struct {
	// contains filtered or unexported fields
}

Reader reads results from a series, or a series block retriever or both. It is implemented as a struct so it can be allocated on the stack.

func NewReaderUsingRetriever 

func NewReaderUsingRetriever(
	id ident.ID,
	retriever QueryableBlockRetriever,
	onRetrieveBlock block.OnRetrieveBlock,
	onReadBlock block.OnReadBlock,
	opts Options,
) Reader

NewReaderUsingRetriever returns a reader for a series block retriever, it will use the block retriever as the source to read blocks from.

func (Reader) FetchBlocks 

func (r Reader) FetchBlocks(
	ctx context.Context,
	starts []time.Time,
	nsCtx namespace.Context,
) ([]block.FetchBlockResult, error)

FetchBlocks returns data blocks given a list of block start times using just a block retriever.

func (Reader) ReadEncoded 

func (r Reader) ReadEncoded(
	ctx context.Context,
	start, end time.Time,
	nsCtx namespace.Context,
) ([][]xio.BlockReader, error)

ReadEncoded reads encoded blocks using just a block retriever.

type ShardBlockStateSnapshot added in v0.11.0 

type ShardBlockStateSnapshot struct {
	// contains filtered or unexported fields
}

ShardBlockStateSnapshot represents a snapshot of a shard's block state at a moment in time.

func NewShardBlockStateSnapshot added in v0.11.0 

func NewShardBlockStateSnapshot(
	bootstrapped bool,
	snapshot BootstrappedBlockStateSnapshot,
) ShardBlockStateSnapshot

NewShardBlockStateSnapshot constructs a new NewShardBlockStateSnapshot.

func (ShardBlockStateSnapshot) UnwrapValue added in v0.11.0 

func (s ShardBlockStateSnapshot) UnwrapValue() (BootstrappedBlockStateSnapshot, bool)

UnwrapValue returns a BootstrappedBlockStateSnapshot and a boolean indicating whether the snapshot is bootstrapped or not.

type SnapshotResult added in v0.15.14 

type SnapshotResult struct {
	Persist bool
	Stats   SnapshotResultStats
}

SnapshotResult contains metadata regarding the snapshot.

type SnapshotResultStats added in v0.15.14 

type SnapshotResultStats struct {
	TimeMergeByBucket      time.Duration
	TimeMergeAcrossBuckets time.Duration
	TimeChecksum           time.Duration
	TimePersist            time.Duration
}

SnapshotResultStats contains stats regarding the snapshot.

func (*SnapshotResultStats) Add added in v0.15.14 

func (r *SnapshotResultStats) Add(other SnapshotResultStats)

Add adds the result of a snapshot result to this result.

type Stats 

type Stats struct {
	// contains filtered or unexported fields
}

Stats is passed down from namespace/shard to avoid allocations per series.

func NewStats 

func NewStats(scope tally.Scope) Stats

NewStats returns a new Stats for the provided scope.

func (Stats) IncColdWrites added in v0.11.0 

func (s Stats) IncColdWrites()

IncColdWrites incs the ColdWrites stat.

func (Stats) IncCreatedEncoders 

func (s Stats) IncCreatedEncoders()

IncCreatedEncoders incs the EncoderCreated stat.

func (Stats) IncEncoderLimitWriteRejected added in v0.15.13 

func (s Stats) IncEncoderLimitWriteRejected()

IncEncoderLimitWriteRejected incs the encoderLimitWriteRejected stat.

func (Stats) RecordEncodersPerBlock added in v0.15.13 

func (s Stats) RecordEncodersPerBlock(num int)

RecordEncodersPerBlock records the number of encoders histogram.

type TickResult 

type TickResult struct {
	TickStatus
	// MadeExpiredBlocks is count of blocks just expired.
	MadeExpiredBlocks int
	// MadeUnwiredBlocks is count of blocks just unwired from memory.
	MadeUnwiredBlocks int
	// MergedOutOfOrderBlocks is count of blocks merged from out of order streams.
	MergedOutOfOrderBlocks int
	// EvictedBuckets is count of buckets just evicted from the buffer map.
	EvictedBuckets int
}

TickResult is a set of results from a tick.

type TickStatus 

type TickStatus struct {
	// ActiveBlocks is the number of total active blocks.
	ActiveBlocks int
	// WiredBlocks is the number of blocks wired in memory (all data kept)
	WiredBlocks int
	// UnwiredBlocks is the number of blocks unwired (data kept on disk).
	UnwiredBlocks int
	// PendingMergeBlocks is the number of blocks pending merges.
	PendingMergeBlocks int
}

TickStatus is the status of a series for a given tick.

type TruncateType added in v0.9.0 

type TruncateType uint8

TruncateType determines the scheme for truncating transforms. 

const (
	// TypeNone indicates that no truncation occurs.
	TypeNone TruncateType = iota

	// TypeBlock truncates incoming writes to the block boundary immediately
	// preceding this point's timestamp.
	TypeBlock
)

func (TruncateType) String added in v0.9.0 

func (t TruncateType) String() string

func (*TruncateType) UnmarshalYAML added in v0.9.0 

func (t *TruncateType) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML unmarshals a stored truncation type.

func (TruncateType) Validate added in v0.9.0 

func (t TruncateType) Validate() error

Validate validates that the scheme type is valid.

type ValuesByTime added in v0.15.0 

type ValuesByTime []DecodedTestValue

ValuesByTime is a sortable slice of DecodedTestValue.

func (ValuesByTime) Len added in v0.15.0 

func (v ValuesByTime) Len() int

Len is the number of elements in the collection.

func (ValuesByTime) Less added in v0.15.0 

func (v ValuesByTime) Less(lhs, rhs int) bool

Less reports whether the element with index i should sort before the element with index j.

func (ValuesByTime) Swap added in v0.15.0 

func (v ValuesByTime) Swap(lhs, rhs int)

Swap swaps the elements with indexes i and j.

type WriteOptions added in v0.8.2 

type WriteOptions struct {
	// SchemaDesc is the schema description.
	SchemaDesc namespace.SchemaDescr
	// TruncateType is the truncation type for incoming writes.
	TruncateType TruncateType
	// TransformOptions describes transformation options for incoming writes.
	TransformOptions WriteTransformOptions
	// BootstrapWrite allows a warm write outside the time window as long as the
	// block hasn't already been flushed to disk. This is useful for
	// bootstrappers filling data that they know has not yet been flushed to
	// disk.
	BootstrapWrite bool
	// SkipOutOfRetention allows for skipping writes that are out of retention
	// by just returning success, this allows for callers to not have to
	// deal with clock skew when they are trying to write a value that may not
	// fall into retention but they do not care if it fails to write due to
	// it just having fallen out of retention (time race).
	SkipOutOfRetention bool
}

WriteOptions provides a set of options for a write.

type WriteTransformOptions added in v0.9.0 

type WriteTransformOptions struct {
	// ForceValueEnabled indicates if the values for incoming writes
	// should be forced to `ForceValue`.
	ForceValueEnabled bool
	// ForceValue is the value that incoming writes should be forced to.
	ForceValue float64
}

WriteTransformOptions describes transforms to run on incoming writes.

type WriteType added in v0.8.2 

type WriteType int

WriteType is an enum for warm/cold write types. 

const (
	// WarmWrite represents warm writes (within the buffer past/future window).
	WarmWrite WriteType = iota

	// ColdWrite represents cold writes (outside the buffer past/future window).
	ColdWrite
)

Source Files

View all Source files

Directories

Path Synopsis
Package lookup is a generated GoMock package.
 Package lookup is a generated GoMock package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.