storage

README

File Version Compatibility Information

The current file version is 1.0.0, compatible with versions specified in version.yml.

Documentation

Overview

Package storage implements a time-series-based storage engine. It provides:

• Partition data based on a time axis.
• Sharding data based on a series id which represents a unique entity of stream/measure
• Retrieving data based on index.Filter.
• Cleaning expired data, or the data retention.

Index

• Variables
• type IndexDB
• type IndexGranularity
• type IntervalRule
  • func MustToIntervalRule(ir *commonv1.IntervalRule) (result IntervalRule)
• type IntervalUnit
  • func (iu IntervalUnit) String() string
• type SupplyTSDB
• type TSDB
  • func OpenTSDB[T TSTable, O any](ctx context.Context, opts TSDBOpts[T, O]) (TSDB[T, O], error)
• type TSDBOpts
• type TSTable
• type TSTableCreator
• type TSTableWrapper

Variables

var ErrExpiredData = errors.New("expired data")

ErrExpiredData is returned when the data is expired.

var (
	// ErrUnknownShard indicates that the shard is not found.
	ErrUnknownShard = errors.New("unknown shard")
)

Types

type IndexDB

type IndexDB interface {
	Write(docs index.Documents) error
	Search(ctx context.Context, series []*pbv1.Series, filter index.Filter, order *pbv1.OrderBy, preloadSize int) (pbv1.SeriesList, error)
}

IndexDB is the interface of index database.

type IndexGranularity

type IndexGranularity int

IndexGranularity denotes the granularity of the local index.

const (
	IndexGranularityBlock IndexGranularity = iota
	IndexGranularitySeries
)

The options of the local index granularity.

type IntervalRule

type IntervalRule struct {
	Unit IntervalUnit
	Num  int
}

IntervalRule defines a length of two points in time.

func MustToIntervalRule

func MustToIntervalRule(ir *commonv1.IntervalRule) (result IntervalRule)

MustToIntervalRule converts a commonv1.IntervalRule to IntervalRule.

type IntervalUnit

type IntervalUnit int

IntervalUnit denotes the unit of a time point.

const (
	HOUR IntervalUnit = iota
	DAY
)

Available IntervalUnits. HOUR and DAY are adequate for the APM scenario.

func (IntervalUnit) String

func (iu IntervalUnit) String() string

type SupplyTSDB

type SupplyTSDB[T TSTable] func() T

SupplyTSDB allows getting a tsdb's runtime.

type TSDB

type TSDB[T TSTable, O any] interface {
	io.Closer
	Lookup(ctx context.Context, series []*pbv1.Series) (pbv1.SeriesList, error)
	CreateTSTableIfNotExist(shardID common.ShardID, ts time.Time) (TSTableWrapper[T], error)
	SelectTSTables(timeRange timestamp.TimeRange) []TSTableWrapper[T]
	IndexDB() IndexDB
	Tick(ts int64)
}

TSDB allows listing and getting shard details.

func OpenTSDB

func OpenTSDB[T TSTable, O any](ctx context.Context, opts TSDBOpts[T, O]) (TSDB[T, O], error)

OpenTSDB returns a new tsdb runtime. This constructor will create a new database if it's absent, or load an existing one.

type TSDBOpts

type TSDBOpts[T TSTable, O any] struct {
	Option                         O
	TSTableCreator                 TSTableCreator[T, O]
	Location                       string
	SegmentInterval                IntervalRule
	TTL                            IntervalRule
	ShardNum                       uint32
	SeriesIndexFlushTimeoutSeconds int64
}

TSDBOpts wraps options to create a tsdb.

type TSTable

type TSTable interface {
	io.Closer
}

TSTable is time series table.

type TSTableCreator

type TSTableCreator[T TSTable, O any] func(fileSystem fs.FileSystem, root string, position common.Position,
	l *logger.Logger, timeRange timestamp.TimeRange, option O) (T, error)

TSTableCreator creates a TSTable.

type TSTableWrapper

type TSTableWrapper[T TSTable] interface {
	DecRef()
	Table() T
	GetTimeRange() timestamp.TimeRange
}

TSTableWrapper is a wrapper of TSTable. It is used to manage the reference count of TSTable.