tsdb

package
v2.0.0-rc.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 21, 2020 License: MIT Imports: 42 Imported by: 2

README

Line Protocol

The line protocol is a text based format for writing points to InfluxDB. Each line defines a single point. Multiple lines must be separated by the newline character \n. The format of the line consists of three parts:

[key] [fields] [timestamp]

Each section is separated by spaces. The minimum required point consists of a measurement name and at least one field. Points without a specified timestamp will be written using the server's local timestamp. Timestamps are assumed to be in nanoseconds unless a precision value is passed in the query string.

Key

The key is the measurement name and any optional tags separated by commas. Measurement names, tag keys, and tag values must escape any spaces or commas using a backslash (\). For example: \ and \,. All tag values are stored as strings and should not be surrounded in quotes.

Tags should be sorted by key before being sent for best performance. The sort should match that from the Go bytes.Compare function (http://golang.org/pkg/bytes/#Compare).

Examples
# measurement only
cpu

# measurement and tags
cpu,host=serverA,region=us-west

# measurement with commas
cpu\,01,host=serverA,region=us-west

# tag value with spaces
cpu,host=server\ A,region=us\ west

Fields

Fields are key-value metrics associated with the measurement. Every line must have at least one field. Multiple fields must be separated with commas and not spaces.

Field keys are always strings and follow the same syntactical rules as described above for tag keys and values. Field values can be one of four types. The first value written for a given field on a given measurement defines the type of that field for all series under that measurement.

  • integer - Numeric values that do not include a decimal and are followed by a trailing i when inserted (e.g. 1i, 345i, 2015i, -10i). Note that all values must have a trailing i. If they do not they will be written as floats.
  • float - Numeric values that are not followed by a trailing i. (e.g. 1, 1.0, -3.14, 6.0+e5, 10).
  • boolean - A value indicating true or false. Valid boolean strings are (t, T, true, TRUE, f, F, false, and FALSE).
  • string - A text value. All string values must be surrounded in double-quotes ". If the string contains a double-quote or backslashes, it must be escaped with a backslash, e.g. \", \\.
# integer value
cpu value=1i

cpu value=1.1i # will result in a parse error

# float value
cpu_load value=1

cpu_load value=1.0

cpu_load value=1.2

# boolean value
error fatal=true

# string value
event msg="logged out"

# multiple values
cpu load=10,alert=true,reason="value above maximum threshold"

Timestamp

The timestamp section is optional but should be specified if possible. The value is an integer representing nanoseconds since the epoch. If the timestamp is not provided the point will inherit the server's local timestamp.

Some write APIs allow passing a lower precision. If the API supports a lower precision, the timestamp may also be an integer epoch in microseconds, milliseconds, seconds, minutes or hours.

Full Example

A full example is shown below.

cpu,host=server01,region=uswest value=1 1434055562000000000
cpu,host=server02,region=uswest value=3 1434055562000010000

In this example the first line shows a measurement of "cpu", there are two tags "host" and "region, the value is 1.0, and the timestamp is 1434055562000000000. Following this is a second line, also a point in the measurement "cpu" but belonging to a different "host".

cpu,host=server\ 01,region=uswest value=1,msg="all systems nominal"
cpu,host=server\ 01,region=us\,west value_int=1i

In these examples, the "host" is set to server 01. The field value associated with field key msg is double-quoted, as it is a string. The second example shows a region of us,west with the comma properly escaped. In the first example value is written as a floating point number. In the second, value_int is an integer.

Distributed Queries

Documentation

Index

Constants

View Source 
const (
	// DefaultEngine is the default engine for new shards
	DefaultEngine = "tsm1"

	// DefaultIndex is the default index for new shards
	DefaultIndex = TSI1IndexName

	// DefaultCacheMaxMemorySize is the maximum size a shard's cache can
	// reach before it starts rejecting writes.
	DefaultCacheMaxMemorySize = 1024 * 1024 * 1024 // 1GB

	// DefaultCacheSnapshotMemorySize is the size at which the engine will
	// snapshot the cache and write it to a TSM file, freeing up memory
	DefaultCacheSnapshotMemorySize = 25 * 1024 * 1024 // 25MB

	// DefaultCacheSnapshotWriteColdDuration is the length of time at which
	// the engine will snapshot the cache and write it to a new TSM file if
	// the shard hasn't received writes or deletes
	DefaultCacheSnapshotWriteColdDuration = time.Duration(10 * time.Minute)

	// DefaultCompactFullWriteColdDuration is the duration at which the engine
	// will compact all TSM files in a shard if it hasn't received a write or delete
	DefaultCompactFullWriteColdDuration = time.Duration(4 * time.Hour)

	// DefaultCompactThroughput is the rate limit in bytes per second that we
	// will allow TSM compactions to write to disk. Not that short bursts are allowed
	// to happen at a possibly larger value, set by DefaultCompactThroughputBurst.
	// A value of 0 here will disable compaction rate limiting
	DefaultCompactThroughput = 48 * 1024 * 1024

	// DefaultCompactThroughputBurst is the rate limit in bytes per second that we
	// will allow TSM compactions to write to disk. If this is not set, the burst value
	// will be set to equal the normal throughput
	DefaultCompactThroughputBurst = 48 * 1024 * 1024

	// DefaultMaxPointsPerBlock is the maximum number of points in an encoded
	// block in a TSM file
	DefaultMaxPointsPerBlock = 1000

	// DefaultMaxSeriesPerDatabase is the maximum number of series a node can hold per database.
	// This limit only applies to the "inmem" index.
	DefaultMaxSeriesPerDatabase = 1000000

	// DefaultMaxValuesPerTag is the maximum number of values a tag can have within a measurement.
	DefaultMaxValuesPerTag = 100000

	// DefaultMaxConcurrentCompactions is the maximum number of concurrent full and level compactions
	// that can run at one time.  A value of 0 results in 50% of runtime.GOMAXPROCS(0) used at runtime.
	DefaultMaxConcurrentCompactions = 0

	// DefaultMaxIndexLogFileSize is the default threshold, in bytes, when an index
	// write-ahead log file will compact into an index file.
	DefaultMaxIndexLogFileSize = 1 * 1024 * 1024 // 1MB

	// DefaultSeriesIDSetCacheSize is the default number of series ID sets to cache in the TSI index.
	DefaultSeriesIDSetCacheSize = 100

	// DefaultSeriesFileMaxConcurrentSnapshotCompactions is the maximum number of concurrent series
	// partition snapshot compactions that can run at one time.
	// A value of 0 results in runtime.GOMAXPROCS(0).
	DefaultSeriesFileMaxConcurrentSnapshotCompactions = 0
)
View Source 
const (
	InmemIndexName = "inmem"
	TSI1IndexName  = "tsi1"
)

Available index types.

View Source 
const (
	SeriesIndexVersion = 1
	SeriesIndexMagic   = "SIDX"
)
View Source 
const (
	SeriesIndexElemSize   = 16 // offset + id
	SeriesIndexLoadFactor = 90 // rhh load factor

	SeriesIndexHeaderSize = 0 +
		4 + 1 +
		8 + 8 +
		8 + 8 +
		8 + 8 +
		8 + 8 +
		0
)
View Source 
const (
	SeriesSegmentVersion = 1
	SeriesSegmentMagic   = "SSEG"

	SeriesSegmentHeaderSize = 4 + 1 // magic + version
)
View Source 
const (
	SeriesEntryFlagSize   = 1
	SeriesEntryHeaderSize = 1 + 8 // flag + id

	SeriesEntryInsertFlag    = 0x01
	SeriesEntryTombstoneFlag = 0x02
)

Series entry constants.

View Source 
const DefaultSeriesPartitionCompactThreshold = 1 << 17 // 128K

DefaultSeriesPartitionCompactThreshold is the number of series IDs to hold in the in-memory series map before compacting and rebuilding the on-disk representation.

View Source 
const EOF = query.ZeroTime

EOF represents a "not found" key returned by a Cursor.

View Source 
const SeriesFileDirectory = "_series"

SeriesFileDirectory is the name of the directory containing series files for a database.

View Source 
const (
	// SeriesFilePartitionN is the number of partitions a series file is split into.
	SeriesFilePartitionN = 8
)
View Source 
const SeriesIDSize = 8

SeriesIDSize is the size in bytes of a series key ID.

Variables

View Source 
var (
	// ErrFormatNotFound is returned when no format can be determined from a path.
	ErrFormatNotFound = errors.New("format not found")

	// ErrUnknownEngineFormat is returned when the engine format is
	// unknown. ErrUnknownEngineFormat is currently returned if a format
	// other than tsm1 is encountered.
	ErrUnknownEngineFormat = errors.New("unknown engine format")
)
View Source 
var (
	ErrSeriesFileClosed         = errors.New("tsdb: series file closed")
	ErrInvalidSeriesPartitionID = errors.New("tsdb: invalid series partition id")
)
View Source 
var (
	ErrSeriesPartitionClosed              = errors.New("tsdb: series partition closed")
	ErrSeriesPartitionCompactionCancelled = errors.New("tsdb: series partition compaction cancelled")
)
View Source 
var (
	ErrInvalidSeriesSegment        = errors.New("invalid series segment")
	ErrInvalidSeriesSegmentVersion = errors.New("invalid series segment version")
	ErrSeriesSegmentNotWritable    = errors.New("series segment not writable")
)
View Source 
var (
	// ErrFieldOverflow is returned when too many fields are created on a measurement.
	ErrFieldOverflow = errors.New("field overflow")

	// ErrFieldTypeConflict is returned when a new field already exists with a different type.
	ErrFieldTypeConflict = errors.New("field type conflict")

	// ErrFieldNotFound is returned when a field cannot be found.
	ErrFieldNotFound = errors.New("field not found")

	// ErrFieldUnmappedID is returned when the system is presented, during decode, with a field ID
	// there is no mapping for.
	ErrFieldUnmappedID = errors.New("field ID not mapped")

	// ErrEngineClosed is returned when a caller attempts indirectly to
	// access the shard's underlying engine.
	ErrEngineClosed = errors.New("engine is closed")

	// ErrShardDisabled is returned when a the shard is not available for
	// queries or writes.
	ErrShardDisabled = errors.New("shard is disabled")

	// ErrUnknownFieldsFormat is returned when the fields index file is not identifiable by
	// the file's magic number.
	ErrUnknownFieldsFormat = errors.New("unknown field index format")

	// ErrUnknownFieldType is returned when the type of a field cannot be determined.
	ErrUnknownFieldType = errors.New("unknown field type")

	// ErrShardNotIdle is returned when an operation requring the shard to be idle/cold is
	// attempted on a hot shard.
	ErrShardNotIdle = errors.New("shard not idle")
)
View Source 
var (
	// ErrShardNotFound is returned when trying to get a non existing shard.
	ErrShardNotFound = fmt.Errorf("shard not found")
	// ErrStoreClosed is returned when trying to use a closed Store.
	ErrStoreClosed = fmt.Errorf("store is closed")
	// ErrShardDeletion is returned when trying to create a shard that is being deleted
	ErrShardDeletion = errors.New("shard is being deleted")
	// ErrMultipleIndexTypes is returned when trying to do deletes on a database with
	// multiple index types.
	ErrMultipleIndexTypes = errors.New("cannot delete data. DB contains shards using both inmem and tsi1 indexes. Please convert all shards to use the same index type to delete data.")
)
View Source 
var ErrIndexClosing = errors.New("index is closing")

ErrIndexClosing can be returned to from an Index method if the index is currently closing.

View Source 
var ErrInvalidSeriesIndex = errors.New("invalid series index")
View Source 
var NewInmemIndex func(name string, sfile *SeriesFile) (interface{}, error)

NewInmemIndex returns a new "inmem" index type.

Functions

func AppendSeriesEntry 

func AppendSeriesEntry(dst []byte, flag uint8, id uint64, key []byte) []byte

func AppendSeriesKey 

func AppendSeriesKey(dst []byte, name []byte, tags models.Tags) []byte

AppendSeriesKey serializes name and tags to a byte slice. The total length is prepended as a uvarint.

func CompareSeriesKeys 

func CompareSeriesKeys(a, b []byte) int

func GenerateSeriesKeys 

func GenerateSeriesKeys(names [][]byte, tagsSlice []models.Tags) [][]byte

GenerateSeriesKeys generates series keys for a list of names & tags using a single large memory block.

func IsValidSeriesEntryFlag 

func IsValidSeriesEntryFlag(flag byte) bool

IsValidSeriesEntryFlag returns true if flag is valid.

func IsValidSeriesSegmentFilename 

func IsValidSeriesSegmentFilename(filename string) bool

IsValidSeriesSegmentFilename returns true if filename is a 4-character lowercase hexadecimal number.

func JoinSeriesOffset 

func JoinSeriesOffset(segmentID uint16, pos uint32) int64

JoinSeriesOffset returns an offset that combines the 2-byte segmentID and 4-byte pos.

func MakeTagsKey 

func MakeTagsKey(keys []string, tags models.Tags) []byte

MakeTagsKey converts a tag set to bytes for use as a lookup key.

func MarshalTags 

func MarshalTags(tags map[string]string) []byte

MarshalTags converts a tag set to bytes for use as a lookup key.

func NewFieldKeysIterator 

func NewFieldKeysIterator(sh *Shard, opt query.IteratorOptions) (query.Iterator, error)

NewFieldKeysIterator returns an iterator that can be iterated over to retrieve field keys.

func NewMeasurementSliceIterator 

func NewMeasurementSliceIterator(names [][]byte) *measurementSliceIterator

NewMeasurementSliceIterator returns an iterator over a slice of in-memory measurement names.

func NewSeriesPointIterator 

func NewSeriesPointIterator(indexSet IndexSet, opt query.IteratorOptions) (_ query.Iterator, err error)

newSeriesPointIterator returns a new instance of seriesPointIterator.

func NewSeriesQueryAdapterIterator 

func NewSeriesQueryAdapterIterator(sfile *SeriesFile, itr SeriesIDIterator, fieldset *MeasurementFieldSet, opt query.IteratorOptions) query.Iterator

NewSeriesQueryAdapterIterator returns a new instance of SeriesQueryAdapterIterator.

func NewShardError 

func NewShardError(id uint64, err error) error

NewShardError returns a new ShardError.

func NewTagKeySliceIterator 

func NewTagKeySliceIterator(keys [][]byte) *tagKeySliceIterator

NewTagKeySliceIterator returns a TagKeyIterator that iterates over a slice.

func NewTagKeysIterator 

func NewTagKeysIterator(sh *Shard, opt query.IteratorOptions) (query.Iterator, error)

NewTagKeysIterator returns a new instance of TagKeysIterator.

func NewTagValueSliceIterator 

func NewTagValueSliceIterator(values [][]byte) *tagValueSliceIterator

NewTagValueSliceIterator returns a TagValueIterator that iterates over a slice.

func ParseSeriesKey 

func ParseSeriesKey(data []byte) (name []byte, tags models.Tags)

ParseSeriesKey extracts the name & tags from a series key.

func ParseSeriesKeyInto 

func ParseSeriesKeyInto(data []byte, dstTags models.Tags) ([]byte, models.Tags)

ParseSeriesKeyInto extracts the name and tags for data, parsing the tags into dstTags, which is then returned.

The returned dstTags may have a different length and capacity.

func ParseSeriesSegmentFilename 

func ParseSeriesSegmentFilename(filename string) (uint16, error)

ParseSeriesSegmentFilename returns the id represented by the hexadecimal filename.

func ReadAllSeriesIDIterator 

func ReadAllSeriesIDIterator(itr SeriesIDIterator) ([]uint64, error)

ReadAllSeriesIDIterator returns all ids from the iterator.

func ReadSeriesEntry 

func ReadSeriesEntry(data []byte) (flag uint8, id uint64, key []byte, sz int64)

func ReadSeriesKey 

func ReadSeriesKey(data []byte) (key, remainder []byte)

ReadSeriesKey returns the series key from the beginning of the buffer.

func ReadSeriesKeyFromSegments 

func ReadSeriesKeyFromSegments(a []*SeriesSegment, offset int64) []byte

ReadSeriesKeyFromSegments returns a series key from an offset within a set of segments.

func ReadSeriesKeyLen 

func ReadSeriesKeyLen(data []byte) (sz int, remainder []byte)

func ReadSeriesKeyMeasurement 

func ReadSeriesKeyMeasurement(data []byte) (name, remainder []byte)

func ReadSeriesKeyTag 

func ReadSeriesKeyTag(data []byte) (key, value, remainder []byte)

func ReadSeriesKeyTagN 

func ReadSeriesKeyTagN(data []byte) (n int, remainder []byte)

func RegisterEngine 

func RegisterEngine(name string, fn NewEngineFunc)

RegisterEngine registers a storage engine initializer by name.

func RegisterIndex 

func RegisterIndex(name string, fn NewIndexFunc)

RegisterIndex registers a storage index initializer by name.

func RegisteredEngines 

func RegisteredEngines() []string

RegisteredEngines returns the slice of currently registered engines.

func RegisteredIndexes 

func RegisteredIndexes() []string

RegisteredIndexes returns the slice of currently registered indexes.

func SeriesKeySize 

func SeriesKeySize(name []byte, tags models.Tags) int

SeriesKeySize returns the number of bytes required to encode a series key.

func SeriesKeysSize 

func SeriesKeysSize(names [][]byte, tagsSlice []models.Tags) int

SeriesKeysSize returns the number of bytes required to encode a list of name/tags.

func SeriesSegmentSize 

func SeriesSegmentSize(id uint16) uint32

SeriesSegmentSize returns the maximum size of the segment. The size goes up by powers of 2 starting from 4MB and reaching 256MB.

func SplitSeriesOffset 

func SplitSeriesOffset(offset int64) (segmentID uint16, pos uint32)

SplitSeriesOffset splits a offset into its 2-byte segmentID and 4-byte pos parts.

Types

type BooleanArray 

type BooleanArray = cursors.BooleanArray

func NewBooleanArrayLen 

func NewBooleanArrayLen(sz int) *BooleanArray

type BooleanArrayCursor 

type BooleanArrayCursor = cursors.BooleanArrayCursor

type CompactionPlannerCreator 

type CompactionPlannerCreator func(cfg Config) interface{}

type Config 

type Config struct {
	Dir    string `toml:"dir"`
	Engine string `toml:"-"`
	Index  string `toml:"index-version"`

	// General WAL configuration options
	WALDir string `toml:"wal-dir"`

	// WALFsyncDelay is the amount of time that a write will wait before fsyncing.  A duration
	// greater than 0 can be used to batch up multiple fsync calls.  This is useful for slower
	// disks or when WAL write contention is seen.  A value of 0 fsyncs every write to the WAL.
	WALFsyncDelay toml.Duration `toml:"wal-fsync-delay"`

	// Enables unicode validation on series keys on write.
	ValidateKeys bool `toml:"validate-keys"`

	// Query logging
	QueryLogEnabled bool `toml:"query-log-enabled"`

	// Compaction options for tsm1 (descriptions above with defaults)
	CacheMaxMemorySize             toml.Size     `toml:"cache-max-memory-size"`
	CacheSnapshotMemorySize        toml.Size     `toml:"cache-snapshot-memory-size"`
	CacheSnapshotWriteColdDuration toml.Duration `toml:"cache-snapshot-write-cold-duration"`
	CompactFullWriteColdDuration   toml.Duration `toml:"compact-full-write-cold-duration"`
	CompactThroughput              toml.Size     `toml:"compact-throughput"`
	CompactThroughputBurst         toml.Size     `toml:"compact-throughput-burst"`

	// MaxSeriesPerDatabase is the maximum number of series a node can hold per database.
	// When this limit is exceeded, writes return a 'max series per database exceeded' error.
	// A value of 0 disables the limit. This limit only applies when using the "inmem" index.
	MaxSeriesPerDatabase int `toml:"max-series-per-database"`

	// MaxValuesPerTag is the maximum number of tag values a single tag key can have within
	// a measurement.  When the limit is execeeded, writes return an error.
	// A value of 0 disables the limit.
	MaxValuesPerTag int `toml:"max-values-per-tag"`

	// MaxConcurrentCompactions is the maximum number of concurrent level and full compactions
	// that can be running at one time across all shards.  Compactions scheduled to run when the
	// limit is reached are blocked until a running compaction completes.  Snapshot compactions are
	// not affected by this limit.  A value of 0 limits compactions to runtime.GOMAXPROCS(0).
	MaxConcurrentCompactions int `toml:"max-concurrent-compactions"`

	// MaxIndexLogFileSize is the threshold, in bytes, when an index write-ahead log file will
	// compact into an index file. Lower sizes will cause log files to be compacted more quickly
	// and result in lower heap usage at the expense of write throughput. Higher sizes will
	// be compacted less frequently, store more series in-memory, and provide higher write throughput.
	MaxIndexLogFileSize toml.Size `toml:"max-index-log-file-size"`

	// SeriesIDSetCacheSize is the number items that can be cached within the TSI index. TSI caching can help
	// with query performance when the same tag key/value predicates are commonly used on queries.
	// Setting series-id-set-cache-size to 0 disables the cache.
	SeriesIDSetCacheSize int `toml:"series-id-set-cache-size"`

	// SeriesFileMaxConcurrentSnapshotCompactions is the maximum number of concurrent snapshot compactions
	// that can be running at one time across all series partitions in a database. Snapshots scheduled
	// to run when the limit is reached are blocked until a running snaphsot completes.  Only snapshot
	// compactions are affected by this limit. A value of 0 limits snapshot compactions to the lesser of
	// 8 (series file partition quantity) and runtime.GOMAXPROCS(0).
	SeriesFileMaxConcurrentSnapshotCompactions int `toml:"series-file-max-concurrent-snapshot-compactions"`

	TraceLoggingEnabled bool `toml:"trace-logging-enabled"`

	// TSMWillNeed controls whether we hint to the kernel that we intend to
	// page in mmap'd sections of TSM files. This setting defaults to off, as it has
	// been found to be problematic in some cases. It may help users who have
	// slow disks.
	TSMWillNeed bool `toml:"tsm-use-madv-willneed"`
}

Config holds the configuration for the tsbd package.

func NewConfig 

func NewConfig() Config

NewConfig returns the default configuration for tsdb.

func (Config) Diagnostics 

func (c Config) Diagnostics() (*diagnostics.Diagnostics, error)

Diagnostics returns a diagnostics representation of a subset of the Config.

func (*Config) Validate 

func (c *Config) Validate() error

Validate validates the configuration hold by c.

type Cursor 

type Cursor = cursors.Cursor

type CursorIterator 

type CursorIterator = cursors.CursorIterator

type CursorIterators 

type CursorIterators = cursors.CursorIterators

func CreateCursorIterators 

func CreateCursorIterators(ctx context.Context, shards []*Shard) (CursorIterators, error)

type CursorRequest 

type CursorRequest = cursors.CursorRequest

type CursorStats 

type CursorStats = cursors.CursorStats

type Engine 

type Engine interface {
	Open() error
	Close() error
	SetEnabled(enabled bool)
	SetCompactionsEnabled(enabled bool)
	ScheduleFullCompaction() error

	WithLogger(*zap.Logger)

	LoadMetadataIndex(shardID uint64, index Index) error

	CreateSnapshot() (string, error)
	Backup(w io.Writer, basePath string, since time.Time) error
	Export(w io.Writer, basePath string, start time.Time, end time.Time) error
	Restore(r io.Reader, basePath string) error
	Import(r io.Reader, basePath string) error
	Digest() (io.ReadCloser, int64, error)

	CreateIterator(ctx context.Context, measurement string, opt query.IteratorOptions) (query.Iterator, error)
	CreateCursorIterator(ctx context.Context) (CursorIterator, error)
	IteratorCost(measurement string, opt query.IteratorOptions) (query.IteratorCost, error)
	WritePoints(points []models.Point) error

	CreateSeriesIfNotExists(key, name []byte, tags models.Tags) error
	CreateSeriesListIfNotExists(keys, names [][]byte, tags []models.Tags) error
	DeleteSeriesRange(itr SeriesIterator, min, max int64) error
	DeleteSeriesRangeWithPredicate(itr SeriesIterator, predicate func(name []byte, tags models.Tags) (int64, int64, bool)) error

	MeasurementsSketches() (estimator.Sketch, estimator.Sketch, error)
	SeriesSketches() (estimator.Sketch, estimator.Sketch, error)
	SeriesN() int64

	MeasurementExists(name []byte) (bool, error)

	MeasurementNamesByRegex(re *regexp.Regexp) ([][]byte, error)
	MeasurementFieldSet() *MeasurementFieldSet
	MeasurementFields(measurement []byte) *MeasurementFields
	ForEachMeasurementName(fn func(name []byte) error) error
	DeleteMeasurement(name []byte) error

	HasTagKey(name, key []byte) (bool, error)
	MeasurementTagKeysByExpr(name []byte, expr influxql.Expr) (map[string]struct{}, error)
	TagKeyCardinality(name, key []byte) int

	// Statistics will return statistics relevant to this engine.
	Statistics(tags map[string]string) []models.Statistic
	LastModified() time.Time
	DiskSize() int64
	IsIdle() bool
	Free() error

	io.WriterTo
}

Engine represents a swappable storage engine for the shard.

func NewEngine 

func NewEngine(id uint64, i Index, path string, walPath string, sfile *SeriesFile, options EngineOptions) (Engine, error)

NewEngine returns an instance of an engine based on its format. If the path does not exist then the DefaultFormat is used.

type EngineFormat 

type EngineFormat int

EngineFormat represents the format for an engine. 

const (
	// TSM1Format is the format used by the tsm1 engine.
	TSM1Format EngineFormat = 2
)

type EngineOptions 

type EngineOptions struct {
	EngineVersion string
	IndexVersion  string
	ShardID       uint64
	InmemIndex    interface{} // shared in-memory index

	// Limits the concurrent number of TSM files that can be loaded at once.
	OpenLimiter limiter.Fixed

	// CompactionDisabled specifies shards should not schedule compactions.
	// This option is intended for offline tooling.
	CompactionDisabled          bool
	CompactionPlannerCreator    CompactionPlannerCreator
	CompactionLimiter           limiter.Fixed
	CompactionThroughputLimiter limiter.Rate
	WALEnabled                  bool
	MonitorDisabled             bool

	// DatabaseFilter is a predicate controlling which databases may be opened.
	// If no function is set, all databases will be opened.
	DatabaseFilter func(database string) bool

	// RetentionPolicyFilter is a predicate controlling which combination of database and retention policy may be opened.
	// nil will allow all combinations to pass.
	RetentionPolicyFilter func(database, rp string) bool

	// ShardFilter is a predicate controlling which combination of database, retention policy and shard group may be opened.
	// nil will allow all combinations to pass.
	ShardFilter func(database, rp string, id uint64) bool

	Config         Config
	SeriesIDSets   SeriesIDSets
	FieldValidator FieldValidator

	OnNewEngine func(Engine)

	FileStoreObserver FileStoreObserver
}

EngineOptions represents the options used to initialize the engine.

func NewEngineOptions 

func NewEngineOptions() EngineOptions

NewEngineOptions constructs an EngineOptions object with safe default values. This should only be used in tests; production environments should read from a config file.

type Field 

type Field struct {
	ID   uint8             `json:"id,omitempty"`
	Name string            `json:"name,omitempty"`
	Type influxql.DataType `json:"type,omitempty"`
}

Field represents a series field. All of the fields must be hashable.

type FieldCreate 

type FieldCreate struct {
	Measurement []byte
	Field       *Field
}

FieldCreate holds information for a field to create on a measurement.

type FieldValidator 

type FieldValidator interface {
	Validate(mf *MeasurementFields, point models.Point) error
}

FieldValidator should return a PartialWriteError if the point should not be written.

type FileStoreObserver 

type FileStoreObserver interface {
	// FileFinishing is called before a file is renamed to it's final name.
	FileFinishing(path string) error

	// FileUnlinking is called before a file is unlinked.
	FileUnlinking(path string) error
}

FileStoreObserver is passed notifications before the file store adds or deletes files. In this way, it can be sure to observe every file that is added or removed even in the presence of process death.

type FloatArray 

type FloatArray = cursors.FloatArray

func NewFloatArrayLen 

func NewFloatArrayLen(sz int) *FloatArray

type FloatArrayCursor 

type FloatArrayCursor = cursors.FloatArrayCursor

type Index 

type Index interface {
	Open() error
	Close() error
	WithLogger(*zap.Logger)

	Database() string
	MeasurementExists(name []byte) (bool, error)
	MeasurementNamesByRegex(re *regexp.Regexp) ([][]byte, error)
	DropMeasurement(name []byte) error
	ForEachMeasurementName(fn func(name []byte) error) error

	InitializeSeries(keys, names [][]byte, tags []models.Tags) error
	CreateSeriesIfNotExists(key, name []byte, tags models.Tags) error
	CreateSeriesListIfNotExists(keys, names [][]byte, tags []models.Tags) error
	DropSeries(seriesID uint64, key []byte, cascade bool) error
	DropMeasurementIfSeriesNotExist(name []byte) (bool, error)

	// Used to clean up series in inmem index that were dropped with a shard.
	DropSeriesGlobal(key []byte) error

	MeasurementsSketches() (estimator.Sketch, estimator.Sketch, error)
	SeriesN() int64
	SeriesSketches() (estimator.Sketch, estimator.Sketch, error)
	SeriesIDSet() *SeriesIDSet

	HasTagKey(name, key []byte) (bool, error)
	HasTagValue(name, key, value []byte) (bool, error)

	MeasurementTagKeysByExpr(name []byte, expr influxql.Expr) (map[string]struct{}, error)

	TagKeyCardinality(name, key []byte) int

	// InfluxQL system iterators
	MeasurementIterator() (MeasurementIterator, error)
	TagKeyIterator(name []byte) (TagKeyIterator, error)
	TagValueIterator(name, key []byte) (TagValueIterator, error)
	MeasurementSeriesIDIterator(name []byte) (SeriesIDIterator, error)
	TagKeySeriesIDIterator(name, key []byte) (SeriesIDIterator, error)
	TagValueSeriesIDIterator(name, key, value []byte) (SeriesIDIterator, error)

	// Sets a shared fieldset from the engine.
	FieldSet() *MeasurementFieldSet
	SetFieldSet(fs *MeasurementFieldSet)

	// Size of the index on disk, if applicable.
	DiskSizeBytes() int64

	// Bytes estimates the memory footprint of this Index, in bytes.
	Bytes() int

	// To be removed w/ tsi1.
	SetFieldName(measurement []byte, name string)

	Type() string
	// Returns a unique reference ID to the index instance.
	// For inmem, returns a reference to the backing Index, not ShardIndex.
	UniqueReferenceID() uintptr

	Rebuild()
}

func MustOpenIndex 

func MustOpenIndex(id uint64, database, path string, seriesIDSet *SeriesIDSet, sfile *SeriesFile, options EngineOptions) Index

func NewIndex 

func NewIndex(id uint64, database, path string, seriesIDSet *SeriesIDSet, sfile *SeriesFile, options EngineOptions) (Index, error)

NewIndex returns an instance of an index based on its format. If the path does not exist then the DefaultFormat is used.

type IndexFormat 

type IndexFormat int

IndexFormat represents the format for an index. 

const (
	// InMemFormat is the format used by the original in-memory shared index.
	InMemFormat IndexFormat = 1

	// TSI1Format is the format used by the tsi1 index.
	TSI1Format IndexFormat = 2
)

type IndexSet 

type IndexSet struct {
	Indexes    []Index     // The set of indexes comprising this IndexSet.
	SeriesFile *SeriesFile // The Series File associated with the db for this set.
	// contains filtered or unexported fields
}

IndexSet represents a list of indexes, all belonging to one database.

func (IndexSet) Database 

func (is IndexSet) Database() string

Database returns the database name of the first index.

func (IndexSet) DedupeInmemIndexes 

func (is IndexSet) DedupeInmemIndexes() IndexSet

DedupeInmemIndexes returns an index set which removes duplicate indexes. Useful because inmem indexes are shared by shards per database.

func (IndexSet) ForEachMeasurementTagKey 

func (is IndexSet) ForEachMeasurementTagKey(name []byte, fn func(key []byte) error) error

ForEachMeasurementTagKey iterates over all tag keys in a measurement and applies the provided function.

func (IndexSet) HasField 

func (is IndexSet) HasField(measurement []byte, field string) bool

HasField determines if any of the field sets on the set of indexes in the IndexSet have the provided field for the provided measurement.

func (IndexSet) HasInmemIndex 

func (is IndexSet) HasInmemIndex() bool

HasInmemIndex returns true if any in-memory index is in use.

func (IndexSet) HasTagKey 

func (is IndexSet) HasTagKey(name, key []byte) (bool, error)

HasTagKey returns true if the tag key exists in any index for the provided measurement.

func (IndexSet) HasTagValue 

func (is IndexSet) HasTagValue(name, key, value []byte) (bool, error)

HasTagValue returns true if the tag value exists in any index for the provided measurement and tag key.

func (IndexSet) MatchTagValueSeriesIDIterator 

func (is IndexSet) MatchTagValueSeriesIDIterator(name, key []byte, value *regexp.Regexp, matches bool) (SeriesIDIterator, error)

MatchTagValueSeriesIDIterator returns a series iterator for tags which match value. If matches is false, returns iterators which do not match value.

func (IndexSet) MeasurementIterator 

func (is IndexSet) MeasurementIterator() (MeasurementIterator, error)

MeasurementIterator returns an iterator over all measurements in the index.

func (IndexSet) MeasurementNamesByExpr 

func (is IndexSet) MeasurementNamesByExpr(auth query.Authorizer, expr influxql.Expr) ([][]byte, error)

MeasurementNamesByExpr returns a slice of measurement names matching the provided condition. If no condition is provided then all names are returned.

func (IndexSet) MeasurementNamesByPredicate 

func (is IndexSet) MeasurementNamesByPredicate(auth query.Authorizer, expr influxql.Expr) ([][]byte, error)

MeasurementNamesByPredicate returns a slice of measurement names matching the provided condition. If no condition is provided then all names are returned. This behaves differently from MeasurementNamesByExpr because it will return measurements using flux predicates.

func (IndexSet) MeasurementSeriesByExprIterator 

func (is IndexSet) MeasurementSeriesByExprIterator(name []byte, expr influxql.Expr) (SeriesIDIterator, error)

MeasurementSeriesByExprIterator returns a series iterator for a measurement that is filtered by expr. If expr only contains time expressions then this call is equivalent to MeasurementSeriesIDIterator().

func (IndexSet) MeasurementSeriesIDIterator 

func (is IndexSet) MeasurementSeriesIDIterator(name []byte) (SeriesIDIterator, error)

MeasurementSeriesIDIterator returns an iterator over all non-tombstoned series for the provided measurement.

func (IndexSet) MeasurementSeriesKeysByExpr 

func (is IndexSet) MeasurementSeriesKeysByExpr(name []byte, expr influxql.Expr) ([][]byte, error)

MeasurementSeriesKeysByExpr returns a list of series keys matching expr.

func (IndexSet) MeasurementTagKeyValuesByExpr 

func (is IndexSet) MeasurementTagKeyValuesByExpr(auth query.Authorizer, name []byte, keys []string, expr influxql.Expr, keysSorted bool) ([][]string, error)

MeasurementTagKeyValuesByExpr returns a set of tag values filtered by an expression.

func (IndexSet) MeasurementTagKeysByExpr 

func (is IndexSet) MeasurementTagKeysByExpr(name []byte, expr influxql.Expr) (map[string]struct{}, error)

MeasurementTagKeysByExpr extracts the tag keys wanted by the expression.

func (IndexSet) TagKeyHasAuthorizedSeries 

func (is IndexSet) TagKeyHasAuthorizedSeries(auth query.Authorizer, name, tagKey []byte) (bool, error)

TagKeyHasAuthorizedSeries determines if there exists an authorized series for the provided measurement name and tag key.

func (IndexSet) TagKeyIterator 

func (is IndexSet) TagKeyIterator(name []byte) (TagKeyIterator, error)

TagKeyIterator returns a key iterator for a measurement.

func (IndexSet) TagKeySeriesIDIterator 

func (is IndexSet) TagKeySeriesIDIterator(name, key []byte) (SeriesIDIterator, error)

TagKeySeriesIDIterator returns a series iterator for all values across a single key.

func (IndexSet) TagSets 

func (is IndexSet) TagSets(sfile *SeriesFile, name []byte, opt query.IteratorOptions) ([]*query.TagSet, error)

TagSets returns an ordered list of tag sets for a measurement by dimension and filtered by an optional conditional expression.

func (IndexSet) TagValueIterator 

func (is IndexSet) TagValueIterator(name, key []byte) (TagValueIterator, error)

TagValueIterator returns a value iterator for a tag key.

func (IndexSet) TagValueSeriesIDIterator 

func (is IndexSet) TagValueSeriesIDIterator(name, key, value []byte) (SeriesIDIterator, error)

TagValueSeriesIDIterator returns a series iterator for a single tag value.

func (IndexSet) TagValuesByKeyAndExpr 

func (is IndexSet) TagValuesByKeyAndExpr(auth query.Authorizer, name []byte, keys []string, expr influxql.Expr, fieldset *MeasurementFieldSet) ([]map[string]struct{}, error)

TagValuesByKeyAndExpr retrieves tag values for the provided tag keys.

TagValuesByKeyAndExpr returns sets of values for each key, indexable by the position of the tag key in the keys argument.

N.B tagValuesByKeyAndExpr relies on keys being sorted in ascending lexicographic order.

type IntegerArray 

type IntegerArray = cursors.IntegerArray

func NewIntegerArrayLen 

func NewIntegerArrayLen(sz int) *IntegerArray

type IntegerArrayCursor 

type IntegerArrayCursor = cursors.IntegerArrayCursor

type KeyValue 

type KeyValue struct {
	Key, Value string
}

KeyValue holds a string key and a string value.

type KeyValues 

type KeyValues []KeyValue

KeyValues is a sortable slice of KeyValue.

func (KeyValues) Len 

func (a KeyValues) Len() int

Len implements sort.Interface.

func (KeyValues) Less 

func (a KeyValues) Less(i, j int) bool

Less implements sort.Interface. Keys are compared before values.

func (KeyValues) Swap 

func (a KeyValues) Swap(i, j int)

Swap implements sort.Interface.

type LimitError 

type LimitError struct {
	Reason string
}

LimitError represents an error caused by a configurable limit.

func (*LimitError) Error 

func (e *LimitError) Error() string

type MeasurementFieldSet 

type MeasurementFieldSet struct {
	// contains filtered or unexported fields
}

MeasurementFieldSet represents a collection of fields by measurement. This safe for concurrent use.

func NewMeasurementFieldSet 

func NewMeasurementFieldSet(path string) (*MeasurementFieldSet, error)

NewMeasurementFieldSet returns a new instance of MeasurementFieldSet.

func (*MeasurementFieldSet) Bytes 

func (fs *MeasurementFieldSet) Bytes() int

Bytes estimates the memory footprint of this MeasurementFieldSet, in bytes.

func (*MeasurementFieldSet) CreateFieldsIfNotExists 

func (fs *MeasurementFieldSet) CreateFieldsIfNotExists(name []byte) *MeasurementFields

CreateFieldsIfNotExists returns fields for a measurement by name.

func (*MeasurementFieldSet) Delete 

func (fs *MeasurementFieldSet) Delete(name string)

Delete removes a field set for a measurement.

func (*MeasurementFieldSet) DeleteWithLock 

func (fs *MeasurementFieldSet) DeleteWithLock(name string, fn func() error) error

DeleteWithLock executes fn and removes a field set from a measurement under lock.

func (*MeasurementFieldSet) Fields 

func (fs *MeasurementFieldSet) Fields(name []byte) *MeasurementFields

Fields returns fields for a measurement by name.

func (*MeasurementFieldSet) FieldsByString 

func (fs *MeasurementFieldSet) FieldsByString(name string) *MeasurementFields

FieldsByString returns fields for a measurment by name.

func (*MeasurementFieldSet) IsEmpty 

func (fs *MeasurementFieldSet) IsEmpty() bool

func (*MeasurementFieldSet) Save 

func (fs *MeasurementFieldSet) Save() error

type MeasurementFields 

type MeasurementFields struct {
	// contains filtered or unexported fields
}

MeasurementFields holds the fields of a measurement and their codec.

func NewMeasurementFields 

func NewMeasurementFields() *MeasurementFields

NewMeasurementFields returns an initialised *MeasurementFields value.

func (*MeasurementFields) CreateFieldIfNotExists 

func (m *MeasurementFields) CreateFieldIfNotExists(name []byte, typ influxql.DataType) error

CreateFieldIfNotExists creates a new field with an autoincrementing ID. Returns an error if 255 fields have already been created on the measurement or the fields already exists with a different type.

func (*MeasurementFields) Field 

func (m *MeasurementFields) Field(name string) *Field

Field returns the field for name, or nil if there is no field for name.

func (*MeasurementFields) FieldBytes 

func (m *MeasurementFields) FieldBytes(name []byte) *Field

FieldBytes returns the field for name, or nil if there is no field for name. FieldBytes should be preferred to Field when the caller has a []byte, because it avoids a string allocation, which can't be avoided if the caller converts the []byte to a string and calls Field.

func (*MeasurementFields) FieldKeys 

func (m *MeasurementFields) FieldKeys() []string

func (*MeasurementFields) FieldN 

func (m *MeasurementFields) FieldN() int

func (*MeasurementFields) FieldSet 

func (m *MeasurementFields) FieldSet() map[string]influxql.DataType

FieldSet returns the set of fields and their types for the measurement.

func (*MeasurementFields) ForEachField 

func (m *MeasurementFields) ForEachField(fn func(name string, typ influxql.DataType) bool)

func (*MeasurementFields) HasField 

func (m *MeasurementFields) HasField(name string) bool

type MeasurementIterator 

type MeasurementIterator interface {
	Close() error
	Next() ([]byte, error)
}

MeasurementIterator represents a iterator over a list of measurements.

func MergeMeasurementIterators 

func MergeMeasurementIterators(itrs ...MeasurementIterator) MeasurementIterator

MergeMeasurementIterators returns an iterator that merges a set of iterators. Iterators that are first in the list take precedence and a deletion by those early iterators will invalidate elements by later iterators.

type MeasurementIterators 

type MeasurementIterators []MeasurementIterator

func (MeasurementIterators) Close 

func (a MeasurementIterators) Close() (err error)

type NewEngineFunc 

type NewEngineFunc func(id uint64, i Index, path string, walPath string, sfile *SeriesFile, options EngineOptions) Engine

NewEngineFunc creates a new engine.

type NewIndexFunc 

type NewIndexFunc func(id uint64, database, path string, seriesIDSet *SeriesIDSet, sfile *SeriesFile, options EngineOptions) Index

NewIndexFunc creates a new index.

type PartialWriteError 

type PartialWriteError struct {
	Reason  string
	Dropped int

	// A sorted slice of series keys that were dropped.
	DroppedKeys [][]byte
}

PartialWriteError indicates a write request could only write a portion of the requested values.

func (PartialWriteError) Error 

func (e PartialWriteError) Error() string

type SeriesCursor 

type SeriesCursor interface {
	Close() error
	Next() (*SeriesCursorRow, error)
}

type SeriesCursorRequest 

type SeriesCursorRequest struct {
	Measurements MeasurementIterator
}

type SeriesCursorRow 

type SeriesCursorRow struct {
	Name []byte
	Tags models.Tags
}

func (*SeriesCursorRow) Compare 

func (r *SeriesCursorRow) Compare(other *SeriesCursorRow) int

type SeriesElem 

type SeriesElem interface {
	Name() []byte
	Tags() models.Tags
	Deleted() bool

	// InfluxQL expression associated with series during filtering.
	Expr() influxql.Expr
}

SeriesElem represents a generic series element.

type SeriesFile 

type SeriesFile struct {
	Logger *zap.Logger
	// contains filtered or unexported fields
}

SeriesFile represents the section of the index that holds series data.

func NewSeriesFile 

func NewSeriesFile(path string) *SeriesFile

NewSeriesFile returns a new instance of SeriesFile.

func (*SeriesFile) Close 

func (f *SeriesFile) Close() (err error)

Close unmaps the data file.

func (*SeriesFile) CreateSeriesListIfNotExists 

func (f *SeriesFile) CreateSeriesListIfNotExists(names [][]byte, tagsSlice []models.Tags) ([]uint64, error)

CreateSeriesListIfNotExists creates a list of series in bulk if they don't exist. The returned ids slice returns IDs for every name+tags, creating new series IDs as needed.

func (*SeriesFile) DeleteSeriesID 

func (f *SeriesFile) DeleteSeriesID(id uint64) error

DeleteSeriesID flags a series as permanently deleted. If the series is reintroduced later then it must create a new id.

func (*SeriesFile) DisableCompactions 

func (f *SeriesFile) DisableCompactions()

DisableCompactions prevents new compactions from running.

func (*SeriesFile) EnableCompactions 

func (f *SeriesFile) EnableCompactions()

EnableCompactions allows compactions to run.

func (*SeriesFile) FileSize 

func (f *SeriesFile) FileSize() (n int64, err error)

FileSize returns the size of all partitions, in bytes.

func (*SeriesFile) HasSeries 

func (f *SeriesFile) HasSeries(name []byte, tags models.Tags, buf []byte) bool

HasSeries return true if the series exists.

func (*SeriesFile) IsDeleted 

func (f *SeriesFile) IsDeleted(id uint64) bool

IsDeleted returns true if the ID has been deleted before.

func (*SeriesFile) Open 

func (f *SeriesFile) Open() error

Open memory maps the data file at the file's path.

func (*SeriesFile) Partitions 

func (f *SeriesFile) Partitions() []*SeriesPartition

Partitions returns all partitions.

func (*SeriesFile) Path 

func (f *SeriesFile) Path() string

Path returns the path to the file.

func (*SeriesFile) Retain 

func (f *SeriesFile) Retain() func()

Retain adds a reference count to the file. It returns a release func.

func (*SeriesFile) Series 

func (f *SeriesFile) Series(id uint64) ([]byte, models.Tags)

Series returns the parsed series name and tags for an offset.

func (*SeriesFile) SeriesCount 

func (f *SeriesFile) SeriesCount() uint64

SeriesCount returns the number of series.

func (*SeriesFile) SeriesID 

func (f *SeriesFile) SeriesID(name []byte, tags models.Tags, buf []byte) uint64

SeriesID return the series id for the series.

func (*SeriesFile) SeriesIDIterator 

func (f *SeriesFile) SeriesIDIterator() SeriesIDIterator

SeriesIterator returns an iterator over all the series.

func (*SeriesFile) SeriesIDPartition 

func (f *SeriesFile) SeriesIDPartition(id uint64) *SeriesPartition

func (*SeriesFile) SeriesIDPartitionID 

func (f *SeriesFile) SeriesIDPartitionID(id uint64) int

func (*SeriesFile) SeriesKey 

func (f *SeriesFile) SeriesKey(id uint64) []byte

SeriesKey returns the series key for a given id.

func (*SeriesFile) SeriesKeyPartition 

func (f *SeriesFile) SeriesKeyPartition(key []byte) *SeriesPartition

func (*SeriesFile) SeriesKeyPartitionID 

func (f *SeriesFile) SeriesKeyPartitionID(key []byte) int

func (*SeriesFile) SeriesKeys 

func (f *SeriesFile) SeriesKeys(ids []uint64) [][]byte

SeriesKeys returns a list of series keys from a list of ids.

func (*SeriesFile) SeriesKeysPartitionIDs 

func (f *SeriesFile) SeriesKeysPartitionIDs(keys [][]byte) []int

func (*SeriesFile) SeriesPartitionPath 

func (f *SeriesFile) SeriesPartitionPath(i int) string

SeriesPartitionPath returns the path to a given partition.

func (*SeriesFile) Wait 

func (f *SeriesFile) Wait()

Wait waits for all Retains to be released.

func (*SeriesFile) WithMaxCompactionConcurrency 

func (f *SeriesFile) WithMaxCompactionConcurrency(maxCompactionConcurrency int)

type SeriesIDElem 

type SeriesIDElem struct {
	SeriesID uint64
	Expr     influxql.Expr
}

SeriesIDElem represents a single series and optional expression.

type SeriesIDElems 

type SeriesIDElems []SeriesIDElem

SeriesIDElems represents a list of series id elements.

func (SeriesIDElems) Len 

func (a SeriesIDElems) Len() int

func (SeriesIDElems) Less 

func (a SeriesIDElems) Less(i, j int) bool

func (SeriesIDElems) Swap 

func (a SeriesIDElems) Swap(i, j int)

type SeriesIDIterator 

type SeriesIDIterator interface {
	Next() (SeriesIDElem, error)
	Close() error
}

SeriesIDIterator represents a iterator over a list of series ids.

func DifferenceSeriesIDIterators 

func DifferenceSeriesIDIterators(itr0, itr1 SeriesIDIterator) SeriesIDIterator

DifferenceSeriesIDIterators returns an iterator that only returns series which occur the first iterator but not the second iterator.

func FilterUndeletedSeriesIDIterator 

func FilterUndeletedSeriesIDIterator(sfile *SeriesFile, itr SeriesIDIterator) SeriesIDIterator

FilterUndeletedSeriesIDIterator returns an iterator which filters all deleted series.

func IntersectSeriesIDIterators 

func IntersectSeriesIDIterators(itr0, itr1 SeriesIDIterator) SeriesIDIterator

IntersectSeriesIDIterators returns an iterator that only returns series which occur in both iterators. If both series have associated expressions then they are combined together.

func MergeSeriesIDIterators 

func MergeSeriesIDIterators(itrs ...SeriesIDIterator) SeriesIDIterator

MergeSeriesIDIterators returns an iterator that merges a set of iterators. Iterators that are first in the list take precedence and a deletion by those early iterators will invalidate elements by later iterators.

func UnionSeriesIDIterators 

func UnionSeriesIDIterators(itr0, itr1 SeriesIDIterator) SeriesIDIterator

UnionSeriesIDIterators returns an iterator that returns series from both both iterators. If both series have associated expressions then they are combined together.

type SeriesIDIterators 

type SeriesIDIterators []SeriesIDIterator

func (SeriesIDIterators) Close 

func (a SeriesIDIterators) Close() (err error)

type SeriesIDSet 

type SeriesIDSet struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

SeriesIDSet represents a lockable bitmap of series ids.

func NewSeriesIDSet 

func NewSeriesIDSet(a ...uint64) *SeriesIDSet

NewSeriesIDSet returns a new instance of SeriesIDSet.

func (*SeriesIDSet) Add 

func (s *SeriesIDSet) Add(id uint64)

Add adds the series id to the set.

func (*SeriesIDSet) AddMany 

func (s *SeriesIDSet) AddMany(ids ...uint64)

AddMany adds multiple ids to the SeriesIDSet. AddMany takes a lock, so may not be optimal to call many times with few ids.

func (*SeriesIDSet) AddNoLock 

func (s *SeriesIDSet) AddNoLock(id uint64)

AddNoLock adds the series id to the set. Add is not safe for use from multiple goroutines. Callers must manage synchronization.

func (*SeriesIDSet) And 

func (s *SeriesIDSet) And(other *SeriesIDSet) *SeriesIDSet

And returns a new SeriesIDSet containing elements that were present in s and other.

func (*SeriesIDSet) AndNot 

func (s *SeriesIDSet) AndNot(other *SeriesIDSet) *SeriesIDSet

AndNot returns a new SeriesIDSet containing elements that were present in s, but not present in other.

func (*SeriesIDSet) Bytes 

func (s *SeriesIDSet) Bytes() int

Bytes estimates the memory footprint of this SeriesIDSet, in bytes.

func (*SeriesIDSet) Cardinality 

func (s *SeriesIDSet) Cardinality() uint64

Cardinality returns the cardinality of the SeriesIDSet.

func (*SeriesIDSet) Clear 

func (s *SeriesIDSet) Clear()

Clear clears the underlying bitmap for re-use. Clear is safe for use by multiple goroutines.

func (*SeriesIDSet) ClearNoLock 

func (s *SeriesIDSet) ClearNoLock()

ClearNoLock clears the underlying bitmap for re-use without taking a lock.

func (*SeriesIDSet) Clone 

func (s *SeriesIDSet) Clone() *SeriesIDSet

Clone returns a new SeriesIDSet with a deep copy of the underlying bitmap.

func (*SeriesIDSet) CloneNoLock 

func (s *SeriesIDSet) CloneNoLock() *SeriesIDSet

CloneNoLock calls Clone without taking a lock.

func (*SeriesIDSet) Contains 

func (s *SeriesIDSet) Contains(id uint64) bool

Contains returns true if the id exists in the set.

func (*SeriesIDSet) ContainsNoLock 

func (s *SeriesIDSet) ContainsNoLock(id uint64) bool

ContainsNoLock returns true if the id exists in the set. ContainsNoLock is not safe for use from multiple goroutines. The caller must manage synchronization.

func (*SeriesIDSet) Diff 

func (s *SeriesIDSet) Diff(other *SeriesIDSet)

Diff removes from s any elements also present in other.

func (*SeriesIDSet) Equals 

func (s *SeriesIDSet) Equals(other *SeriesIDSet) bool

Equals returns true if other and s are the same set of ids.

func (*SeriesIDSet) ForEach 

func (s *SeriesIDSet) ForEach(f func(id uint64))

ForEach calls f for each id in the set. The function is applied to the IDs in ascending order.

func (*SeriesIDSet) ForEachNoLock 

func (s *SeriesIDSet) ForEachNoLock(f func(id uint64))

ForEachNoLock calls f for each id in the set without taking a lock.

func (*SeriesIDSet) Iterator 

func (s *SeriesIDSet) Iterator() SeriesIDSetIterable

Iterator returns an iterator to the underlying bitmap. This iterator is not protected by a lock.

func (*SeriesIDSet) Merge 

func (s *SeriesIDSet) Merge(others ...*SeriesIDSet)

Merge merged the contents of others into s. The caller does not need to provide s as an argument, and the contents of s will always be present in s after Merge returns.

func (*SeriesIDSet) MergeInPlace 

func (s *SeriesIDSet) MergeInPlace(other *SeriesIDSet)

MergeInPlace merges other into s, modifying s in the process.

func (*SeriesIDSet) Remove 

func (s *SeriesIDSet) Remove(id uint64)

Remove removes the id from the set.

func (*SeriesIDSet) RemoveNoLock 

func (s *SeriesIDSet) RemoveNoLock(id uint64)

RemoveNoLock removes the id from the set. RemoveNoLock is not safe for use from multiple goroutines. The caller must manage synchronization.

func (*SeriesIDSet) Slice 

func (s *SeriesIDSet) Slice() []uint64

Slice returns a slice of series ids.

func (*SeriesIDSet) String 

func (s *SeriesIDSet) String() string

func (*SeriesIDSet) UnmarshalBinary 

func (s *SeriesIDSet) UnmarshalBinary(data []byte) error

UnmarshalBinary unmarshals data into the set.

func (*SeriesIDSet) UnmarshalBinaryUnsafe 

func (s *SeriesIDSet) UnmarshalBinaryUnsafe(data []byte) error

UnmarshalBinaryUnsafe unmarshals data into the set. References to the underlying data are used so data should not be reused by caller.

func (*SeriesIDSet) WriteTo 

func (s *SeriesIDSet) WriteTo(w io.Writer) (int64, error)

WriteTo writes the set to w.

type SeriesIDSetIterable 

type SeriesIDSetIterable interface {
	HasNext() bool
	Next() uint32
}

type SeriesIDSetIterator 

type SeriesIDSetIterator interface {
	SeriesIDIterator
	SeriesIDSet() *SeriesIDSet
}

SeriesIDSetIterator represents an iterator that can produce a SeriesIDSet.

func NewSeriesIDSetIterator 

func NewSeriesIDSetIterator(ss *SeriesIDSet) SeriesIDSetIterator

func NewSeriesIDSetIterators 

func NewSeriesIDSetIterators(itrs []SeriesIDIterator) []SeriesIDSetIterator

NewSeriesIDSetIterators returns a slice of SeriesIDSetIterator if all itrs can be type casted. Otherwise returns nil.

type SeriesIDSets 

type SeriesIDSets interface {
	ForEach(f func(ids *SeriesIDSet)) error
}

SeriesIDSets provides access to the total set of series IDs

type SeriesIDSliceIterator 

type SeriesIDSliceIterator struct {
	// contains filtered or unexported fields
}

SeriesIDSliceIterator iterates over a slice of series ids.

func NewSeriesIDSliceIterator 

func NewSeriesIDSliceIterator(ids []uint64) *SeriesIDSliceIterator

NewSeriesIDSliceIterator returns a SeriesIDIterator that iterates over a slice.

func (*SeriesIDSliceIterator) Close 

func (itr *SeriesIDSliceIterator) Close() error

func (*SeriesIDSliceIterator) Next 

func (itr *SeriesIDSliceIterator) Next() (SeriesIDElem, error)

Next returns the next series id in the slice.

func (*SeriesIDSliceIterator) SeriesIDSet 

func (itr *SeriesIDSliceIterator) SeriesIDSet() *SeriesIDSet

SeriesIDSet returns a set of all remaining ids.

type SeriesIndex 

type SeriesIndex struct {
	// contains filtered or unexported fields
}

SeriesIndex represents an index of key-to-id & id-to-offset mappings.

func NewSeriesIndex 

func NewSeriesIndex(path string) *SeriesIndex

func (*SeriesIndex) Clone 

func (idx *SeriesIndex) Clone() *SeriesIndex

Clone returns a copy of idx for use during compaction. In-memory maps are not cloned.

func (*SeriesIndex) Close 

func (idx *SeriesIndex) Close() (err error)

Close unmaps the index file.

func (*SeriesIndex) Count 

func (idx *SeriesIndex) Count() uint64

Count returns the number of series in the index.

func (*SeriesIndex) Delete 

func (idx *SeriesIndex) Delete(id uint64)

Delete marks the series id as deleted.

func (*SeriesIndex) FindIDByNameTags 

func (idx *SeriesIndex) FindIDByNameTags(segments []*SeriesSegment, name []byte, tags models.Tags, buf []byte) uint64

func (*SeriesIndex) FindIDBySeriesKey 

func (idx *SeriesIndex) FindIDBySeriesKey(segments []*SeriesSegment, key []byte) uint64

func (*SeriesIndex) FindIDListByNameTags 

func (idx *SeriesIndex) FindIDListByNameTags(segments []*SeriesSegment, names [][]byte, tagsSlice []models.Tags, buf []byte) (ids []uint64, ok bool)

func (*SeriesIndex) FindOffsetByID 

func (idx *SeriesIndex) FindOffsetByID(id uint64) int64

func (*SeriesIndex) InMemCount 

func (idx *SeriesIndex) InMemCount() uint64

InMemCount returns the number of series in the in-memory index.

func (*SeriesIndex) Insert 

func (idx *SeriesIndex) Insert(key []byte, id uint64, offset int64)

func (*SeriesIndex) IsDeleted 

func (idx *SeriesIndex) IsDeleted(id uint64) bool

IsDeleted returns true if series id has been deleted.

func (*SeriesIndex) OnDiskCount 

func (idx *SeriesIndex) OnDiskCount() uint64

OnDiskCount returns the number of series in the on-disk index.

func (*SeriesIndex) Open 

func (idx *SeriesIndex) Open() (err error)

Open memory-maps the index file.

func (*SeriesIndex) Recover 

func (idx *SeriesIndex) Recover(segments []*SeriesSegment) error

Recover rebuilds the in-memory index for all new entries.

type SeriesIndexHeader 

type SeriesIndexHeader struct {
	Version uint8

	MaxSeriesID uint64
	MaxOffset   int64

	Count    uint64
	Capacity int64

	KeyIDMap struct {
		Offset int64
		Size   int64
	}

	IDOffsetMap struct {
		Offset int64
		Size   int64
	}
}

SeriesIndexHeader represents the header of a series index.

func NewSeriesIndexHeader 

func NewSeriesIndexHeader() SeriesIndexHeader

NewSeriesIndexHeader returns a new instance of SeriesIndexHeader.

func ReadSeriesIndexHeader 

func ReadSeriesIndexHeader(data []byte) (hdr SeriesIndexHeader, err error)

ReadSeriesIndexHeader returns the header from data.

func (*SeriesIndexHeader) WriteTo 

func (hdr *SeriesIndexHeader) WriteTo(w io.Writer) (n int64, err error)

WriteTo writes the header to w.

type SeriesIterator 

type SeriesIterator interface {
	Close() error
	Next() (SeriesElem, error)
}

SeriesIterator represents a iterator over a list of series.

func NewSeriesIteratorAdapter 

func NewSeriesIteratorAdapter(sfile *SeriesFile, itr SeriesIDIterator) SeriesIterator

NewSeriesIteratorAdapter returns an adapter for converting series ids to series.

type SeriesPartition 

type SeriesPartition struct {
	CompactThreshold int

	Logger *zap.Logger
	// contains filtered or unexported fields
}

SeriesPartition represents a subset of series file data.

func NewSeriesPartition 

func NewSeriesPartition(id int, path string, compactionLimiter limiter.Fixed) *SeriesPartition

NewSeriesPartition returns a new instance of SeriesPartition.

func (*SeriesPartition) AppendSeriesIDs 

func (p *SeriesPartition) AppendSeriesIDs(a []uint64) []uint64

AppendSeriesIDs returns a list of all series ids.

func (*SeriesPartition) Close 

func (p *SeriesPartition) Close() (err error)

Close unmaps the data files.

func (*SeriesPartition) Compacting 

func (p *SeriesPartition) Compacting() bool

Compacting returns if the SeriesPartition is currently compacting.

func (*SeriesPartition) CreateSeriesListIfNotExists 

func (p *SeriesPartition) CreateSeriesListIfNotExists(keys [][]byte, keyPartitionIDs []int, ids []uint64) error

CreateSeriesListIfNotExists creates a list of series in bulk if they don't exist. The ids parameter is modified to contain series IDs for all keys belonging to this partition.

func (*SeriesPartition) DeleteSeriesID 

func (p *SeriesPartition) DeleteSeriesID(id uint64) error

DeleteSeriesID flags a series as permanently deleted. If the series is reintroduced later then it must create a new id.

func (*SeriesPartition) DisableCompactions 

func (p *SeriesPartition) DisableCompactions()

func (*SeriesPartition) EnableCompactions 

func (p *SeriesPartition) EnableCompactions()

func (*SeriesPartition) FileSize 

func (p *SeriesPartition) FileSize() (n int64, err error)

FileSize returns the size of all partitions, in bytes.

func (*SeriesPartition) FindIDBySeriesKey 

func (p *SeriesPartition) FindIDBySeriesKey(key []byte) uint64

FindIDBySeriesKey return the series id for the series key.

func (*SeriesPartition) ID 

func (p *SeriesPartition) ID() int

ID returns the partition id.

func (*SeriesPartition) Index 

func (p *SeriesPartition) Index() *SeriesIndex

Index returns the partition's index.

func (*SeriesPartition) IndexPath 

func (p *SeriesPartition) IndexPath() string

IndexPath returns the path to the series index.

func (*SeriesPartition) IsDeleted 

func (p *SeriesPartition) IsDeleted(id uint64) bool

IsDeleted returns true if the ID has been deleted before.

func (*SeriesPartition) Open 

func (p *SeriesPartition) Open() error

Open memory maps the data file at the partition's path.

func (*SeriesPartition) Path 

func (p *SeriesPartition) Path() string

Path returns the path to the partition.

func (*SeriesPartition) Segments 

func (p *SeriesPartition) Segments() []*SeriesSegment

Segments returns a list of partition segments. Used for testing.

func (*SeriesPartition) Series 

func (p *SeriesPartition) Series(id uint64) ([]byte, models.Tags)

Series returns the parsed series name and tags for an offset.

func (*SeriesPartition) SeriesCount 

func (p *SeriesPartition) SeriesCount() uint64

SeriesCount returns the number of series.

func (*SeriesPartition) SeriesKey 

func (p *SeriesPartition) SeriesKey(id uint64) []byte

SeriesKey returns the series key for a given id.

type SeriesPartitionCompactor 

type SeriesPartitionCompactor struct {
	// contains filtered or unexported fields
}

SeriesPartitionCompactor represents an object reindexes a series partition and optionally compacts segments.

func NewSeriesPartitionCompactor 

func NewSeriesPartitionCompactor() *SeriesPartitionCompactor

NewSeriesPartitionCompactor returns a new instance of SeriesPartitionCompactor.

func (*SeriesPartitionCompactor) Compact 

func (c *SeriesPartitionCompactor) Compact(p *SeriesPartition) error

Compact rebuilds the series partition index.

type SeriesSegment 

type SeriesSegment struct {
	// contains filtered or unexported fields
}

SeriesSegment represents a log of series entries.

func CloneSeriesSegments 

func CloneSeriesSegments(a []*SeriesSegment) []*SeriesSegment

CloneSeriesSegments returns a copy of a slice of segments.

func CreateSeriesSegment 

func CreateSeriesSegment(id uint16, path string) (*SeriesSegment, error)

CreateSeriesSegment generates an empty segment at path.

func FindSegment 

func FindSegment(a []*SeriesSegment, id uint16) *SeriesSegment

FindSegment returns a segment by id.

func NewSeriesSegment 

func NewSeriesSegment(id uint16, path string) *SeriesSegment

NewSeriesSegment returns a new instance of SeriesSegment.

func (*SeriesSegment) AppendSeriesIDs 

func (s *SeriesSegment) AppendSeriesIDs(a []uint64) []uint64

AppendSeriesIDs appends all the segments ids to a slice. Returns the new slice.

func (*SeriesSegment) CanWrite 

func (s *SeriesSegment) CanWrite(data []byte) bool

CanWrite returns true if segment has space to write entry data.

func (*SeriesSegment) Clone 

func (s *SeriesSegment) Clone() *SeriesSegment

Clone returns a copy of the segment. Excludes the write handler, if set.

func (*SeriesSegment) Close 

func (s *SeriesSegment) Close() (err error)

Close unmaps the segment.

func (*SeriesSegment) CloseForWrite 

func (s *SeriesSegment) CloseForWrite() (err error)

func (*SeriesSegment) CompactToPath 

func (s *SeriesSegment) CompactToPath(path string, index *SeriesIndex) error

CompactToPath rewrites the segment to a new file and removes tombstoned entries.

func (*SeriesSegment) Data 

func (s *SeriesSegment) Data() []byte

Data returns the raw data.

func (*SeriesSegment) Flush 

func (s *SeriesSegment) Flush() error

Flush flushes the buffer to disk.

func (*SeriesSegment) ForEachEntry 

func (s *SeriesSegment) ForEachEntry(fn func(flag uint8, id uint64, offset int64, key []byte) error) error

ForEachEntry executes fn for every entry in the segment.

func (*SeriesSegment) ID 

func (s *SeriesSegment) ID() uint16

ID returns the id the segment was initialized with.

func (*SeriesSegment) InitForWrite 

func (s *SeriesSegment) InitForWrite() (err error)

InitForWrite initializes a write handle for the segment. This is only used for the last segment in the series file.

func (*SeriesSegment) MaxSeriesID 

func (s *SeriesSegment) MaxSeriesID() uint64

MaxSeriesID returns the highest series id in the segment.

func (*SeriesSegment) Open 

func (s *SeriesSegment) Open() error

Open memory maps the data file at the file's path.

func (*SeriesSegment) Path 

func (s *SeriesSegment) Path() string

Path returns the file path to the segment.

func (*SeriesSegment) Size 

func (s *SeriesSegment) Size() int64

Size returns the size of the data in the segment. This is only populated once InitForWrite() is called.

func (*SeriesSegment) Slice 

func (s *SeriesSegment) Slice(pos uint32) []byte

Slice returns a byte slice starting at pos.

func (*SeriesSegment) WriteLogEntry 

func (s *SeriesSegment) WriteLogEntry(data []byte) (offset int64, err error)

WriteLogEntry writes entry data into the segment. Returns the offset of the beginning of the entry.

type SeriesSegmentHeader 

type SeriesSegmentHeader struct {
	Version uint8
}

SeriesSegmentHeader represents the header of a series segment.

func NewSeriesSegmentHeader 

func NewSeriesSegmentHeader() SeriesSegmentHeader

NewSeriesSegmentHeader returns a new instance of SeriesSegmentHeader.

func ReadSeriesSegmentHeader 

func ReadSeriesSegmentHeader(data []byte) (hdr SeriesSegmentHeader, err error)

ReadSeriesSegmentHeader returns the header from data.

func (*SeriesSegmentHeader) WriteTo 

func (hdr *SeriesSegmentHeader) WriteTo(w io.Writer) (n int64, err error)

WriteTo writes the header to w.

type Shard 

type Shard struct {
	EnableOnOpen bool

	// CompactionDisabled specifies the shard should not schedule compactions.
	// This option is intended for offline tooling.
	CompactionDisabled bool
	// contains filtered or unexported fields
}

Shard represents a self-contained time series database. An inverted index of the measurement and tag data is kept along with the raw time series data. Data can be split across many shards. The query engine in TSDB is responsible for combining the output of many shards into a single query result.

func NewShard 

func NewShard(id uint64, path string, walPath string, sfile *SeriesFile, opt EngineOptions) *Shard

NewShard returns a new initialized Shard. walPath doesn't apply to the b1 type index

func (*Shard) Backup 

func (s *Shard) Backup(w io.Writer, basePath string, since time.Time) error

Backup backs up the shard by creating a tar archive of all TSM files that have been modified since the provided time. See Engine.Backup for more details.

func (*Shard) Close 

func (s *Shard) Close() error

Close shuts down the shard's store.

func (*Shard) CreateCursorIterator 

func (s *Shard) CreateCursorIterator(ctx context.Context) (CursorIterator, error)

func (*Shard) CreateIterator 

func (s *Shard) CreateIterator(ctx context.Context, m *influxql.Measurement, opt query.IteratorOptions) (query.Iterator, error)

CreateIterator returns an iterator for the data in the shard.

func (*Shard) CreateSeriesCursor 

func (s *Shard) CreateSeriesCursor(ctx context.Context, req SeriesCursorRequest, cond influxql.Expr) (SeriesCursor, error)

func (*Shard) CreateSnapshot 

func (s *Shard) CreateSnapshot() (string, error)

CreateSnapshot will return a path to a temp directory containing hard links to the underlying shard files.

func (*Shard) Database 

func (s *Shard) Database() string

Database returns the database of the shard.

func (*Shard) DeleteMeasurement 

func (s *Shard) DeleteMeasurement(name []byte) error

DeleteMeasurement deletes a measurement and all underlying series.

func (*Shard) DeleteSeriesRange 

func (s *Shard) DeleteSeriesRange(itr SeriesIterator, min, max int64) error

DeleteSeriesRange deletes all values from for seriesKeys between min and max (inclusive)

func (*Shard) DeleteSeriesRangeWithPredicate 

func (s *Shard) DeleteSeriesRangeWithPredicate(itr SeriesIterator, predicate func(name []byte, tags models.Tags) (int64, int64, bool)) error

DeleteSeriesRangeWithPredicate deletes all values from for seriesKeys between min and max (inclusive) for which predicate() returns true. If predicate() is nil, then all values in range are deleted.

func (*Shard) Digest 

func (s *Shard) Digest() (io.ReadCloser, int64, error)

Digest returns a digest of the shard.

func (*Shard) DiskSize 

func (s *Shard) DiskSize() (int64, error)

DiskSize returns the size on disk of this shard.

func (*Shard) Engine 

func (s *Shard) Engine() (Engine, error)

engine safely (under an RLock) returns a reference to the shard's Engine, or an error if the Engine is closed, or the shard is currently disabled.

The shard's Engine should always be accessed via a call to engine(), rather than directly referencing Shard.engine.

If a caller needs an Engine reference but is already under a lock, then they should use engineNoLock().

func (*Shard) Export 

func (s *Shard) Export(w io.Writer, basePath string, start time.Time, end time.Time) error

func (*Shard) FieldDimensions 

func (s *Shard) FieldDimensions(measurements []string) (fields map[string]influxql.DataType, dimensions map[string]struct{}, err error)

FieldDimensions returns unique sets of fields and dimensions across a list of sources.

func (*Shard) ForEachMeasurementName 

func (s *Shard) ForEachMeasurementName(fn func(name []byte) error) error

ForEachMeasurementName iterates over each measurement in the shard.

func (*Shard) Free 

func (s *Shard) Free() error

func (*Shard) ID 

func (s *Shard) ID() uint64

ID returns the shards ID.

func (*Shard) Import 

func (s *Shard) Import(r io.Reader, basePath string) error

Import imports data to the underlying engine for the shard. r should be a reader from a backup created by Backup.

func (*Shard) Index 

func (s *Shard) Index() (Index, error)

Index returns a reference to the underlying index. It returns an error if the index is nil.

func (*Shard) IndexType 

func (s *Shard) IndexType() string

IndexType returns the index version being used for this shard.

IndexType returns the empty string if it is called before the shard is opened, since it is only that point that the underlying index type is known.

func (*Shard) IsIdle 

func (s *Shard) IsIdle() bool

IsIdle return true if the shard is not receiving writes and is fully compacted.

func (*Shard) LastModified 

func (s *Shard) LastModified() time.Time

LastModified returns the time when this shard was last modified.

func (*Shard) MeasurementExists 

func (s *Shard) MeasurementExists(name []byte) (bool, error)

MeasurementExists returns true if the shard contains name. TODO(edd): This method is currently only being called from tests; do we really need it?

func (*Shard) MeasurementFields 

func (s *Shard) MeasurementFields(name []byte) *MeasurementFields

MeasurementFields returns fields for a measurement.

func (*Shard) MeasurementNamesByPredicate 

func (s *Shard) MeasurementNamesByPredicate(expr influxql.Expr) ([][]byte, error)

MeasurementNamesByPredicate returns fields for a measurement filtered by an expression.

func (*Shard) MeasurementNamesByRegex 

func (s *Shard) MeasurementNamesByRegex(re *regexp.Regexp) ([][]byte, error)

MeasurementNamesByRegex returns names of measurements matching the regular expression.

func (*Shard) MeasurementTagKeyValuesByExpr 

func (s *Shard) MeasurementTagKeyValuesByExpr(auth query.Authorizer, name []byte, key []string, expr influxql.Expr, keysSorted bool) ([][]string, error)

MeasurementTagKeyValuesByExpr returns all the tag keys values for the provided expression.

func (*Shard) MeasurementTagKeysByExpr 

func (s *Shard) MeasurementTagKeysByExpr(name []byte, expr influxql.Expr) (map[string]struct{}, error)

MeasurementTagKeysByExpr returns all the tag keys for the provided expression.

func (*Shard) MeasurementsSketches 

func (s *Shard) MeasurementsSketches() (estimator.Sketch, estimator.Sketch, error)

MeasurementsSketches returns the measurement sketches for the shard.

func (*Shard) Open 

func (s *Shard) Open() error

Open initializes and opens the shard's store.

func (*Shard) Path 

func (s *Shard) Path() string

Path returns the path set on the shard when it was created.

func (*Shard) Restore 

func (s *Shard) Restore(r io.Reader, basePath string) error

Restore restores data to the underlying engine for the shard. The shard is reopened after restore.

func (*Shard) RetentionPolicy 

func (s *Shard) RetentionPolicy() string

RetentionPolicy returns the retention policy of the shard.

func (*Shard) ScheduleFullCompaction 

func (s *Shard) ScheduleFullCompaction() error

ScheduleFullCompaction forces a full compaction to be schedule on the shard.

func (*Shard) SeriesFile 

func (s *Shard) SeriesFile() (*SeriesFile, error)

SeriesFile returns a reference the underlying series file. If return an error if the series file is nil.

func (*Shard) SeriesN 

func (s *Shard) SeriesN() int64

SeriesN returns the unique number of series in the shard.

func (*Shard) SeriesSketches 

func (s *Shard) SeriesSketches() (estimator.Sketch, estimator.Sketch, error)

SeriesSketches returns the measurement sketches for the shard.

func (*Shard) SetCompactionsEnabled 

func (s *Shard) SetCompactionsEnabled(enabled bool)

SetCompactionsEnabled enables or disable shard background compactions.

func (*Shard) SetEnabled 

func (s *Shard) SetEnabled(enabled bool)

SetEnabled enables the shard for queries and write. When disabled, all writes and queries return an error and compactions are stopped for the shard.

func (*Shard) Statistics 

func (s *Shard) Statistics(tags map[string]string) []models.Statistic

Statistics returns statistics for periodic monitoring.

func (*Shard) TagKeyCardinality 

func (s *Shard) TagKeyCardinality(name, key []byte) int

func (*Shard) WithLogger 

func (s *Shard) WithLogger(log *zap.Logger)

WithLogger sets the logger on the shard. It must be called before Open.

func (*Shard) WritePoints 

func (s *Shard) WritePoints(points []models.Point) error

WritePoints will write the raw data points and any new metadata to the index in the shard.

func (*Shard) WriteTo 

func (s *Shard) WriteTo(w io.Writer) (int64, error)

WriteTo writes the shard's data to w.

type ShardError 

type ShardError struct {
	Err error
	// contains filtered or unexported fields
}

A ShardError implements the error interface, and contains extra context about the shard that generated the error.

func (ShardError) Error 

func (e ShardError) Error() string

Error returns the string representation of the error, to satisfy the error interface.

type ShardGroup 

type ShardGroup interface {
	MeasurementsByRegex(re *regexp.Regexp) []string
	FieldKeysByMeasurement(name []byte) []string
	FieldDimensions(measurements []string) (fields map[string]influxql.DataType, dimensions map[string]struct{}, err error)
	MapType(measurement, field string) influxql.DataType
	CreateIterator(ctx context.Context, measurement *influxql.Measurement, opt query.IteratorOptions) (query.Iterator, error)
	IteratorCost(measurement string, opt query.IteratorOptions) (query.IteratorCost, error)
	ExpandSources(sources influxql.Sources) (influxql.Sources, error)
}

type ShardStatistics 

type ShardStatistics struct {
	WriteReq           int64
	WriteReqOK         int64
	WriteReqErr        int64
	FieldsCreated      int64
	WritePointsErr     int64
	WritePointsDropped int64
	WritePointsOK      int64
	BytesWritten       int64
	DiskBytes          int64
}

ShardStatistics maintains statistics for a shard.

type Shards 

type Shards []*Shard

Shards represents a sortable list of shards.

func (Shards) CallType 

func (a Shards) CallType(name string, args []influxql.DataType) (influxql.DataType, error)

func (Shards) CreateIterator 

func (a Shards) CreateIterator(ctx context.Context, measurement *influxql.Measurement, opt query.IteratorOptions) (query.Iterator, error)

func (Shards) CreateSeriesCursor 

func (a Shards) CreateSeriesCursor(ctx context.Context, req SeriesCursorRequest, cond influxql.Expr) (_ SeriesCursor, err error)

func (Shards) ExpandSources 

func (a Shards) ExpandSources(sources influxql.Sources) (influxql.Sources, error)

func (Shards) FieldDimensions 

func (a Shards) FieldDimensions(measurements []string) (fields map[string]influxql.DataType, dimensions map[string]struct{}, err error)

func (Shards) FieldKeysByMeasurement 

func (a Shards) FieldKeysByMeasurement(name []byte) []string

FieldKeysByMeasurement returns a de-duplicated, sorted, set of field keys for the provided measurement name.

func (Shards) FieldKeysByPredicate 

func (a Shards) FieldKeysByPredicate(expr influxql.Expr) (map[string][]string, error)

FieldKeysByPredicate returns the field keys for series that match the given predicate.

func (Shards) IteratorCost 

func (a Shards) IteratorCost(measurement string, opt query.IteratorOptions) (query.IteratorCost, error)

func (Shards) Len 

func (a Shards) Len() int

Len implements sort.Interface.

func (Shards) Less 

func (a Shards) Less(i, j int) bool

Less implements sort.Interface.

func (Shards) MapType 

func (a Shards) MapType(measurement, field string) influxql.DataType

func (Shards) MeasurementNamesByPredicate 

func (a Shards) MeasurementNamesByPredicate(expr influxql.Expr) ([][]byte, error)

MeasurementNamesByPredicate returns the measurements that match the given predicate.

func (Shards) MeasurementsByRegex 

func (a Shards) MeasurementsByRegex(re *regexp.Regexp) []string

MeasurementsByRegex returns the unique set of measurements matching the provided regex, for all the shards.

func (Shards) Swap 

func (a Shards) Swap(i, j int)

Swap implements sort.Interface.

type Store 

type Store struct {
	SeriesFileMaxSize int64 // Determines size of series file mmap. Can be altered in tests.

	EngineOptions EngineOptions

	Logger *zap.Logger
	// contains filtered or unexported fields
}

Store manages shards and indexes for databases.

func NewStore 

func NewStore(path string) *Store

NewStore returns a new store with the given path and a default configuration. The returned store must be initialized by calling Open before using it.

func (*Store) BackupShard 

func (s *Store) BackupShard(id uint64, since time.Time, w io.Writer) error

BackupShard will get the shard and have the engine backup since the passed in time to the writer.

func (*Store) Close 

func (s *Store) Close() error

Close closes the store and all associated shards. After calling Close accessing shards through the Store will result in ErrStoreClosed being returned.

func (*Store) CreateShard 

func (s *Store) CreateShard(database, retentionPolicy string, shardID uint64, enabled bool) error

CreateShard creates a shard with the given id and retention policy on a database.

func (*Store) CreateShardSnapshot 

func (s *Store) CreateShardSnapshot(id uint64) (string, error)

CreateShardSnapShot will create a hard link to the underlying shard and return a path. The caller is responsible for cleaning up (removing) the file path returned.

func (*Store) Databases 

func (s *Store) Databases() []string

Databases returns the names of all databases managed by the store.

func (*Store) DeleteDatabase 

func (s *Store) DeleteDatabase(name string) error

DeleteDatabase will close all shards associated with a database and remove the directory and files from disk.

func (*Store) DeleteMeasurement 

func (s *Store) DeleteMeasurement(database, name string) error

DeleteMeasurement removes a measurement and all associated series from a database.

func (*Store) DeleteRetentionPolicy 

func (s *Store) DeleteRetentionPolicy(database, name string) error

DeleteRetentionPolicy will close all shards associated with the provided retention policy, remove the retention policy directories on both the DB and WAL, and remove all shard files from disk.

func (*Store) DeleteSeries 

func (s *Store) DeleteSeries(database string, sources []influxql.Source, condition influxql.Expr) error

DeleteSeries loops through the local shards and deletes the series data for the passed in series keys.

func (*Store) DeleteShard 

func (s *Store) DeleteShard(shardID uint64) error

DeleteShard removes a shard from disk.

func (*Store) DiskSize 

func (s *Store) DiskSize() (int64, error)

DiskSize returns the size of all the shard files in bytes. This size does not include the WAL size.

func (*Store) ExpandSources 

func (s *Store) ExpandSources(sources influxql.Sources) (influxql.Sources, error)

ExpandSources expands sources against all local shards.

func (*Store) ExportShard 

func (s *Store) ExportShard(id uint64, start time.Time, end time.Time, w io.Writer) error

func (*Store) ImportShard 

func (s *Store) ImportShard(id uint64, r io.Reader) error

ImportShard imports the contents of r to a given shard. All files in the backup are added as new files which may cause duplicated data to occur requiring more expensive compactions.

func (*Store) IndexBytes 

func (s *Store) IndexBytes() int

func (*Store) MeasurementNames 

func (s *Store) MeasurementNames(auth query.Authorizer, database string, cond influxql.Expr) ([][]byte, error)

MeasurementNames returns a slice of all measurements. Measurements accepts an optional condition expression. If cond is nil, then all measurements for the database will be returned.

func (*Store) MeasurementSeriesCounts 

func (s *Store) MeasurementSeriesCounts(database string) (measuments int, series int)

MeasurementSeriesCounts returns the number of measurements and series in all the shards' indices.

func (*Store) MeasurementsCardinality 

func (s *Store) MeasurementsCardinality(database string) (int64, error)

MeasurementsCardinality returns an estimation of the measurement cardinality for the provided database.

Cardinality is calculated using a sketch-based estimation. The result of this method cannot be combined with any other results.

func (*Store) MeasurementsSketches 

func (s *Store) MeasurementsSketches(database string) (estimator.Sketch, estimator.Sketch, error)

MeasurementsSketches returns the sketches associated with the measurement data in all the shards in the provided database.

The returned sketches can be combined with other sketches to provide an estimation across distributed databases.

func (*Store) Open 

func (s *Store) Open() error

Open initializes the store, creating all necessary directories, loading all shards as well as initializing periodic maintenance of them.

func (*Store) Path 

func (s *Store) Path() string

Path returns the store's root path.

func (*Store) RestoreShard 

func (s *Store) RestoreShard(id uint64, r io.Reader) error

RestoreShard restores a backup from r to a given shard. This will only overwrite files included in the backup.

func (*Store) SeriesCardinality 

func (s *Store) SeriesCardinality(database string) (int64, error)

SeriesCardinality returns the exact series cardinality for the provided database.

Cardinality is calculated exactly by unioning all shards' bitsets of series IDs. The result of this method cannot be combined with any other results.

func (*Store) SeriesSketches 

func (s *Store) SeriesSketches(database string) (estimator.Sketch, estimator.Sketch, error)

SeriesSketches returns the sketches associated with the series data in all the shards in the provided database.

The returned sketches can be combined with other sketches to provide an estimation across distributed databases.

func (*Store) SetShardEnabled 

func (s *Store) SetShardEnabled(shardID uint64, enabled bool) error

SetShardEnabled enables or disables a shard for read and writes.

func (*Store) Shard 

func (s *Store) Shard(id uint64) *Shard

Shard returns a shard by id.

func (*Store) ShardDigest 

func (s *Store) ShardDigest(id uint64) (io.ReadCloser, int64, error)

ShardDigest returns a digest of the shard with the specified ID.

func (*Store) ShardGroup 

func (s *Store) ShardGroup(ids []uint64) ShardGroup

ShardGroup returns a ShardGroup with a list of shards by id.

func (*Store) ShardIDs 

func (s *Store) ShardIDs() []uint64

ShardIDs returns a slice of all ShardIDs under management.

func (*Store) ShardN 

func (s *Store) ShardN() int

ShardN returns the number of shards in the store.

func (*Store) ShardRelativePath 

func (s *Store) ShardRelativePath(id uint64) (string, error)

ShardRelativePath will return the relative path to the shard, i.e., <database>/<retention>/<id>.

func (*Store) Shards 

func (s *Store) Shards(ids []uint64) []*Shard

Shards returns a list of shards by id.

func (*Store) Statistics 

func (s *Store) Statistics(tags map[string]string) []models.Statistic

Statistics returns statistics for period monitoring.

func (*Store) TagKeys 

func (s *Store) TagKeys(auth query.Authorizer, shardIDs []uint64, cond influxql.Expr) ([]TagKeys, error)

TagKeys returns the tag keys in the given database, matching the condition.

func (*Store) TagValues 

func (s *Store) TagValues(auth query.Authorizer, shardIDs []uint64, cond influxql.Expr) ([]TagValues, error)

TagValues returns the tag keys and values for the provided shards, where the tag values satisfy the provided condition.

func (*Store) WithLogger 

func (s *Store) WithLogger(log *zap.Logger)

WithLogger sets the logger for the store.

func (*Store) WriteToShard 

func (s *Store) WriteToShard(shardID uint64, points []models.Point) error

WriteToShard writes a list of points to a shard identified by its ID.

type StringArray 

type StringArray = cursors.StringArray

func NewStringArrayLen 

func NewStringArrayLen(sz int) *StringArray

type StringArrayCursor 

type StringArrayCursor = cursors.StringArrayCursor

type TagKeyIterator 

type TagKeyIterator interface {
	Close() error
	Next() ([]byte, error)
}

TagKeyIterator represents a iterator over a list of tag keys.

func MergeTagKeyIterators 

func MergeTagKeyIterators(itrs ...TagKeyIterator) TagKeyIterator

MergeTagKeyIterators returns an iterator that merges a set of iterators.

type TagKeyIterators 

type TagKeyIterators []TagKeyIterator

func (TagKeyIterators) Close 

func (a TagKeyIterators) Close() (err error)

type TagKeys 

type TagKeys struct {
	Measurement string
	Keys        []string
}

type TagKeysSlice 

type TagKeysSlice []TagKeys

func (TagKeysSlice) Len 

func (a TagKeysSlice) Len() int

func (TagKeysSlice) Less 

func (a TagKeysSlice) Less(i, j int) bool

func (TagKeysSlice) Swap 

func (a TagKeysSlice) Swap(i, j int)

type TagValueIterator 

type TagValueIterator interface {
	Close() error
	Next() ([]byte, error)
}

TagValueIterator represents a iterator over a list of tag values.

func MergeTagValueIterators 

func MergeTagValueIterators(itrs ...TagValueIterator) TagValueIterator

MergeTagValueIterators returns an iterator that merges a set of iterators.

type TagValueIterators 

type TagValueIterators []TagValueIterator

func (TagValueIterators) Close 

func (a TagValueIterators) Close() (err error)

type TagValues 

type TagValues struct {
	Measurement string
	Values      []KeyValue
}

type TagValuesSlice 

type TagValuesSlice []TagValues

func (TagValuesSlice) Len 

func (a TagValuesSlice) Len() int

func (TagValuesSlice) Less 

func (a TagValuesSlice) Less(i, j int) bool

func (TagValuesSlice) Swap 

func (a TagValuesSlice) Swap(i, j int)

type UnsignedArray 

type UnsignedArray = cursors.UnsignedArray

func NewUnsignedArrayLen 

func NewUnsignedArrayLen(sz int) *UnsignedArray

type UnsignedArrayCursor 

type UnsignedArrayCursor = cursors.UnsignedArrayCursor

Source Files

View all Source files

Directories

Path Synopsis
Package engine can be imported to initialize and register all available TSDB engines.
 Package engine can be imported to initialize and register all available TSDB engines.
tsm1
Package tsm1 provides a TSDB in the Time Structured Merge tree format.
 Package tsm1 provides a TSDB in the Time Structured Merge tree format.
inmem
Package inmem implements a shared, in-memory index for each database.
 Package inmem implements a shared, in-memory index for each database.
internal
tsi1
Package tsi1 provides a memory-mapped index implementation that supports high cardinality series.
 Package tsi1 provides a memory-mapped index implementation that supports high cardinality series.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.