tsdb

package
v1.4.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 30, 2018 License: MIT Imports: 29 Imported by: 0

README

Line Protocol

The line protocol is a text based format for writing points to InfluxDB. Each line defines a single point. Multiple lines must be separated by the newline character \n. The format of the line consists of three parts:

[key] [fields] [timestamp]

Each section is separated by spaces. The minimum required point consists of a measurement name and at least one field. Points without a specified timestamp will be written using the server's local timestamp. Timestamps are assumed to be in nanoseconds unless a precision value is passed in the query string.

Key

The key is the measurement name and any optional tags separated by commas. Measurement names, tag keys, and tag values must escape any spaces or commas using a backslash (\). For example: \ and \,. All tag values are stored as strings and should not be surrounded in quotes.

Tags should be sorted by key before being sent for best performance. The sort should match that from the Go bytes.Compare function (http://golang.org/pkg/bytes/#Compare).

Examples
# measurement only
cpu

# measurement and tags
cpu,host=serverA,region=us-west

# measurement with commas
cpu\,01,host=serverA,region=us-west

# tag value with spaces
cpu,host=server\ A,region=us\ west

Fields

Fields are key-value metrics associated with the measurement. Every line must have at least one field. Multiple fields must be separated with commas and not spaces.

Field keys are always strings and follow the same syntactical rules as described above for tag keys and values. Field values can be one of four types. The first value written for a given field on a given measurement defines the type of that field for all series under that measurement.

  • integer - Numeric values that do not include a decimal and are followed by a trailing i when inserted (e.g. 1i, 345i, 2015i, -10i). Note that all values must have a trailing i. If they do not they will be written as floats.
  • float - Numeric values that are not followed by a trailing i. (e.g. 1, 1.0, -3.14, 6.0+e5, 10).
  • boolean - A value indicating true or false. Valid boolean strings are (t, T, true, TRUE, f, F, false, and FALSE).
  • string - A text value. All string values must be surrounded in double-quotes ". If the string contains a double-quote or backslashes, it must be escaped with a backslash, e.g. \", \\.
# integer value
cpu value=1i

cpu value=1.1i # will result in a parse error

# float value
cpu_load value=1

cpu_load value=1.0

cpu_load value=1.2

# boolean value
error fatal=true

# string value
event msg="logged out"

# multiple values
cpu load=10,alert=true,reason="value above maximum threshold"

Timestamp

The timestamp section is optional but should be specified if possible. The value is an integer representing nanoseconds since the epoch. If the timestamp is not provided the point will inherit the server's local timestamp.

Some write APIs allow passing a lower precision. If the API supports a lower precision, the timestamp may also be an integer epoch in microseconds, milliseconds, seconds, minutes or hours.

Full Example

A full example is shown below.

cpu,host=server01,region=uswest value=1 1434055562000000000
cpu,host=server02,region=uswest value=3 1434055562000010000

In this example the first line shows a measurement of "cpu", there are two tags "host" and "region, the value is 1.0, and the timestamp is 1434055562000000000. Following this is a second line, also a point in the measurement "cpu" but belonging to a different "host".

cpu,host=server\ 01,region=uswest value=1,msg="all systems nominal"
cpu,host=server\ 01,region=us\,west value_int=1i

In these examples, the "host" is set to server 01. The field value associated with field key msg is double-quoted, as it is a string. The second example shows a region of us,west with the comma properly escaped. In the first example value is written as a floating point number. In the second, value_int is an integer.

Distributed Queries

Documentation

Overview

Package tsdb implements a durable time series database.

Index

Constants

View Source 
const (
	// DefaultEngine is the default engine for new shards
	DefaultEngine = "tsm1"

	// DefaultIndex is the default index for new shards
	DefaultIndex = "inmem"

	// DefaultCacheMaxMemorySize is the maximum size a shard's cache can
	// reach before it starts rejecting writes.
	DefaultCacheMaxMemorySize = 1024 * 1024 * 1024 // 1GB

	// DefaultCacheSnapshotMemorySize is the size at which the engine will
	// snapshot the cache and write it to a TSM file, freeing up memory
	DefaultCacheSnapshotMemorySize = 25 * 1024 * 1024 // 25MB

	// DefaultCacheSnapshotWriteColdDuration is the length of time at which
	// the engine will snapshot the cache and write it to a new TSM file if
	// the shard hasn't received writes or deletes
	DefaultCacheSnapshotWriteColdDuration = time.Duration(10 * time.Minute)

	// DefaultCompactFullWriteColdDuration is the duration at which the engine
	// will compact all TSM files in a shard if it hasn't received a write or delete
	DefaultCompactFullWriteColdDuration = time.Duration(4 * time.Hour)

	// DefaultMaxPointsPerBlock is the maximum number of points in an encoded
	// block in a TSM file
	DefaultMaxPointsPerBlock = 1000

	// DefaultMaxSeriesPerDatabase is the maximum number of series a node can hold per database.
	// This limit only applies to the "inmem" index.
	DefaultMaxSeriesPerDatabase = 1000000

	// DefaultMaxValuesPerTag is the maximum number of values a tag can have within a measurement.
	DefaultMaxValuesPerTag = 100000

	// DefaultMaxConcurrentCompactions is the maximum number of concurrent full and level compactions
	// that can run at one time.  A value of 0 results in 50% of runtime.GOMAXPROCS(0) used at runtime.
	DefaultMaxConcurrentCompactions = 0
)
View Source 
const EOF = query.ZeroTime

EOF represents a "not found" key returned by a Cursor.

Variables

View Source 
var (
	// ErrFormatNotFound is returned when no format can be determined from a path.
	ErrFormatNotFound = errors.New("format not found")

	// ErrUnknownEngineFormat is returned when the engine format is
	// unknown. ErrUnknownEngineFormat is currently returned if a format
	// other than tsm1 is encountered.
	ErrUnknownEngineFormat = errors.New("unknown engine format")
)
View Source 
var (
	// ErrFieldOverflow is returned when too many fields are created on a measurement.
	ErrFieldOverflow = errors.New("field overflow")

	// ErrFieldTypeConflict is returned when a new field already exists with a different type.
	ErrFieldTypeConflict = errors.New("field type conflict")

	// ErrFieldNotFound is returned when a field cannot be found.
	ErrFieldNotFound = errors.New("field not found")

	// ErrFieldUnmappedID is returned when the system is presented, during decode, with a field ID
	// there is no mapping for.
	ErrFieldUnmappedID = errors.New("field ID not mapped")

	// ErrEngineClosed is returned when a caller attempts indirectly to
	// access the shard's underlying engine.
	ErrEngineClosed = errors.New("engine is closed")

	// ErrShardDisabled is returned when a the shard is not available for
	// queries or writes.
	ErrShardDisabled = errors.New("shard is disabled")
)
View Source 
var (
	// ErrShardNotFound is returned when trying to get a non existing shard.
	ErrShardNotFound = fmt.Errorf("shard not found")
	// ErrStoreClosed is returned when trying to use a closed Store.
	ErrStoreClosed = fmt.Errorf("store is closed")
)
View Source 
var NewInmemIndex func(name string) (interface{}, error)

NewInmemIndex returns a new "inmem" index type.

Functions

func MakeTagsKey added in v1.3.0 

func MakeTagsKey(keys []string, tags models.Tags) []byte

MakeTagsKey converts a tag set to bytes for use as a lookup key.

func MarshalTags added in v0.9.3 

func MarshalTags(tags map[string]string) []byte

MarshalTags converts a tag set to bytes for use as a lookup key.

func MeasurementFromSeriesKey added in v0.9.3 

func MeasurementFromSeriesKey(key []byte) []byte

MeasurementFromSeriesKey returns the name of the measurement from a key that contains a measurement name.

func NewFieldKeysIterator added in v0.12.0 

func NewFieldKeysIterator(engine Engine, opt query.IteratorOptions) (query.Iterator, error)

NewFieldKeysIterator returns an iterator that can be iterated over to retrieve field keys.

func NewShardError added in v0.12.0 

func NewShardError(id uint64, err error) error

NewShardError returns a new ShardError.

func NewTagKeysIterator added in v0.12.0 

func NewTagKeysIterator(engine Engine, opt query.IteratorOptions) (query.Iterator, error)

NewTagKeysIterator returns a new instance of TagKeysIterator.

func RegisterEngine added in v0.9.3 

func RegisterEngine(name string, fn NewEngineFunc)

RegisterEngine registers a storage engine initializer by name.

func RegisterIndex added in v1.3.0 

func RegisterIndex(name string, fn NewIndexFunc)

RegisterIndex registers a storage index initializer by name.

func RegisteredEngines added in v0.9.5 

func RegisteredEngines() []string

RegisteredEngines returns the slice of currently registered engines.

func RegisteredIndexes added in v1.3.0 

func RegisteredIndexes() []string

RegisteredIndexs returns the slice of currently registered indexes.

Types

type BooleanBatchCursor added in v1.4.1 

type BooleanBatchCursor interface {
	Cursor
	Next() (keys []int64, values []bool)
}

type Config 

type Config struct {
	Dir    string `toml:"dir"`
	Engine string `toml:"-"`
	Index  string `toml:"index-version"`

	// General WAL configuration options
	WALDir string `toml:"wal-dir"`

	// WALFsyncDelay is the amount of time that a write will wait before fsyncing.  A duration
	// greater than 0 can be used to batch up multiple fsync calls.  This is useful for slower
	// disks or when WAL write contention is seen.  A value of 0 fsyncs every write to the WAL.
	WALFsyncDelay toml.Duration `toml:"wal-fsync-delay"`

	// Query logging
	QueryLogEnabled bool `toml:"query-log-enabled"`

	// Compaction options for tsm1 (descriptions above with defaults)
	CacheMaxMemorySize             toml.Size     `toml:"cache-max-memory-size"`
	CacheSnapshotMemorySize        toml.Size     `toml:"cache-snapshot-memory-size"`
	CacheSnapshotWriteColdDuration toml.Duration `toml:"cache-snapshot-write-cold-duration"`
	CompactFullWriteColdDuration   toml.Duration `toml:"compact-full-write-cold-duration"`

	// MaxSeriesPerDatabase is the maximum number of series a node can hold per database.
	// When this limit is exceeded, writes return a 'max series per database exceeded' error.
	// A value of 0 disables the limit. This limit only applies when using the "inmem" index.
	MaxSeriesPerDatabase int `toml:"max-series-per-database"`

	// MaxValuesPerTag is the maximum number of tag values a single tag key can have within
	// a measurement.  When the limit is execeeded, writes return an error.
	// A value of 0 disables the limit.
	MaxValuesPerTag int `toml:"max-values-per-tag"`

	// MaxConcurrentCompactions is the maximum number of concurrent level and full compactions
	// that can be running at one time across all shards.  Compactions scheduled to run when the
	// limit is reached are blocked until a running compaction completes.  Snapshot compactions are
	// not affected by this limit.  A value of 0 limits compactions to runtime.GOMAXPROCS(0).
	MaxConcurrentCompactions int `toml:"max-concurrent-compactions"`

	TraceLoggingEnabled bool `toml:"trace-logging-enabled"`
}

Config holds the configuration for the tsbd package.

func NewConfig 

func NewConfig() Config

NewConfig returns the default configuration for tsdb.

func (Config) Diagnostics added in v1.3.0 

func (c Config) Diagnostics() (*diagnostics.Diagnostics, error)

Diagnostics returns a diagnostics representation of a subset of the Config.

func (*Config) Validate added in v0.9.5 

func (c *Config) Validate() error

Validate validates the configuration hold by c.

type Cursor added in v0.9.3 

type Cursor interface {
	Close()
	SeriesKey() string
	Err() error
}

Cursor represents an iterator over a series.

type CursorRequest added in v1.4.1 

type CursorRequest struct {
	Measurement string
	Series      string
	Field       string
	Ascending   bool
	StartTime   int64
	EndTime     int64
}

type Engine added in v0.9.3 

type Engine interface {
	Open() error
	Close() error
	SetEnabled(enabled bool)
	SetCompactionsEnabled(enabled bool)

	WithLogger(zap.Logger)

	LoadMetadataIndex(shardID uint64, index Index) error

	CreateSnapshot() (string, error)
	Backup(w io.Writer, basePath string, since time.Time) error
	Restore(r io.Reader, basePath string) error
	Import(r io.Reader, basePath string) error

	CreateIterator(ctx context.Context, measurement string, opt query.IteratorOptions) (query.Iterator, error)
	CreateCursor(ctx context.Context, r *CursorRequest) (Cursor, error)
	IteratorCost(measurement string, opt query.IteratorOptions) (query.IteratorCost, error)
	WritePoints(points []models.Point) error

	CreateSeriesIfNotExists(key, name []byte, tags models.Tags) error
	CreateSeriesListIfNotExists(keys, names [][]byte, tags []models.Tags) error
	DeleteSeriesRange(keys [][]byte, min, max int64) error

	SeriesSketches() (estimator.Sketch, estimator.Sketch, error)
	MeasurementsSketches() (estimator.Sketch, estimator.Sketch, error)
	SeriesN() int64

	MeasurementExists(name []byte) (bool, error)
	MeasurementNamesByExpr(auth query.Authorizer, expr influxql.Expr) ([][]byte, error)
	MeasurementNamesByRegex(re *regexp.Regexp) ([][]byte, error)
	MeasurementFields(measurement []byte) *MeasurementFields
	ForEachMeasurementName(fn func(name []byte) error) error
	DeleteMeasurement(name []byte) error

	HasTagKey(name, key []byte) (bool, error)
	MeasurementTagKeysByExpr(name []byte, expr influxql.Expr) (map[string]struct{}, error)
	MeasurementTagKeyValuesByExpr(auth query.Authorizer, name []byte, key []string, expr influxql.Expr, keysSorted bool) ([][]string, error)
	TagKeyHasAuthorizedSeries(auth query.Authorizer, name []byte, key string) bool
	ForEachMeasurementTagKey(name []byte, fn func(key []byte) error) error
	TagKeyCardinality(name, key []byte) int

	// InfluxQL iterators
	MeasurementSeriesKeysByExpr(name []byte, condition influxql.Expr) ([][]byte, error)
	SeriesPointIterator(opt query.IteratorOptions) (query.Iterator, error)

	// Statistics will return statistics relevant to this engine.
	Statistics(tags map[string]string) []models.Statistic
	LastModified() time.Time
	DiskSize() int64
	IsIdle() bool
	Free() error

	io.WriterTo
}

Engine represents a swappable storage engine for the shard.

func NewEngine added in v0.9.3 

func NewEngine(id uint64, i Index, database, path string, walPath string, options EngineOptions) (Engine, error)

NewEngine returns an instance of an engine based on its format. If the path does not exist then the DefaultFormat is used.

type EngineFormat added in v0.9.5 

type EngineFormat int

EngineFormat represents the format for an engine. 

const (
	// TSM1Format is the format used by the tsm1 engine.
	TSM1Format EngineFormat = 2
)

type EngineOptions added in v0.9.3 

type EngineOptions struct {
	EngineVersion string
	IndexVersion  string
	ShardID       uint64
	InmemIndex    interface{} // shared in-memory index

	CompactionLimiter           limiter.Fixed
	CompactionThroughputLimiter limiter.Rate

	Config Config
}

EngineOptions represents the options used to initialize the engine.

func NewEngineOptions added in v0.9.3 

func NewEngineOptions() EngineOptions

NewEngineOptions returns the default options.

type Field added in v0.9.3 

type Field struct {
	ID   uint8             `json:"id,omitempty"`
	Name string            `json:"name,omitempty"`
	Type influxql.DataType `json:"type,omitempty"`
}

Field represents a series field.

type FieldCreate added in v0.9.3 

type FieldCreate struct {
	Measurement []byte
	Field       *Field
}

FieldCreate holds information for a field to create on a measurement.

type FloatBatchCursor added in v1.4.1 

type FloatBatchCursor interface {
	Cursor
	Next() (keys []int64, values []float64)
}

type Index added in v1.3.0 

type Index interface {
	Open() error
	Close() error
	WithLogger(zap.Logger)

	MeasurementExists(name []byte) (bool, error)
	MeasurementNamesByExpr(auth query.Authorizer, expr influxql.Expr) ([][]byte, error)
	MeasurementNamesByRegex(re *regexp.Regexp) ([][]byte, error)
	DropMeasurement(name []byte) error
	ForEachMeasurementName(fn func(name []byte) error) error

	InitializeSeries(key, name []byte, tags models.Tags) error
	CreateSeriesIfNotExists(key, name []byte, tags models.Tags) error
	CreateSeriesListIfNotExists(keys, names [][]byte, tags []models.Tags) error
	DropSeries(key []byte) error

	SeriesSketches() (estimator.Sketch, estimator.Sketch, error)
	MeasurementsSketches() (estimator.Sketch, estimator.Sketch, error)
	SeriesN() int64

	HasTagKey(name, key []byte) (bool, error)
	TagSets(name []byte, options query.IteratorOptions) ([]*query.TagSet, error)
	MeasurementTagKeysByExpr(name []byte, expr influxql.Expr) (map[string]struct{}, error)
	MeasurementTagKeyValuesByExpr(auth query.Authorizer, name []byte, keys []string, expr influxql.Expr, keysSorted bool) ([][]string, error)
	TagKeyHasAuthorizedSeries(auth query.Authorizer, name []byte, key string) bool

	ForEachMeasurementTagKey(name []byte, fn func(key []byte) error) error
	TagKeyCardinality(name, key []byte) int

	// InfluxQL system iterators
	MeasurementSeriesKeysByExpr(name []byte, condition influxql.Expr) ([][]byte, error)
	SeriesPointIterator(opt query.IteratorOptions) (query.Iterator, error)

	// Sets a shared fieldset from the engine.
	SetFieldSet(fs *MeasurementFieldSet)

	// Creates hard links inside path for snapshotting.
	SnapshotTo(path string) error

	// To be removed w/ tsi1.
	SetFieldName(measurement []byte, name string)
	AssignShard(k string, shardID uint64)
	UnassignShard(k string, shardID uint64) error
	RemoveShard(shardID uint64)

	Type() string

	Rebuild()
}

func MustOpenIndex added in v1.3.0 

func MustOpenIndex(id uint64, database, path string, options EngineOptions) Index

func NewIndex added in v1.3.0 

func NewIndex(id uint64, database, path string, options EngineOptions) (Index, error)

NewIndex returns an instance of an index based on its format. If the path does not exist then the DefaultFormat is used.

type IndexFormat added in v1.3.0 

type IndexFormat int

IndexFormat represents the format for an index. 

const (
	// InMemFormat is the format used by the original in-memory shared index.
	InMemFormat IndexFormat = 1

	// TSI1Format is the format used by the tsi1 index.
	TSI1Format IndexFormat = 2
)

type IntegerBatchCursor added in v1.4.1 

type IntegerBatchCursor interface {
	Cursor
	Next() (keys []int64, values []int64)
}

type KeyValue added in v1.0.0 

type KeyValue struct {
	Key, Value string
}

KeyValue holds a string key and a string value.

type KeyValues added in v1.0.0 

type KeyValues []KeyValue

KeyValues is a sortable slice of KeyValue.

func (KeyValues) Len added in v1.0.0 

func (a KeyValues) Len() int

Len implements sort.Interface.

func (KeyValues) Less added in v1.0.0 

func (a KeyValues) Less(i, j int) bool

Less implements sort.Interface. Keys are compared before values.

func (KeyValues) Swap added in v1.0.0 

func (a KeyValues) Swap(i, j int)

Swap implements sort.Interface.

type LimitError added in v1.3.0 

type LimitError struct {
	Reason string
}

LimitError represents an error caused by a configurable limit.

func (*LimitError) Error added in v1.3.0 

func (e *LimitError) Error() string

type MeasurementFieldSet added in v1.3.0 

type MeasurementFieldSet struct {
	// contains filtered or unexported fields
}

MeasurementFieldSet represents a collection of fields by measurement. This safe for concurrent use.

func NewMeasurementFieldSet added in v1.3.0 

func NewMeasurementFieldSet() *MeasurementFieldSet

NewMeasurementFieldSet returns a new instance of MeasurementFieldSet.

func (*MeasurementFieldSet) CreateFieldsIfNotExists added in v1.3.0 

func (fs *MeasurementFieldSet) CreateFieldsIfNotExists(name []byte) *MeasurementFields

CreateFieldsIfNotExists returns fields for a measurement by name.

func (*MeasurementFieldSet) Delete added in v1.3.0 

func (fs *MeasurementFieldSet) Delete(name string)

Delete removes a field set for a measurement.

func (*MeasurementFieldSet) DeleteWithLock added in v1.3.0 

func (fs *MeasurementFieldSet) DeleteWithLock(name string, fn func() error) error

DeleteWithLock executes fn and removes a field set from a measurement under lock.

func (*MeasurementFieldSet) Fields added in v1.3.0 

func (fs *MeasurementFieldSet) Fields(name string) *MeasurementFields

Fields returns fields for a measurement by name.

type MeasurementFields added in v0.9.3 

type MeasurementFields struct {
	// contains filtered or unexported fields
}

MeasurementFields holds the fields of a measurement and their codec.

func NewMeasurementFields added in v1.0.0 

func NewMeasurementFields() *MeasurementFields

NewMeasurementFields returns an initialised *MeasurementFields value.

func (*MeasurementFields) Clone added in v1.3.0 

func (m *MeasurementFields) Clone() *MeasurementFields

Clone returns copy of the MeasurementFields

func (*MeasurementFields) CreateFieldIfNotExists added in v0.9.3 

func (m *MeasurementFields) CreateFieldIfNotExists(name []byte, typ influxql.DataType, limitCount bool) error

CreateFieldIfNotExists creates a new field with an autoincrementing ID. Returns an error if 255 fields have already been created on the measurement or the fields already exists with a different type.

func (*MeasurementFields) Field added in v1.0.0 

func (m *MeasurementFields) Field(name string) *Field

Field returns the field for name, or nil if there is no field for name.

func (*MeasurementFields) FieldBytes added in v1.1.1 

func (m *MeasurementFields) FieldBytes(name []byte) *Field

FieldBytes returns the field for name, or nil if there is no field for name. FieldBytes should be preferred to Field when the caller has a []byte, because it avoids a string allocation, which can't be avoided if the caller converts the []byte to a string and calls Field.

func (*MeasurementFields) FieldKeys added in v1.4.1 

func (m *MeasurementFields) FieldKeys() []string

func (*MeasurementFields) FieldN added in v1.3.0 

func (m *MeasurementFields) FieldN() int

func (*MeasurementFields) FieldSet added in v1.0.0 

func (m *MeasurementFields) FieldSet() map[string]influxql.DataType

FieldSet returns the set of fields and their types for the measurement.

func (*MeasurementFields) HasField added in v1.3.0 

func (m *MeasurementFields) HasField(name string) bool

func (*MeasurementFields) MarshalBinary added in v0.9.3 

func (m *MeasurementFields) MarshalBinary() ([]byte, error)

MarshalBinary encodes the object to a binary format.

func (*MeasurementFields) UnmarshalBinary added in v0.9.3 

func (m *MeasurementFields) UnmarshalBinary(buf []byte) error

UnmarshalBinary decodes the object from a binary format.

type NewEngineFunc added in v0.9.3 

type NewEngineFunc func(id uint64, i Index, database, path string, walPath string, options EngineOptions) Engine

NewEngineFunc creates a new engine.

type NewIndexFunc added in v1.3.0 

type NewIndexFunc func(id uint64, database, path string, options EngineOptions) Index

NewIndexFunc creates a new index.

type PartialWriteError added in v1.1.1 

type PartialWriteError struct {
	Reason  string
	Dropped int

	// The set of series keys that were dropped. Can be nil.
	DroppedKeys map[string]struct{}
}

PartialWriteError indicates a write request could only write a portion of the requested values.

func (PartialWriteError) Error added in v1.1.1 

func (e PartialWriteError) Error() string

type PointBatcher 

type PointBatcher struct {
	// contains filtered or unexported fields
}

PointBatcher accepts Points and will emit a batch of those points when either a) the batch reaches a certain size, or b) a certain time passes.

func NewPointBatcher 

func NewPointBatcher(sz int, bp int, d time.Duration) *PointBatcher

NewPointBatcher returns a new PointBatcher. sz is the batching size, bp is the maximum number of batches that may be pending. d is the time after which a batch will be emitted after the first point is received for the batch, regardless of its size.

func (*PointBatcher) Flush 

func (b *PointBatcher) Flush()

Flush instructs the batcher to emit any pending points in a batch, regardless of batch size. If there are no pending points, no batch is emitted.

func (*PointBatcher) In 

func (b *PointBatcher) In() chan<- models.Point

In returns the channel to which points should be written.

func (*PointBatcher) Out 

func (b *PointBatcher) Out() <-chan []models.Point

Out returns the channel from which batches should be read.

func (*PointBatcher) Start 

func (b *PointBatcher) Start()

Start starts the batching process. Returns the in and out channels for points and point-batches respectively.

func (*PointBatcher) Stats 

func (b *PointBatcher) Stats() *PointBatcherStats

Stats returns a PointBatcherStats object for the PointBatcher. While the each statistic should be closely correlated with each other statistic, it is not guaranteed.

func (*PointBatcher) Stop 

func (b *PointBatcher) Stop()

Stop stops the batching process. Stop waits for the batching routine to stop before returning.

type PointBatcherStats 

type PointBatcherStats struct {
	BatchTotal   uint64 // Total count of batches transmitted.
	PointTotal   uint64 // Total count of points processed.
	SizeTotal    uint64 // Number of batches that reached size threshold.
	TimeoutTotal uint64 // Number of timeouts that occurred.
}

PointBatcherStats are the statistics each batcher tracks.

type Shard 

type Shard struct {
	EnableOnOpen bool
	// contains filtered or unexported fields
}

Shard represents a self-contained time series database. An inverted index of the measurement and tag data is kept along with the raw time series data. Data can be split across many shards. The query engine in TSDB is responsible for combining the output of many shards into a single query result.

func NewShard 

func NewShard(id uint64, path string, walPath string, opt EngineOptions) *Shard

NewShard returns a new initialized Shard. walPath doesn't apply to the b1 type index

func (*Shard) Backup added in v1.3.6 

func (s *Shard) Backup(w io.Writer, basePath string, since time.Time) error

Backup backs up the shard by creating a tar archive of all TSM files that have been modified since the provided time. See Engine.Backup for more details.

func (*Shard) Close 

func (s *Shard) Close() error

Close shuts down the shard's store.

func (*Shard) CloseFast added in v1.3.0 

func (s *Shard) CloseFast() error

CloseFast closes the shard without cleaning up the shard ID or any of the shard's series keys from the index it belongs to.

CloseFast can be called when the entire index is being removed, e.g., when the database the shard belongs to is being dropped.

func (*Shard) CreateCursor added in v1.4.1 

func (s *Shard) CreateCursor(ctx context.Context, r *CursorRequest) (Cursor, error)

func (*Shard) CreateIterator added in v0.12.0 

func (s *Shard) CreateIterator(ctx context.Context, m *influxql.Measurement, opt query.IteratorOptions) (query.Iterator, error)

CreateIterator returns an iterator for the data in the shard.

func (*Shard) CreateSnapshot added in v1.0.0 

func (s *Shard) CreateSnapshot() (string, error)

CreateSnapshot will return a path to a temp directory containing hard links to the underlying shard files.

func (*Shard) Database added in v1.3.0 

func (s *Shard) Database() string

Database returns the database of the shard.

func (*Shard) DeleteMeasurement added in v0.9.3 

func (s *Shard) DeleteMeasurement(name []byte) error

DeleteMeasurement deletes a measurement and all underlying series.

func (*Shard) DeleteSeries added in v0.9.3 

func (s *Shard) DeleteSeries(seriesKeys [][]byte) error

DeleteSeries deletes a list of series.

func (*Shard) DeleteSeriesRange added in v1.0.0 

func (s *Shard) DeleteSeriesRange(seriesKeys [][]byte, min, max int64) error

DeleteSeriesRange deletes all values from for seriesKeys between min and max (inclusive)

func (*Shard) DiskSize added in v0.9.5 

func (s *Shard) DiskSize() (int64, error)

DiskSize returns the size on disk of this shard.

func (*Shard) FieldDimensions added in v0.12.0 

func (s *Shard) FieldDimensions(measurements []string) (fields map[string]influxql.DataType, dimensions map[string]struct{}, err error)

FieldDimensions returns unique sets of fields and dimensions across a list of sources.

func (*Shard) ForEachMeasurementName added in v1.3.6 

func (s *Shard) ForEachMeasurementName(fn func(name []byte) error) error

ForEachMeasurementName iterates over each measurement in the shard.

func (*Shard) ForEachMeasurementTagKey added in v1.3.0 

func (s *Shard) ForEachMeasurementTagKey(name []byte, fn func(key []byte) error) error

func (*Shard) Free added in v1.4.1 

func (s *Shard) Free() error

func (*Shard) ID added in v1.3.0 

func (s *Shard) ID() uint64

ID returns the shards ID.

func (*Shard) Import added in v1.3.0 

func (s *Shard) Import(r io.Reader, basePath string) error

Import imports data to the underlying engine for the shard. r should be a reader from a backup created by Backup.

func (*Shard) Index added in v1.4.1 

func (s *Shard) Index() Index

Index returns a reference to the underlying index. This should only be used by utilities and not directly accessed by the database.

func (*Shard) IndexType added in v1.3.0 

func (s *Shard) IndexType() string

func (*Shard) IsIdle added in v1.3.0 

func (s *Shard) IsIdle() bool

IsIdle return true if the shard is not receiving writes and is fully compacted.

func (*Shard) LastModified added in v1.2.3 

func (s *Shard) LastModified() time.Time

LastModified returns the time when this shard was last modified.

func (*Shard) MeasurementExists added in v1.3.0 

func (s *Shard) MeasurementExists(name []byte) (bool, error)

MeasurementExists returns true if the shard contains name. TODO(edd): This method is currently only being called from tests; do we really need it?

func (*Shard) MeasurementFields added in v1.3.0 

func (s *Shard) MeasurementFields(name []byte) *MeasurementFields

MeasurementFields returns fields for a measurement. TODO(edd): This method is currently only being called from tests; do we really need it?

func (*Shard) MeasurementNamesByExpr added in v1.3.0 

func (s *Shard) MeasurementNamesByExpr(auth query.Authorizer, cond influxql.Expr) ([][]byte, error)

MeasurementNamesByExpr returns names of measurements matching the condition. If cond is nil then all measurement names are returned.

func (*Shard) MeasurementNamesByRegex added in v1.3.6 

func (s *Shard) MeasurementNamesByRegex(re *regexp.Regexp) ([][]byte, error)

MeasurementNamesByRegex returns names of measurements matching the regular expression.

func (*Shard) MeasurementSeriesKeysByExpr added in v1.3.6 

func (s *Shard) MeasurementSeriesKeysByExpr(name []byte, expr influxql.Expr) ([][]byte, error)

MeasurementSeriesKeysByExpr returns a list of series keys from the shard matching expr.

func (*Shard) MeasurementTagKeyValuesByExpr added in v1.3.6 

func (s *Shard) MeasurementTagKeyValuesByExpr(auth query.Authorizer, name []byte, key []string, expr influxql.Expr, keysSorted bool) ([][]string, error)

MeasurementTagKeyValuesByExpr returns all the tag keys values for the provided expression.

func (*Shard) MeasurementTagKeysByExpr added in v1.3.6 

func (s *Shard) MeasurementTagKeysByExpr(name []byte, expr influxql.Expr) (map[string]struct{}, error)

MeasurementTagKeysByExpr returns all the tag keys for the provided expression.

func (*Shard) MeasurementsSketches added in v1.3.0 

func (s *Shard) MeasurementsSketches() (estimator.Sketch, estimator.Sketch, error)

MeasurementsSketches returns the measurement sketches for the shard.

func (*Shard) Open 

func (s *Shard) Open() error

Open initializes and opens the shard's store.

func (*Shard) Path 

func (s *Shard) Path() string

Path returns the path set on the shard when it was created.

func (*Shard) Restore added in v1.0.0 

func (s *Shard) Restore(r io.Reader, basePath string) error

Restore restores data to the underlying engine for the shard. The shard is reopened after restore.

func (*Shard) RetentionPolicy added in v1.3.0 

func (s *Shard) RetentionPolicy() string

RetentionPolicy returns the retention policy of the shard.

func (*Shard) SeriesN added in v1.3.0 

func (s *Shard) SeriesN() int64

SeriesN returns the unique number of series in the shard.

func (*Shard) SeriesSketches added in v1.3.0 

func (s *Shard) SeriesSketches() (estimator.Sketch, estimator.Sketch, error)

SeriesSketches returns the series sketches for the shard.

func (*Shard) SetCompactionsEnabled added in v1.3.0 

func (s *Shard) SetCompactionsEnabled(enabled bool)

SetCompactionsEnabled enables or disable shard background compactions.

func (*Shard) SetEnabled added in v1.0.0 

func (s *Shard) SetEnabled(enabled bool)

SetEnabled enables the shard for queries and write. When disabled, all writes and queries return an error and compactions are stopped for the shard.

func (*Shard) Statistics added in v1.0.0 

func (s *Shard) Statistics(tags map[string]string) []models.Statistic

Statistics returns statistics for periodic monitoring.

func (*Shard) TagKeyCardinality added in v1.3.0 

func (s *Shard) TagKeyCardinality(name, key []byte) int

func (*Shard) TagKeyHasAuthorizedSeries added in v1.4.3 

func (s *Shard) TagKeyHasAuthorizedSeries(auth query.Authorizer, name []byte, key string) bool

TagKeyHasAuthorizedSeries determines if there exists an authorised series on the provided measurement with the provided tag key.

func (*Shard) UnloadIndex added in v1.0.2 

func (s *Shard) UnloadIndex()

UnloadIndex removes all references to this shard from the DatabaseIndex

func (*Shard) WithLogger added in v1.2.3 

func (s *Shard) WithLogger(log zap.Logger)

WithLogger sets the logger on the shard.

func (*Shard) WritePoints 

func (s *Shard) WritePoints(points []models.Point) error

WritePoints will write the raw data points and any new metadata to the index in the shard.

func (*Shard) WriteTo added in v0.9.5 

func (s *Shard) WriteTo(w io.Writer) (int64, error)

WriteTo writes the shard's data to w.

type ShardError added in v0.12.0 

type ShardError struct {
	Err error
	// contains filtered or unexported fields
}

A ShardError implements the error interface, and contains extra context about the shard that generated the error.

func (ShardError) Error added in v0.12.0 

func (e ShardError) Error() string

Error returns the string representation of the error, to satisfy the error interface.

type ShardGroup added in v1.2.3 

type ShardGroup interface {
	MeasurementsByRegex(re *regexp.Regexp) []string
	FieldDimensions(measurements []string) (fields map[string]influxql.DataType, dimensions map[string]struct{}, err error)
	MapType(measurement, field string) influxql.DataType
	CreateIterator(ctx context.Context, measurement *influxql.Measurement, opt query.IteratorOptions) (query.Iterator, error)
	IteratorCost(measurement string, opt query.IteratorOptions) (query.IteratorCost, error)
	ExpandSources(sources influxql.Sources) (influxql.Sources, error)
}

type ShardStatistics added in v1.0.0 

type ShardStatistics struct {
	WriteReq           int64
	WriteReqOK         int64
	WriteReqErr        int64
	FieldsCreated      int64
	WritePointsErr     int64
	WritePointsDropped int64
	WritePointsOK      int64
	BytesWritten       int64
	DiskBytes          int64
}

ShardStatistics maintains statistics for a shard.

type Shards added in v0.12.0 

type Shards []*Shard

Shards represents a sortable list of shards.

func (Shards) CreateIterator added in v1.2.3 

func (a Shards) CreateIterator(ctx context.Context, measurement *influxql.Measurement, opt query.IteratorOptions) (query.Iterator, error)

func (Shards) ExpandSources added in v1.2.3 

func (a Shards) ExpandSources(sources influxql.Sources) (influxql.Sources, error)

func (Shards) FieldDimensions added in v1.2.3 

func (a Shards) FieldDimensions(measurements []string) (fields map[string]influxql.DataType, dimensions map[string]struct{}, err error)

func (Shards) IteratorCost added in v1.4.1 

func (a Shards) IteratorCost(measurement string, opt query.IteratorOptions) (query.IteratorCost, error)

func (Shards) Len added in v0.12.0 

func (a Shards) Len() int

Len implements sort.Interface.

func (Shards) Less added in v0.12.0 

func (a Shards) Less(i, j int) bool

Less implements sort.Interface.

func (Shards) MapType added in v1.2.3 

func (a Shards) MapType(measurement, field string) influxql.DataType

func (Shards) MeasurementsByRegex added in v1.2.3 

func (a Shards) MeasurementsByRegex(re *regexp.Regexp) []string

MeasurementsByRegex returns the unique set of measurements matching the provided regex, for all the shards.

func (Shards) Swap added in v0.12.0 

func (a Shards) Swap(i, j int)

Swap implements sort.Interface.

type Store 

type Store struct {
	EngineOptions EngineOptions

	Logger zap.Logger
	// contains filtered or unexported fields
}

Store manages shards and indexes for databases.

func NewStore 

func NewStore(path string) *Store

NewStore returns a new store with the given path and a default configuration. The returned store must be initialized by calling Open before using it.

func (*Store) BackupShard added in v0.10.0 

func (s *Store) BackupShard(id uint64, since time.Time, w io.Writer) error

BackupShard will get the shard and have the engine backup since the passed in time to the writer.

func (*Store) Close 

func (s *Store) Close() error

Close closes the store and all associated shards. After calling Close accessing shards through the Store will result in ErrStoreClosed being returned.

func (*Store) CreateShard 

func (s *Store) CreateShard(database, retentionPolicy string, shardID uint64, enabled bool) error

CreateShard creates a shard with the given id and retention policy on a database.

func (*Store) CreateShardSnapshot added in v1.0.0 

func (s *Store) CreateShardSnapshot(id uint64) (string, error)

CreateShardSnapShot will create a hard link to the underlying shard and return a path. The caller is responsible for cleaning up (removing) the file path returned.

func (*Store) Databases added in v0.9.5 

func (s *Store) Databases() []string

Databases returns the names of all databases managed by the store.

func (*Store) DeleteDatabase 

func (s *Store) DeleteDatabase(name string) error

DeleteDatabase will close all shards associated with a database and remove the directory and files from disk.

func (*Store) DeleteMeasurement added in v0.12.0 

func (s *Store) DeleteMeasurement(database, name string) error

DeleteMeasurement removes a measurement and all associated series from a database.

func (*Store) DeleteRetentionPolicy added in v0.12.0 

func (s *Store) DeleteRetentionPolicy(database, name string) error

DeleteRetentionPolicy will close all shards associated with the provided retention policy, remove the retention policy directories on both the DB and WAL, and remove all shard files from disk.

func (*Store) DeleteSeries added in v0.12.0 

func (s *Store) DeleteSeries(database string, sources []influxql.Source, condition influxql.Expr) error

DeleteSeries loops through the local shards and deletes the series data for the passed in series keys.

func (*Store) DeleteShard 

func (s *Store) DeleteShard(shardID uint64) error

DeleteShard removes a shard from disk.

func (*Store) DiskSize added in v0.9.5 

func (s *Store) DiskSize() (int64, error)

DiskSize returns the size of all the shard files in bytes. This size does not include the WAL size.

func (*Store) ExpandSources added in v0.12.0 

func (s *Store) ExpandSources(sources influxql.Sources) (influxql.Sources, error)

ExpandSources expands sources against all local shards.

func (*Store) ImportShard added in v1.3.0 

func (s *Store) ImportShard(id uint64, r io.Reader) error

ImportShard imports the contents of r to a given shard. All files in the backup are added as new files which may cause duplicated data to occur requiring more expensive compactions.

func (*Store) MeasurementNames added in v1.3.0 

func (s *Store) MeasurementNames(auth query.Authorizer, database string, cond influxql.Expr) ([][]byte, error)

MeasurementNames returns a slice of all measurements. Measurements accepts an optional condition expression. If cond is nil, then all measurements for the database will be returned.

func (*Store) MeasurementSeriesCounts added in v1.3.0 

func (s *Store) MeasurementSeriesCounts(database string) (measuments int, series int)

MeasurementSeriesCounts returns the number of measurements and series in all the shards' indices.

func (*Store) MeasurementsCardinality added in v1.3.0 

func (s *Store) MeasurementsCardinality(database string) (int64, error)

MeasurementsCardinality returns the measurement cardinality for the provided database.

func (*Store) Open 

func (s *Store) Open() error

Open initializes the store, creating all necessary directories, loading all shards as well as initializing periodic maintenance of them.

func (*Store) Path 

func (s *Store) Path() string

Path returns the store's root path.

func (*Store) RestoreShard added in v1.0.0 

func (s *Store) RestoreShard(id uint64, r io.Reader) error

RestoreShard restores a backup from r to a given shard. This will only overwrite files included in the backup.

func (*Store) SeriesCardinality added in v1.3.0 

func (s *Store) SeriesCardinality(database string) (int64, error)

SeriesCardinality returns the series cardinality for the provided database.

func (*Store) SetShardEnabled added in v1.0.0 

func (s *Store) SetShardEnabled(shardID uint64, enabled bool) error

SetShardEnabled enables or disables a shard for read and writes.

func (*Store) Shard 

func (s *Store) Shard(id uint64) *Shard

Shard returns a shard by id.

func (*Store) ShardGroup added in v1.2.3 

func (s *Store) ShardGroup(ids []uint64) ShardGroup

ShardGroup returns a ShardGroup with a list of shards by id.

func (*Store) ShardIDs 

func (s *Store) ShardIDs() []uint64

ShardIDs returns a slice of all ShardIDs under management.

func (*Store) ShardN added in v0.9.3 

func (s *Store) ShardN() int

ShardN returns the number of shards in the store.

func (*Store) ShardRelativePath added in v0.10.0 

func (s *Store) ShardRelativePath(id uint64) (string, error)

ShardRelativePath will return the relative path to the shard, i.e., <database>/<retention>/<id>.

func (*Store) Shards added in v0.12.0 

func (s *Store) Shards(ids []uint64) []*Shard

Shards returns a list of shards by id.

func (*Store) Statistics added in v1.0.0 

func (s *Store) Statistics(tags map[string]string) []models.Statistic

Statistics returns statistics for period monitoring.

func (*Store) TagKeys added in v1.4.1 

func (s *Store) TagKeys(auth query.Authorizer, shardIDs []uint64, cond influxql.Expr) ([]TagKeys, error)

TagKeys returns the tag keys in the given database, matching the condition.

func (*Store) TagValues added in v1.0.0 

func (s *Store) TagValues(auth query.Authorizer, shardIDs []uint64, cond influxql.Expr) ([]TagValues, error)

TagValues returns the tag keys and values for the provided shards, where the tag values satisfy the provided condition.

func (*Store) WithLogger added in v1.2.3 

func (s *Store) WithLogger(log zap.Logger)

WithLogger sets the logger for the store.

func (*Store) WriteToShard 

func (s *Store) WriteToShard(shardID uint64, points []models.Point) error

WriteToShard writes a list of points to a shard identified by its ID.

type StringBatchCursor added in v1.4.1 

type StringBatchCursor interface {
	Cursor
	Next() (keys []int64, values []string)
}

type TagKeys added in v1.4.1 

type TagKeys struct {
	Measurement string
	Keys        []string
}

type TagKeysSlice added in v1.4.1 

type TagKeysSlice []TagKeys

func (TagKeysSlice) Len added in v1.4.1 

func (a TagKeysSlice) Len() int

func (TagKeysSlice) Less added in v1.4.1 

func (a TagKeysSlice) Less(i, j int) bool

func (TagKeysSlice) Swap added in v1.4.1 

func (a TagKeysSlice) Swap(i, j int)

type TagValues added in v1.0.0 

type TagValues struct {
	Measurement string
	Values      []KeyValue
}

type TagValuesSlice added in v1.3.0 

type TagValuesSlice []TagValues

func (TagValuesSlice) Len added in v1.3.0 

func (a TagValuesSlice) Len() int

func (TagValuesSlice) Less added in v1.3.0 

func (a TagValuesSlice) Less(i, j int) bool

func (TagValuesSlice) Swap added in v1.3.0 

func (a TagValuesSlice) Swap(i, j int)

type UnsignedBatchCursor added in v1.4.1 

type UnsignedBatchCursor interface {
	Cursor
	Next() (keys []int64, values []uint64)
}

Source Files

View all Source files

Directories

Path Synopsis
Package engine can be imported to initialize and register all available TSDB engines.
 Package engine can be imported to initialize and register all available TSDB engines.
tsm1
Package tsm1 provides a TSDB in the Time Structured Merge tree format.
 Package tsm1 provides a TSDB in the Time Structured Merge tree format.
inmem
Package inmem implements a shared, in-memory index for each database.
 Package inmem implements a shared, in-memory index for each database.
internal
tsi1
Package tsi1 provides a memory-mapped index implementation that supports high cardinality series.
 Package tsi1 provides a memory-mapped index implementation that supports high cardinality series.
Package meta is a generated protocol buffer package.
 Package meta is a generated protocol buffer package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.