tstorage

Documentation

Index

• Variables
• type DataPoint
• type Option
  • func WithDataPath(dataPath string) Option
  • func WithDatabaseMaxSize(databaseMaxSize int64) Option
  • func WithLkoStorage() Option
  • func WithLogLevel(level log.Level) Option
  • func WithMaxPartitions(maxPartitions int64) Option
  • func WithPartitionDuration(duration time.Duration) Option
  • func WithRetention(retention time.Duration) Option
  • func WithTimestampPrecision(precision TimestampPrecision) Option
  • func WithWALBufferedSize(size int) Option
  • func WithWalRecovery(option WalRecoveryOption) Option
  • func WithWriteTimeout(timeout time.Duration) Option
  • func WithoutLkoStorage() Option
• type Reader
• type Row
• type Storage
  • func NewStorage(opts ...Option) (Storage, error)
• type TimestampPrecision
• type WalRecoveryOption

Examples

• NewStorage (WithDataPath)
• NewStorage (WithPartitionDuration)

Variables

var (
	ErrNoDataPoints = errors.New("no data points found")
)

Types

type DataPoint

type DataPoint struct {
	// The actual value. This field must be set.
	Value float64
	// Unix timestamp.
	Timestamp int64
}

DataPoint represents a data point, the smallest unit of time series data.

type Option

type Option func(*storage)

Option is an optional setting for NewStorage.

func WithDataPath

func WithDataPath(dataPath string) Option

WithDataPath specifies the path to directory that stores time-series data. Use this to make time-series data persistent on disk.

Defaults to empty string which means no data will get persisted.

func WithDatabaseMaxSize

func WithDatabaseMaxSize(databaseMaxSize int64) Option

WithDatabaseMaxSize specifies the maximum size the database can have. Higher size leads to deletion of oldest partitions until the database fits within the maximum stablished size.

Defaults to 10MiB.

func WithLkoStorage

func WithLkoStorage() Option

func WithLogLevel

func WithLogLevel(level log.Level) Option

WithLogLevel specifies the log level to emit output.

Defaults to a InfoLevel

func WithMaxPartitions

func WithMaxPartitions(maxPartitions int64) Option

WithMaxPartitions specifies the maximum amount of partitions the database can have. Higher number leads to deletion of oldest partitions, until the database fits within the maximum. Memory partitions are not taken in consideration.

Defaults to 200 partitions. A value of 0 implies the same behaviour as the in memory mode, which means no data will be persisted.

func WithPartitionDuration

func WithPartitionDuration(duration time.Duration) Option

WithPartitionDuration specifies the timestamp range of partitions. Once it exceeds the given time range, the new partition gets inserted.

A partition is a chunk of time-series data with the timestamp range. It acts as a fully independent database containing all data points for its time range.

Defaults to 1h

func WithRetention

func WithRetention(retention time.Duration) Option

WithRetention specifies when to remove old data. Data points will get automatically removed from the disk after a specified period of time after a disk partition was created. Defaults to 14d.

func WithTimestampPrecision

func WithTimestampPrecision(precision TimestampPrecision) Option

WithTimestampPrecision specifies the precision of timestamps to be used by all operations.

Defaults to Nanoseconds

func WithWALBufferedSize

func WithWALBufferedSize(size int) Option

WithWALBufferedSize specifies the buffered byte size before flushing a WAL file. The larger the size, the less frequently the file is written and more write performance at the expense of durability. Giving 0 means it writes to a file whenever data point comes in. Giving -1 disables using WAL.

Defaults to 4096.

func WithWalRecovery

func WithWalRecovery(option WalRecoveryOption) Option

WithWalRecovery sets the recover mode of the wall

func WithWriteTimeout

func WithWriteTimeout(timeout time.Duration) Option

WithWriteTimeout specifies the timeout to wait when workers are busy.

The storage limits the number of concurrent goroutines to prevent from out of memory errors and CPU trashing even if too many goroutines attempt to write.

Defaults to 30s.

func WithoutLkoStorage

func WithoutLkoStorage() Option

WithoutLkoStorage turns off the last known observation (LKO) storage feature. LKO storage is useful when the current or last known value of a metric is required to be known often. When enabled a separate space in memory is allocated to store the last known observation for every metric.

type Reader

type Reader interface {
	// Select gives back a list of data points that matches a set of the given metric and
	// labels within the given start-end range. Keep in mind that start is inclusive, end is exclusive,
	// and both must be Unix timestamp. ErrNoDataPoints will be returned if no data points found.
	Select(metric uint32, start, end int64) (points []*DataPoint, err error)
	Poll(metric uint32) *DataPoint
}

Reader provides reading access to time series data.

type Row

type Row struct {
	// The unique name of metric.
	// This field must be set.
	Metric uint32
	// This field must be set.
	DataPoint
}

Row includes a data point along with properties to identify a kind of metrics.

type Storage

type Storage interface {
	Reader
	// InsertRows ingests the given rows to the time-series storage.
	// If the timestamp is empty, it uses the machine's local timestamp in UTC.
	// The precision of timestamps is nanoseconds by default. It can be changed using WithTimestampPrecision.
	InsertRows(rows []Row) error
	// Close gracefully shutdowns by flushing any unwritten data to the underlying disk partition.
	Close() error
}

Storage provides goroutine safe capabilities of insertion into and retrieval from the time-series storage.

func NewStorage

func NewStorage(opts ...Option) (Storage, error)

NewStorage gives back a new storage, which stores time-series data in the process memory by default.

Give the WithDataPath option for running as a on-disk storage. Specify a directory with data already exists, then it will be read as the initial data.

Example (WithDataPath)

// It will make time-series data persistent under "./data".
storage, err := NewStorage(
	WithDataPath("./data"),
)
if err != nil {
	panic(err)
}
storage.Close()

Example (WithPartitionDuration)

storage, err := NewStorage(
	WithPartitionDuration(30*time.Minute),
	WithTimestampPrecision(Seconds),
)
if err != nil {
	panic(err)
}
defer storage.Close()

type TimestampPrecision

type TimestampPrecision string

TimestampPrecision represents precision of timestamps. See WithTimestampPrecision

const (
	Nanoseconds  TimestampPrecision = "ns"
	Microseconds TimestampPrecision = "us"
	Milliseconds TimestampPrecision = "ms"
	Seconds      TimestampPrecision = "s"
)

type WalRecoveryOption

type WalRecoveryOption int

const (
	TolerateCorruptedTailRecords WalRecoveryOption = iota
	AbsoluteConsistency
	SkipAnyCorruptedRecord
)