memorystore

Documentation

Index

• Constants
• Variables
• func ArchiveCheckpoints(checkpointsDir, archiveDir string, from int64, deleteInstead bool) (int, error)
• func Archiving(wg *sync.WaitGroup, ctx context.Context)
• func Checkpointing(wg *sync.WaitGroup, ctx context.Context)
• func Init(metrics map[string]config.MetricConfig)
• func Retention(wg *sync.WaitGroup, ctx context.Context)
• func Shutdown()
• type CheckpointFile
• type CheckpointMetrics
  • func (cm *CheckpointMetrics) MarshalJSON() ([]byte, error)
• type Level
• type MemoryStore
  • func GetMemoryStore() *MemoryStore
  • func (m *MemoryStore) DebugDump(w *bufio.Writer, selector []string) error
  • func (m *MemoryStore) Free(selector []string, t int64) (int, error)
  • func (m *MemoryStore) FreeAll() error
  • func (m *MemoryStore) FromCheckpoint(dir string, from int64) (int, error)
  • func (m *MemoryStore) GetLevel(selector []string) *Level
  • func (m *MemoryStore) ListChildren(selector []string) []string
  • func (m *MemoryStore) Read(selector util.Selector, metric string, from, to, resolution int64) ([]util.Float, int64, int64, int64, error)
  • func (m *MemoryStore) SizeInBytes() int64
  • func (m *MemoryStore) Stats(selector util.Selector, metric string, from, to int64) (*Stats, int64, int64, error)
  • func (m *MemoryStore) ToCheckpoint(dir string, from, to int64) (int, error)
  • func (m *MemoryStore) Write(selector []string, ts int64, metrics []Metric) error
  • func (m *MemoryStore) WriteToLevel(l *Level, selector []string, ts int64, metrics []Metric) error
• type Metric
• type Stats

Constants

const (
	BUFFER_CAP int = 512
)

Default buffer capacity. `buffer.data` will only ever grow up to it's capacity and a new link in the buffer chain will be created if needed so that no copying of data or reallocation needs to happen on writes.

Variables

var (
	ErrNoData           error = errors.New("no data for this metric/level")
	ErrDataDoesNotAlign error = errors.New("data from lower granularities does not align")
)

var ErrNoNewData error = errors.New("all data already archived")

var NumWorkers int = 4

Functions

func ArchiveCheckpoints

func ArchiveCheckpoints(checkpointsDir, archiveDir string, from int64, deleteInstead bool) (int, error)

ZIP all checkpoint files older than `from` together and write them to the `archiveDir`, deleting them from the `checkpointsDir`.

func Archiving

func Archiving(wg *sync.WaitGroup, ctx context.Context)

func Checkpointing

func Checkpointing(wg *sync.WaitGroup, ctx context.Context)

func Init

func Init(metrics map[string]config.MetricConfig)

Create a new, initialized instance of a MemoryStore. Will panic if values in the metric configurations are invalid.

func Retention

func Retention(wg *sync.WaitGroup, ctx context.Context)

func Shutdown

func Shutdown()

Types

type CheckpointFile

type CheckpointFile struct {
	Metrics  map[string]*CheckpointMetrics `json:"metrics"`
	Children map[string]*CheckpointFile    `json:"children"`
	From     int64                         `json:"from"`
	To       int64                         `json:"to"`
}

type CheckpointMetrics

type CheckpointMetrics struct {
	Data      []util.Float `json:"data"`
	Frequency int64        `json:"frequency"`
	Start     int64        `json:"start"`
}

Whenever changed, update MarshalJSON as well!

func (*CheckpointMetrics) MarshalJSON

func (cm *CheckpointMetrics) MarshalJSON() ([]byte, error)

As `Float` implements a custom MarshalJSON() function, serializing an array of such types has more overhead than one would assume (because of extra allocations, interfaces and so on).

type Level

type Level struct {
	// contains filtered or unexported fields
}

Could also be called "node" as this forms a node in a tree structure. Called Level because "node" might be confusing here. Can be both a leaf or a inner node. In this tree structue, inner nodes can also hold data (in `metrics`).

type MemoryStore

type MemoryStore struct {
	Metrics map[string]config.MetricConfig
	// contains filtered or unexported fields
}

func GetMemoryStore

func GetMemoryStore() *MemoryStore

func (*MemoryStore) DebugDump

func (m *MemoryStore) DebugDump(w *bufio.Writer, selector []string) error

func (*MemoryStore) Free

func (m *MemoryStore) Free(selector []string, t int64) (int, error)

Release all buffers for the selected level and all its children that contain only values older than `t`.

func (*MemoryStore) FreeAll

func (m *MemoryStore) FreeAll() error

func (*MemoryStore) FromCheckpoint

func (m *MemoryStore) FromCheckpoint(dir string, from int64) (int, error)

Metrics stored at the lowest 2 levels are not loaded (root and cluster)! This function can only be called once and before the very first write or read. Different host's data is loaded to memory in parallel.

func (*MemoryStore) GetLevel

func (m *MemoryStore) GetLevel(selector []string) *Level

func (*MemoryStore) ListChildren

func (m *MemoryStore) ListChildren(selector []string) []string

Given a selector, return a list of all children of the level selected.

func (*MemoryStore) Read

func (m *MemoryStore) Read(selector util.Selector, metric string, from, to, resolution int64) ([]util.Float, int64, int64, int64, error)

Returns all values for metric `metric` from `from` to `to` for the selected level(s). If the level does not hold the metric itself, the data will be aggregated recursively from the children. The second and third return value are the actual from/to for the data. Those can be different from the range asked for if no data was available.

func (*MemoryStore) SizeInBytes

func (m *MemoryStore) SizeInBytes() int64

func (*MemoryStore) Stats

func (m *MemoryStore) Stats(selector util.Selector, metric string, from, to int64) (*Stats, int64, int64, error)

Returns statistics for the requested metric on the selected node/level. Data is aggregated to the selected level the same way as in `MemoryStore.Read`. If `Stats.Samples` is zero, the statistics should not be considered as valid.

func (*MemoryStore) ToCheckpoint

func (m *MemoryStore) ToCheckpoint(dir string, from, to int64) (int, error)

Metrics stored at the lowest 2 levels are not stored away (root and cluster)! On a per-host basis a new JSON file is created. I have no idea if this will scale. The good thing: Only a host at a time is locked, so this function can run in parallel to writes/reads.

func (*MemoryStore) Write

func (m *MemoryStore) Write(selector []string, ts int64, metrics []Metric) error

Write all values in `metrics` to the level specified by `selector` for time `ts`. Look at `findLevelOrCreate` for how selectors work.

func (*MemoryStore) WriteToLevel

func (m *MemoryStore) WriteToLevel(l *Level, selector []string, ts int64, metrics []Metric) error

Assumes that `minfo` in `metrics` is filled in!

type Metric

type Metric struct {
	Name         string
	Value        util.Float
	MetricConfig config.MetricConfig
}

type Stats

type Stats struct {
	Samples int
	Avg     util.Float
	Min     util.Float
	Max     util.Float
}