whisper

README

Go Whisper

Go Whisper is a Go implementation of the Whisper database, which is part of the Graphite Project.

To create a new whisper database you must define it's retention levels (see: storage schemas), aggregation method and the xFilesFactor. The xFilesFactor specifies the fraction of data points in a propagation interval that must have known values for a propagation to occur.

Examples

Create a new whisper database in "/tmp/test.wsp" with two retention levels (1 second for 1 day and 1 hour for 5 weeks), it will sum values when propagating them to the next retention level, and it requires half the values of the first retention level to be set before they are propagated.

retentions, err := whisper.ParseRetentionDefs("1s:1d,1h:5w")
if err == nil {
  wsp, err := whisper.Create("/tmp/test.wsp", retentions, whisper.Sum, 0.5)
}

Alternatively you can open an existing whisper database.

wsp, err := whisper.Open("/tmp/test.wsp")

Once you have a whisper database you can set values at given time points. This sets the time point 1 hour ago to 12345.678.

wsp.Update(12345.678, time.Now().Add(time.ParseDuration("-1h")).Unix())

And you can retrieve time series from it. This example fetches a time series for the last 1 hour and then iterates through it's points.

series, err := wsp.Fetch(time.Now().Add(time.ParseDuration("-1h")).Unix(), time.Now().Unix())
if err != nil {
  // handle
}
for _, point := range series.Points() {
  fmt.Println(point.Time, point.Value)
}

Thread Safety

This implementation is not thread safe. Writing to a database concurrently will cause bad things to happen. It is up to the user to manage this in their application as they need to.

Licence

Go Whisper is licenced under a BSD Licence.

Documentation

Overview

Package whisper implements Graphite's Whisper database format

Index

• Constants
• type AggregationMethod
• type Options
• type Retention
  • func NewRetention(secondsPerPoint, numberOfPoints int) Retention
  • func ParseRetentionDef(retentionDef string) (*Retention, error)
  • func (retention *Retention) MaxRetention() int
  • func (retention *Retention) NumberOfPoints() int
  • func (retention *Retention) SecondsPerPoint() int
  • func (retention *Retention) Size() int
• type Retentions
  • func ParseRetentionDefs(retentionDefs string) (Retentions, error)
  • func (r Retentions) Len() int
  • func (r Retentions) Swap(i, j int)
• type TimeSeries
  • func (ts *TimeSeries) FromTime() int
  • func (ts *TimeSeries) Points() []TimeSeriesPoint
  • func (ts *TimeSeries) Step() int
  • func (ts *TimeSeries) String() string
  • func (ts *TimeSeries) UntilTime() int
  • func (ts *TimeSeries) Values() []float64
• type TimeSeriesPoint
• type Whisper
  • func Create(path string, retentions Retentions, aggregationMethod AggregationMethod, ...) (whisper *Whisper, err error)
  • func CreateWithOptions(path string, retentions Retentions, aggregationMethod AggregationMethod, ...) (whisper *Whisper, err error)
  • func Open(path string) (whisper *Whisper, err error)
  • func (whisper *Whisper) AggregationMethod() string
  • func (whisper *Whisper) Close()
  • func (whisper *Whisper) Fetch(fromTime, untilTime int) (timeSeries *TimeSeries, err error)
  • func (whisper *Whisper) MaxRetention() int
  • func (whisper *Whisper) MetadataSize() int
  • func (whisper *Whisper) Retentions() []Retention
  • func (whisper *Whisper) Size() int
  • func (whisper *Whisper) StartTime() int
  • func (whisper *Whisper) Update(value float64, timestamp int) (err error)
  • func (whisper *Whisper) UpdateMany(points []*TimeSeriesPoint) (err error)
  • func (whisper *Whisper) XFilesFactor() float32

Constants

const (
	IntSize         = 4
	FloatSize       = 4
	Float64Size     = 8
	PointSize       = 12
	MetadataSize    = 16
	ArchiveInfoSize = 12
)

const (
	Seconds = 1
	Minutes = 60
	Hours   = 3600
	Days    = 86400
	Weeks   = 86400 * 7
	Years   = 86400 * 365
)

Types

type AggregationMethod

type AggregationMethod int

const (
	Average AggregationMethod = iota + 1
	Sum
	Last
	Max
	Min
)

type Options

type Options struct {
	Sparse bool
}

type Retention

type Retention struct {
	// contains filtered or unexported fields
}

A retention level.

Retention levels describe a given archive in the database. How detailed it is and how far back it records.

func NewRetention

func NewRetention(secondsPerPoint, numberOfPoints int) Retention

func ParseRetentionDef

func ParseRetentionDef(retentionDef string) (*Retention, error)

Parse a retention definition as you would find in the storage-schemas.conf of a Carbon install. Note that this only parses a single retention definition, if you have multiple definitions (separated by a comma) you will have to split them yourself.

ParseRetentionDef("10s:14d") Retention{10, 120960}

See: http://graphite.readthedocs.org/en/1.0/config-carbon.html#storage-schemas-conf

func (*Retention) MaxRetention

func (retention *Retention) MaxRetention() int

func (*Retention) NumberOfPoints

func (retention *Retention) NumberOfPoints() int

func (*Retention) SecondsPerPoint

func (retention *Retention) SecondsPerPoint() int

func (*Retention) Size

func (retention *Retention) Size() int

type Retentions

type Retentions []*Retention

func ParseRetentionDefs

func ParseRetentionDefs(retentionDefs string) (Retentions, error)

func (Retentions) Len

func (r Retentions) Len() int

func (Retentions) Swap

func (r Retentions) Swap(i, j int)

type TimeSeries

type TimeSeries struct {
	// contains filtered or unexported fields
}

func (*TimeSeries) FromTime

func (ts *TimeSeries) FromTime() int

func (*TimeSeries) Points

func (ts *TimeSeries) Points() []TimeSeriesPoint

func (*TimeSeries) Step

func (ts *TimeSeries) Step() int

func (*TimeSeries) String

func (ts *TimeSeries) String() string

func (*TimeSeries) UntilTime

func (ts *TimeSeries) UntilTime() int

func (*TimeSeries) Values

func (ts *TimeSeries) Values() []float64

type TimeSeriesPoint

type TimeSeriesPoint struct {
	Time  int
	Value float64
}

type Whisper

type Whisper struct {
	// contains filtered or unexported fields
}

Represents a Whisper database file.

func Create

func Create(path string, retentions Retentions, aggregationMethod AggregationMethod, xFilesFactor float32) (whisper *Whisper, err error)

Create a new Whisper database file and write it's header.

func CreateWithOptions

func CreateWithOptions(path string, retentions Retentions, aggregationMethod AggregationMethod, xFilesFactor float32, options *Options) (whisper *Whisper, err error)

CreateWithOptions is more customizable create function

func Open

func Open(path string) (whisper *Whisper, err error)

Open an existing Whisper database and read it's header

func (*Whisper) AggregationMethod

func (whisper *Whisper) AggregationMethod() string

Return aggregation method

func (*Whisper) Close

func (whisper *Whisper) Close()

Close the whisper file

func (*Whisper) Fetch

func (whisper *Whisper) Fetch(fromTime, untilTime int) (timeSeries *TimeSeries, err error)

Fetch a TimeSeries for a given time span from the file.

func (*Whisper) MaxRetention

func (whisper *Whisper) MaxRetention() int

Return max retention in seconds

func (*Whisper) MetadataSize

func (whisper *Whisper) MetadataSize() int

Calculate the number of bytes the metadata section will be.

func (*Whisper) Retentions

func (whisper *Whisper) Retentions() []Retention

Return retentions

func (*Whisper) Size

func (whisper *Whisper) Size() int

Calculate the total number of bytes the Whisper file should be according to the metadata.

func (*Whisper) StartTime

func (whisper *Whisper) StartTime() int

Calculate the starting time for a whisper db.

func (*Whisper) Update

func (whisper *Whisper) Update(value float64, timestamp int) (err error)

Update a value in the database.

If the timestamp is in the future or outside of the maximum retention it will fail immediately.

func (*Whisper) UpdateMany

func (whisper *Whisper) UpdateMany(points []*TimeSeriesPoint) (err error)

func (*Whisper) XFilesFactor

func (whisper *Whisper) XFilesFactor() float32

Return xFilesFactor