btrdb

package module
v4.15.0+incompatible Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 17, 2018 License: GPL-3.0 Imports: 17 Imported by: 4

README

BTrDB

The Berkeley TRee DataBase is a high performance time series database designed to support high density data storage applications.

We are now doing binary and container releases with (mostly) standard semantic versioning. The only variation on this is that we will use odd-numbered minor version numbers to indicate and unstable/development release series. Therefore the meanings of the version numbers are:

  • Major: an increase in major version number indicates that there is no backwards compatibility with existing databases. To upgrade, we recommend using the migration tool.
  • Minor: minor versions are compatible on-disk, but may have an incompatible network API. Therefore while it is safe to upgrade to a new minor version number, you may need to upgrade other programs that connect to BTrDB too. Furthermore, odd-numbered minor version numbers should be considered unstable and for development use only, patch releases within an odd numbered minor version number may not be compatible with eachother.
  • Patch: patch releases on an odd numbered minor version number are not necessarily compatible with eachother in any way. Patch releases on an even minor version number are guaranteed to be compatible both in the disk format and in network API.

While using odd-numbered versions to indicate development releases is a somewhat archaic practice, it allows us to use our production release system for development, which reduces the odds that there is a discrepancy between the well-tested development binaries/containers and the subsequently released production version. Note that we will flag all development releases as "pre-release" on github.

The main distribution of BTrDB v4 is smartgridstore which is basically BTrDB packaged for deployment on Kubernetes, along with some (optional) utilities for working with synchrophasors. You can follow the installation guide on https://docs.smartgrid.store

If you are interested in deploying BTrDB, please file an issue on this repository so we can get an idea for what type of deployments we should focus on supporting. Smartgrid.store is focused on very large city or country-scale data aggregation from smart grid devices, but BTrDB can support other use cases.

Documentation

Index

Constants

View Source 
const LatestGeneration = bstore.LatestGeneration
View Source 
const MaxPQMBufferAge = 8 * time.Hour

TODO this should be more like a few hours Even a few minutes

View Source 
const MaxPQMBufferSize = 32768

This number should be >2000 for decent storage efficiency. If it is too large then recovery of journals can take a long time

View Source 
const MaximumTime = (48 << 56)
View Source 
const MinimumTime = -(16 << 56)

Variables

This section is empty.

Functions

func CreateStatWindows 

func CreateStatWindows(rz []qtree.Record, tCutoffStart int64, tStart int64, tEnd int64, w uint64) []qtree.StatRecord

For all points in rz >= tCutoffStart and < tEnd, align into windows of width w starting from tStart END IS INCLUSIVE, so you need to subtract one for unaligned windows

Types

type ChangedRange 

type ChangedRange struct {
	Start int64
	End   int64
}

type PQM 

type PQM struct {
	// contains filtered or unexported fields
}

func NewPQM 

func NewPQM(si StorageInterface) *PQM

func (*PQM) Flush 

func (pqm *PQM) Flush(ctx context.Context, id uuid.UUID) (maj uint64, min uint64, err bte.BTE)

func (*PQM) GetChangedRanges 

func (pqm *PQM) GetChangedRanges(ctx context.Context, id uuid.UUID, resolution uint8) ([]ChangedRange, bte.BTE, uint64, uint64)

func (*PQM) GetPSHandle 

func (pqm *PQM) GetPSHandle(ctx context.Context) (*psHandle, bte.BTE)

func (*PQM) InitiateShutdown 

func (pqm *PQM) InitiateShutdown() chan struct{}

Flush all open buffers

func (*PQM) Insert 

func (pqm *PQM) Insert(ctx context.Context, id uuid.UUID, r []Record) (major, minor uint64, err bte.BTE)

func (*PQM) MergeNearestValue 

func (pqm *PQM) MergeNearestValue(ctx context.Context, id uuid.UUID, time int64,
	backwards bool, parentRec Record, parentError bte.BTE) (r Record, err bte.BTE, maj uint64, min uint64)

func (*PQM) MergeQueryStatisticalValuesStream 

func (pqm *PQM) MergeQueryStatisticalValuesStream(ctx context.Context, id uuid.UUID, start int64, end int64,
	pointwidth uint8, parentSR chan qtree.StatRecord, parentCE chan bte.BTE) (chan qtree.StatRecord,
	chan bte.BTE, uint64, uint64)

func (*PQM) MergeQueryValuesStream 

func (pqm *PQM) MergeQueryValuesStream(ctx context.Context, id uuid.UUID, start int64, end int64,
	parentCR chan qtree.Record, parentCE chan bte.BTE) (chan qtree.Record, chan bte.BTE, uint64, uint64)

func (*PQM) MergedQueryWindow 

func (pqm *PQM) MergedQueryWindow(ctx context.Context, id uuid.UUID, start int64, end int64,
	width uint64, parentSR chan qtree.StatRecord, parentCE chan bte.BTE) (chan qtree.StatRecord,
	chan bte.BTE, uint64, uint64)

func (*PQM) MuxContents 

func (pqm *PQM) MuxContents(ctx context.Context, id uuid.UUID) (major, minor uint64, contents []Record, err bte.BTE)

func (*PQM) QueryVersion 

func (pqm *PQM) QueryVersion(ctx context.Context, id uuid.UUID) (maj uint64, min uint64, err bte.BTE)

type Quasar 

type Quasar struct {
	// contains filtered or unexported fields
}

func NewQuasar 

func NewQuasar(cfg configprovider.Configuration) (*Quasar, error)

func (*Quasar) CreateStream 

func (q *Quasar) CreateStream(ctx context.Context, uuid []byte, collection string, tags map[string]string, annotations map[string]string) bte.BTE

CreateStream makes a stream with the given uuid, collection and tags. Returns an error if the uuid already exists.

func (*Quasar) DeleteRange 

func (q *Quasar) DeleteRange(ctx context.Context, id uuid.UUID, start int64, end int64) (uint64, uint64, bte.BTE)

func (*Quasar) Flush 

func (q *Quasar) Flush(ctx context.Context, id uuid.UUID) (uint64, uint64, bte.BTE)

func (*Quasar) GetClusterConfiguration 

func (q *Quasar) GetClusterConfiguration() configprovider.ClusterConfiguration

func (*Quasar) GetKeyUsage 

func (q *Quasar) GetKeyUsage(ctx context.Context, prefix string) (tags map[string]int, anns map[string]int, err bte.BTE)

GetMetadataUsage returns the tags and annotations in use for a given collection prefix

func (*Quasar) GetStreamDescriptor 

func (q *Quasar) GetStreamDescriptor(ctx context.Context, uuid []byte) (res *mprovider.LookupResult, err bte.BTE)

Get a stream annotations and tags

func (*Quasar) GetStreamVersion 

func (q *Quasar) GetStreamVersion(ctx context.Context, uuid []byte) (major, minor uint64, err bte.BTE)

Get a stream annotations and tags

func (*Quasar) InitiateShutdown 

func (q *Quasar) InitiateShutdown() chan struct{}

func (*Quasar) InsertValues 

func (q *Quasar) InsertValues(ctx context.Context, id uuid.UUID, r []qtree.Record) (maj, min uint64, err bte.BTE)

func (*Quasar) ListCollections 

func (q *Quasar) ListCollections(ctx context.Context, prefix string, startingFrom string, limit uint64) ([]string, bte.BTE)

ListCollections returns a list of collections beginning with prefix (which may be "") and starting from the given string. If number is > 0, only that many results will be returned. More can be obtained by re-calling ListCollections with a given startingFrom and number.

func (*Quasar) LookupStreams 

func (q *Quasar) LookupStreams(ctx context.Context, collection string, isCollectionPrefix bool, tags map[string]*string, annotations map[string]*string) (chan *mprovider.LookupResult, chan bte.BTE)

Return back all streams in all collections beginning with collection (or exactly equal if prefix is false) provided they have the given tags and annotations, where a nil entry in the map means has the tag but the value is irrelevant

func (*Quasar) ObliterateStream 

func (q *Quasar) ObliterateStream(ctx context.Context, id []byte) bte.BTE

DeleteStream tombstones a stream

func (*Quasar) QueryChangedRanges 

func (q *Quasar) QueryChangedRanges(ctx context.Context, id uuid.UUID, startgen uint64, endgen uint64, resolution uint8) (chan ChangedRange, chan bte.BTE, uint64, uint64)

Resolution is how far down the tree to go when working out which blocks have changed. Higher resolutions are faster but will give you back coarser results.

func (*Quasar) QueryNearestValue 

func (q *Quasar) QueryNearestValue(ctx context.Context, id uuid.UUID, time int64, backwards bool, gen uint64) (qtree.Record, bte.BTE, uint64, uint64)

func (*Quasar) QueryStatisticalValuesStream 

func (q *Quasar) QueryStatisticalValuesStream(ctx context.Context, id uuid.UUID, start int64, end int64,
	gen uint64, pointwidth uint8) (chan qtree.StatRecord, chan bte.BTE, uint64, uint64)

func (*Quasar) QueryValuesStream 

func (q *Quasar) QueryValuesStream(ctx context.Context, id uuid.UUID, start int64, end int64, gen uint64) (chan qtree.Record, chan bte.BTE, uint64, uint64)

func (*Quasar) QueryWindow 

func (q *Quasar) QueryWindow(ctx context.Context, id uuid.UUID, start int64, end int64,
	gen uint64, width uint64, depth uint8) (chan qtree.StatRecord, chan bte.BTE, uint64, uint64)

func (*Quasar) Rez 

func (q *Quasar) Rez() *rez.RezManager

func (*Quasar) SetStreamAnnotations 

func (q *Quasar) SetStreamAnnotations(ctx context.Context, uuid []byte, aver uint64, changes map[string]*string) bte.BTE

Sets the stream annotations. An entry with a nil string implies delete

func (*Quasar) StorageProvider 

func (q *Quasar) StorageProvider() bprovider.StorageProvider

type Record 

type Record = qtree.Record

type StatRecordSlice 

type StatRecordSlice []qtree.StatRecord

func (StatRecordSlice) Len 

func (srs StatRecordSlice) Len() int

func (StatRecordSlice) Less 

func (srs StatRecordSlice) Less(i, j int) bool

func (StatRecordSlice) Swap 

func (srs StatRecordSlice) Swap(i, j int)

type StorageInterface 

type StorageInterface interface {
	JP() jprovider.JournalProvider
	CP() configprovider.ClusterConfiguration

	//Unthrottled
	WritePrimaryStorage(ctx context.Context, id uuid.UUID, r []Record) (major uint64, err bte.BTE)
	//Appropriate locks will be held
	StreamMajorVersion(ctx context.Context, id uuid.UUID) (uint64, bte.BTE)
}

Source Files

View all Source files

Directories

Path Synopsis
internal
bprovider
bstore
cephprovider
configprovider
fileprovider
jprovider
mprovider
pqm
rez

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.