btrdb

README

BTrDB

The Berkeley TRee DataBase is a high performance time series database designed to support high density data storage applications. This project used to be called QUASAR, but we have changed the name partly to match publications, and partly as a flag day. The capnp interface in BTrDB is designed to better support large queries and clusters and is not backwards compatible with the quasar interface.

Dependencies

BTrDB uses a MongoDB collection to store metadata. Also, if installed in High Availability mode, it requires a ceph pool. Note that even if not using ceph, librados needs to be installed.

Rapid installation with docker

If you want to try out BTrDB with zero hassles, you can simply use our docker image.

# a docker network helps by allowing DNS resolution between containers
docker network create mynet
docker run -d --net mynet --name mongo mongo:3.2
docker run -it -v /my/datadir:/srv -e BTRDB_MONGO_SERVER=mongo.mynet btrdb/release:3.4 makedb
docker run -d -v my/datadir:/srv -p 4410:4410 btrdb/release:3.4

Installation

To run an archiver, make sure that you have Go >= 1.4 installed and then run the following:

apt-get install librados-dev
go get github.com/SoftwareDefinedBuildings/btrdb/btrdbd

This will install the tools into your $GOPATH/bin directory. If you have this directory on your $PATH then you do not need to do anything further. Otherwise you will need to add the binaries to your $PATH variable manually.

Note that in order to run the btrdb server, you will need to copy btrdb.conf from the github repository to /etc/btrdb/btrdb.conf (or the directory that you are in).

An alternative to 'go get'ing to your GOPATH is to clone the repository then do:

apt-get install librados-dev
go get -d ./... && go install ./btrdbd

This will also put the btrdbd binary in your $GOPATH/bin.

Configuration

Sensible defaults (for a production deployment) are already found in btrdb.conf. Some things you may need to adjust:

• The MongoDB server and collection name
• The block cache size (defaults to 32GB). Note that quasar uses more than this, this is just a primary contributor to the RAM footprint.
• The file storage path or ceph details

Once your configuration is set up, you can set up the files, and database indices with

btrdbd -makedb

Which should print out:

Configuration OK!
Creating a new database
Done

You can now run a server with:

btrdbd

Using the database

Note that we are presently working on release engineering, and hope to release the first (public) version in August 2016. If you are using it now, bear in mind it is still in development.

To communicate with the database, there are go bindings and python bindings. The go bindings are faster and more maintained.

Documentation

Index

• Constants
• Variables
• type ChangedRange
• type Quasar
  • func NewQuasar(cfg *QuasarConfig) (*Quasar, error)
  • func (q *Quasar) DeleteRange(id uuid.UUID, start int64, end int64) error
  • func (q *Quasar) Flush(id uuid.UUID) error
  • func (q *Quasar) InsertValues(id uuid.UUID, r []qtree.Record)
  • func (q *Quasar) IsPending() bool
  • func (q *Quasar) QueryChangedRanges(id uuid.UUID, startgen uint64, endgen uint64, resolution uint8) ([]ChangedRange, uint64, error)
  • func (q *Quasar) QueryGeneration(id uuid.UUID) (uint64, error)
  • func (q *Quasar) QueryNearestValue(id uuid.UUID, time int64, backwards bool, gen uint64) (qtree.Record, uint64, error)
  • func (q *Quasar) QueryStatisticalValues(id uuid.UUID, start int64, end int64, gen uint64, pointwidth uint8) ([]qtree.StatRecord, uint64, error)
  • func (q *Quasar) QueryStatisticalValuesStream(id uuid.UUID, start int64, end int64, gen uint64, pointwidth uint8) (chan qtree.StatRecord, chan error, uint64)
  • func (q *Quasar) QueryValues(id uuid.UUID, start int64, end int64, gen uint64) ([]qtree.Record, uint64, error)
  • func (q *Quasar) QueryValuesStream(id uuid.UUID, start int64, end int64, gen uint64) (chan qtree.Record, chan error, uint64)
  • func (q *Quasar) QueryWindow(id uuid.UUID, start int64, end int64, gen uint64, width uint64, depth uint8) (chan qtree.StatRecord, uint64)
• type QuasarConfig

Constants

const LatestGeneration = bstore.LatestGeneration

const MaximumTime = (48 << 56)

const MinimumTime = -(16 << 56)

const VersionMajor = 3

const VersionMinor = 4

Variables

var BuildDate string

var VersionString string

Will be set at build time to Major.Minor.Build

Types

type ChangedRange

type ChangedRange struct {
	Start int64
	End   int64
}

type Quasar

type Quasar struct {
	// contains filtered or unexported fields
}

func NewQuasar

func NewQuasar(cfg *QuasarConfig) (*Quasar, error)

func (*Quasar) DeleteRange

func (q *Quasar) DeleteRange(id uuid.UUID, start int64, end int64) error

func (*Quasar) Flush

func (q *Quasar) Flush(id uuid.UUID) error

func (*Quasar) InsertValues

func (q *Quasar) InsertValues(id uuid.UUID, r []qtree.Record)

func (*Quasar) IsPending

func (q *Quasar) IsPending() bool

Return true if there are uncommited results to be written to disk Should only be used during shutdown as it hogs the glock

func (*Quasar) QueryChangedRanges

func (q *Quasar) QueryChangedRanges(id uuid.UUID, startgen uint64, endgen uint64, resolution uint8) ([]ChangedRange, uint64, error)

Resolution is how far down the tree to go when working out which blocks have changed. Higher resolutions are faster but will give you back coarser results.

func (*Quasar) QueryGeneration

func (q *Quasar) QueryGeneration(id uuid.UUID) (uint64, error)

func (*Quasar) QueryNearestValue

func (q *Quasar) QueryNearestValue(id uuid.UUID, time int64, backwards bool, gen uint64) (qtree.Record, uint64, error)

func (*Quasar) QueryStatisticalValues

func (q *Quasar) QueryStatisticalValues(id uuid.UUID, start int64, end int64,
	gen uint64, pointwidth uint8) ([]qtree.StatRecord, uint64, error)

func (*Quasar) QueryStatisticalValuesStream

func (q *Quasar) QueryStatisticalValuesStream(id uuid.UUID, start int64, end int64,
	gen uint64, pointwidth uint8) (chan qtree.StatRecord, chan error, uint64)

func (*Quasar) QueryValues

func (q *Quasar) QueryValues(id uuid.UUID, start int64, end int64, gen uint64) ([]qtree.Record, uint64, error)

These functions are the API. TODO add all the bounds checking on PW, and sanity on start/end

func (*Quasar) QueryValuesStream

func (q *Quasar) QueryValuesStream(id uuid.UUID, start int64, end int64, gen uint64) (chan qtree.Record, chan error, uint64)

func (*Quasar) QueryWindow

func (q *Quasar) QueryWindow(id uuid.UUID, start int64, end int64,
	gen uint64, width uint64, depth uint8) (chan qtree.StatRecord, uint64)

type QuasarConfig

type QuasarConfig struct {
	//Measured in the number of datablocks
	//So 1000 is 8 MB cache
	DatablockCacheSize uint64

	//This enables the grouping of value inserts
	//with a commit every Interval millis
	//If the number of stored values exceeds
	//EarlyTrip
	TransactionCoalesceEnable    bool
	TransactionCoalesceInterval  uint64
	TransactionCoalesceEarlyTrip uint64

	Params map[string]string
}