unitdb

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 20, 2021 License: Apache-2.0 Imports: 29 Imported by: 6

README

unitdb GoDoc Go Report Card Build Status Coverage Status

Unitdb is blazing fast specialized time-series database for microservices, IoT, and realtime internet connected devices. As Unitdb satisfy the requirements for low latency and binary messaging, it is a perfect time-series database for applications such as internet of things and internet connected devices. The Unitdb Server uses uTP (unit Transport Protocol) for the Client Server messaging. Read uTP Specification.

Don't forget to ⭐ this repo if you like Unitdb!

About unitdb

Key characteristics

  • 100% Go
  • Can store larger-than-memory data sets
  • Optimized for fast lookups and writes
  • Supports writing billions of records per hour
  • Supports opening database with immutable flag
  • Supports database encryption
  • Supports time-to-live on message entries
  • Supports writing to wildcard topics
  • Data is safely written to disk with accuracy and high performant block sync technique

Quick Start

To build Unitdb from source code use go get command.

go get github.com/unit-io/unitdb

Usage

Detailed API documentation is available using the go.dev service.

Make use of the client by importing it in your Go client source code. For example,

import "github.com/unit-io/unitdb"

Unitdb supports Get, Put, Delete operations. It also supports encryption, batch operations, and writing to wildcard topics. See usage guide.

Samples are available in the examples directory for reference.

Clustering

To bring up the Unitdb cluster start 2 or more nodes. For fault tolerance 3 nodes or more are recommended.

> ./bin/unitdb -listen=:6060 -grpc_listen=:6080 -cluster_self=one -db_path=/tmp/unitdb/node1
> ./bin/unitdb -listen=:6061 -grpc_listen=:6081 -cluster_self=two -db_path=/tmp/unitdb/node2

Above example shows each Unitdb node running on the same host, so each node must listen on different ports. This would not be necessary if each node ran on a different host.

Client Libraries

Make use of officially supported client libraries to connect to unitdb server running on single node and running on a cluster.

  • unitdb-go Lightweight and high performance unitdb Go client library.
  • unitdb-dart High performance unitdb Flutter/Dart client library.

Architecture Overview

The unitdb engine handles data from the point put request is received through writing data to the physical disk. Data is compressed and encrypted (if encryption is set) then written to a WAL for immediate durability. Entries are written to memdb and become immediately queryable. The memdb entries are periodically written to log files in the form of blocks.

To efficiently compact and store data, the unitdb engine groups entries sequence by topic key, and then orders those sequences by time and each block keep offset of previous block in reverse time order. Index block offset is calculated from entry sequence in the time-window block. Data is read from data block using index entry information and then it un-compresses the data on read (if encryption flag was set then it un-encrypts the data on read).

Unitdb stores compressed data (live records) in a memdb store. Data records in a memdb are partitioned into (live) time-blocks of configured capacity. New time-blocks are created at ingestion, while old time-blocks are appended to the log files and later sync to the disk store.

When Unitdb receives a put or delete request, it first writes records into tiny-log for recovery. Tiny-logs are added to the log queue to write it to the log file. The tiny-log write is triggered by the time or size of tiny-log incase of backoff due to massive loads.

The tiny-log queue is maintained in memory with a pre-configured size, and during massive loads the memdb backoff process will block the incoming requests from proceeding before the tiny-log queue is cleared by a write operation. After records are appended to the tiny-log, and written to the log files the records are then sync to the disk store using blazing fast block sync technique.

Next steps

In the future, we intend to enhance the Unitdb with the following features:

  • Distributed design: We are working on building out the distributed design of Unitdb, including replication and sharding management to improve its scalability.
  • Developer support and tooling: We are working on building more intuitive tooling, refactoring code structures, and enriching documentation to improve the onboarding experience, enabling developers to quickly integrate Unitdb to their time-series database stack.

Contributing

As Unitdb is under active development and at this time Unitdb is not seeking major changes or new features; however, small bugfixes are encouraged. Unitdb is seeking contibution to improve test coverage and documentation.

Licensing

This project is licensed under Apache-2.0 License.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Debug 

func Debug(context, msg string)

Debug logs the debug message with tag if it is turned on.

func Fatal 

func Fatal(context, msg string, err error)

Fatal logs the fatal error messages.

func Info 

func Info(context, action string)

Info logs the action with a tag.

func ParseLevel 

func ParseLevel(level string, defaultLevel zerolog.Level) zerolog.Level

ParseLevel parses a string which represents a log level and returns a zerolog.Level.

func ResponseHandler 

func ResponseHandler(w http.ResponseWriter, r *http.Request, data []byte)

ResponseHandler handles responses for monitoring routes.

Types

type Batch 

type Batch struct {
	// contains filtered or unexported fields
}

Batch is a write batch.

func (*Batch) Abort 

func (b *Batch) Abort()

Abort abort is a batch cleanup operation on batch complete.

func (*Batch) Commit 

func (b *Batch) Commit() error

Commit commits changes to the DB. In batch operation commit is managed and client is not allowed to call Commit. On Commit complete batch operation signal to the caller if the batch is fully committed to DB.

func (*Batch) Delete 

func (b *Batch) Delete(id, topic []byte) error

Delete appends delete entry to batch for given key. It is safe to modify the contents of the argument after Delete returns but not before.

func (*Batch) DeleteEntry 

func (b *Batch) DeleteEntry(e *Entry) error

DeleteEntry appends entry for deletion to a batch for given key. It is safe to modify the contents of the argument after Delete returns but not before.

func (*Batch) Put 

func (b *Batch) Put(topic, payload []byte) error

Put adds entry to batch for given topic->key/value. Client must provide Topic to the BatchOptions. It is safe to modify the contents of the argument after Put returns but not before.

func (*Batch) PutEntry 

func (b *Batch) PutEntry(e *Entry) error

PutEntry appends entries to a bacth for given topic->key/value pair. It is safe to modify the contents of the argument after Put returns but not before.

func (*Batch) SetOptions 

func (b *Batch) SetOptions(opts ...Options)

SetOptions sets batch options.

func (*Batch) Write 

func (b *Batch) Write() error

Write starts writing entries into DB. It returns an error if batch write fails.

type DB 

type DB struct {
	// contains filtered or unexported fields
}

DB represents the message storage for topic->keys-values. All DB methods are safe for concurrent use by multiple goroutines.

func Open 

func Open(path string, opts ...Options) (*DB, error)

Open opens or creates a new DB.

func (*DB) Batch 

func (db *DB) Batch(fn func(*Batch, <-chan struct{}) error) error

Batch executes a function within the context of a read-write managed transaction. If no error is returned from the function then the transaction is written. If an error is returned then the entire transaction is rolled back. Any error that is returned from the function or returned from the write is returned from the Batch() method.

Attempting to manually commit or rollback within the function will cause a panic.

func (*DB) Close 

func (db *DB) Close() error

Close closes the DB.

func (*DB) Count 

func (db *DB) Count() uint64

Count returns the number of items in the DB.

func (*DB) Delete 

func (db *DB) Delete(id, topic []byte) error

Delete sets entry for deletion. It is safe to modify the contents of the argument after Delete returns but not before.

func (*DB) DeleteEntry 

func (db *DB) DeleteEntry(e *Entry) error

DeleteEntry deletes an entry from DB. you must provide an ID to delete an entry. It is safe to modify the contents of the argument after Delete returns but not before.

func (*DB) FileSize 

func (db *DB) FileSize() (int64, error)

FileSize returns the total size of the disk storage used by the DB.

func (*DB) Get 

func (db *DB) Get(q *Query) (items [][]byte, err error)

Get return items matching the query paramater.

func (*DB) HandleVarz 

func (db *DB) HandleVarz(w http.ResponseWriter, r *http.Request)

HandleVarz will process HTTP requests for unitdb stats information.

func (*DB) NewContract 

func (db *DB) NewContract() (uint32, error)

NewContract generates a new Contract.

func (*DB) NewID 

func (db *DB) NewID() []byte

NewID generates new ID that is later used to put entry or delete entry.

func (*DB) Put 

func (db *DB) Put(topic, payload []byte) error

Put puts entry into DB. It uses default Contract to put entry into DB. It is safe to modify the contents of the argument after Put returns but not before.

func (*DB) PutEntry 

func (db *DB) PutEntry(e *Entry) error

PutEntry puts entry into the DB, if Contract is not specified then it uses master Contract. It is safe to modify the contents of the argument after PutEntry returns but not before.

func (*DB) Sync added in v0.1.1 

func (db *DB) Sync() error

Sync syncs entries into DB. Sync happens synchronously. Sync write window entries into summary file and write index, and data to respective index and data files. In case of any error during sync operation recovery is performed on log file (write ahead log).

func (*DB) Varz 

func (db *DB) Varz() (*Varz, error)

Varz returns a Varz struct containing the unitdb information.

type Entry 

type Entry struct {
	ID         []byte // The ID of the message.
	Topic      []byte // The topic of the message.
	Payload    []byte // The payload of the message.
	ExpiresAt  uint32 // The time expiry of the message.
	Contract   uint32 // The contract is used to as salt to hash topic parts and also used as prefix in the message ID.
	Encryption bool
	// contains filtered or unexported fields
}

Entry entry is a message entry structure.

func NewEntry 

func NewEntry(topic, payload []byte) *Entry

NewEntry creates a new entry structure from the topic.

func (*Entry) WithContract added in v0.1.1 

func (e *Entry) WithContract(contract uint32) *Entry

WithContract sets contract on entry.

func (*Entry) WithEncryption added in v0.1.1 

func (e *Entry) WithEncryption() *Entry

WithEncryption sets encryption on entry.

func (*Entry) WithID added in v0.1.1 

func (e *Entry) WithID(id []byte) *Entry

WithID sets entry ID.

func (*Entry) WithPayload added in v0.1.1 

func (e *Entry) WithPayload(payload []byte) *Entry

WithPayload sets payload to put entry into DB.

func (*Entry) WithTTL added in v0.1.1 

func (e *Entry) WithTTL(ttl string) *Entry

WithTTL sets TTL for message expiry for the entry.

type Filter 

type Filter struct {
	// contains filtered or unexported fields
}

Filter filter is bloom filter generator.

func (*Filter) Append 

func (f *Filter) Append(h uint64)

Append appends an entry to bloom filter.

func (*Filter) Test 

func (f *Filter) Test(h uint64) bool

Test tests entry in bloom filter. It returns false if entry definitely does not exist or true may be entry exist in DB.

type Meter 

type Meter struct {
	Metrics    metrics.Metrics
	TimeSeries metrics.TimeSeries
	Gets       metrics.Counter
	Puts       metrics.Counter
	Leases     metrics.Counter
	Syncs      metrics.Counter
	Recovers   metrics.Counter
	Aborts     metrics.Counter
	Dels       metrics.Counter
	InMsgs     metrics.Counter
	OutMsgs    metrics.Counter
	InBytes    metrics.Counter
	OutBytes   metrics.Counter
}

Meter meter provides various db statistics.

func NewMeter 

func NewMeter() *Meter

NewMeter provide meter to capture statistics.

func (*Meter) UnregisterAll 

func (m *Meter) UnregisterAll()

UnregisterAll unregister all metrics from meter.

type Options 

type Options interface {
	// contains filtered or unexported methods
}

Options it contains configurable options and flags for DB.

func WithBackgroundKeyExpiry added in v0.1.1 

func WithBackgroundKeyExpiry() Options

WithBackgroundKeyExpiry sets background key expiry for DB.

func WithBatchContract added in v0.1.1 

func WithBatchContract(contract uint32) Options

WithBatchContract sets contract for batch operation.

func WithBatchEncryption added in v0.1.1 

func WithBatchEncryption() Options

WithBatchEncryption sets encryption on batch operation.

func WithBatchWriteInterval added in v0.1.1 

func WithBatchWriteInterval(dur time.Duration) Options

WithBatchWriteInterval sets batch write interval to partial write large batch.

func WithBufferSize added in v0.1.1 

func WithBufferSize(size int64) Options

WithBufferSize sets Size of buffer to use for pooling.

func WithDefaultBatchOptions added in v0.1.1 

func WithDefaultBatchOptions() Options

WithDefaultBatchOptions will set some default values for Batch operation. 

contract: MasterContract
encryption: False

func WithDefaultFlags added in v0.1.1 

func WithDefaultFlags() Options

WithDefaultFlags will open DB with some default values. 

immutable: True
encryption: False
backgroundKeyExpiry: False

func WithDefaultOptions added in v0.1.1 

func WithDefaultOptions() Options

WithDefaultOptions will open DB with some default values.

func WithDefaultQueryLimit added in v0.1.1 

func WithDefaultQueryLimit(limit int) Options

WithDefaultQueryLimit limits maximum number of records to fetch if the DB Get or DB Iterator method does not specify a limit.

func WithDefaultQueryOptions added in v0.1.1 

func WithDefaultQueryOptions() Options

WithDefaultQueryOptions will set some default values for Query operation. 

defaultQueryLimit: 1000
maxQueryLimit: 100000

func WithEncryption added in v0.1.1 

func WithEncryption() Options

WithEncryption sets encryption on DB.

func WithEncryptionKey added in v0.1.1 

func WithEncryptionKey(key []byte) Options

WithEncryptionKey sets encryption key to use for data encryption.

func WithFreeBlockSize added in v0.1.1 

func WithFreeBlockSize(size int64) Options

WithFreeBlockSize sets minimum freeblocks size before free blocks are allocated and reused.

func WithMaxQueryLimit added in v0.1.1 

func WithMaxQueryLimit(limit int) Options

WithMaxQueryLimit limits maximum number of records to fetch if the DB Get or DB Iterator method does not specify a limit or specify a limit larger than MaxQueryResults.

func WithMaxSyncDuration added in v0.1.1 

func WithMaxSyncDuration(dur time.Duration, interval int) Options

WithMaxSyncDuration sets the amount of time between background fsync() calls.

func WithMemdbSize added in v0.1.1 

func WithMemdbSize(size int64) Options

WithMemdbSize sets Size of blockcache.

func WithMutable added in v0.1.1 

func WithMutable() Options

WithMutable sets Immutable flag to false.

type Query 

type Query struct {
	Topic    []byte // The topic of the message.
	Contract uint32 // The contract is used as prefix in the message ID.
	Limit    int    // The maximum number of elements to return.
	// contains filtered or unexported fields
}

Query represents a topic to query and optional contract information.

func NewQuery added in v0.1.1 

func NewQuery(topic []byte) *Query

NewQuery creates a new query structure from the topic.

func (*Query) WithContract added in v0.1.1 

func (q *Query) WithContract(contract uint32) *Query

WithContract sets contract on query.

func (*Query) WithLast added in v0.1.1 

func (q *Query) WithLast(dur string) *Query

WithLast sets query duration to fetch stored messages.

func (*Query) WithLimit added in v0.1.1 

func (q *Query) WithLimit(limit int) *Query

WithLimit sets query limit.

type Varz 

type Varz struct {
	Start    time.Time `json:"start"`
	Now      time.Time `json:"now"`
	Uptime   string    `json:"uptime"`
	Seq      int64     `json:"seq"`
	Count    int64     `json:"count"`
	Gets     int64     `json:"gets"`
	Puts     int64     `json:"puts"`
	Leases   int64     `json:"leases"`
	Syncs    int64     `json:"syncs"`
	Recovers int64     `json:"recovers"`
	Aborts   int64     `json:"aborts"`
	Dels     int64     `json:"Dels"`
	InMsgs   int64     `json:"in_msgs"`
	OutMsgs  int64     `json:"out_msgs"`
	InBytes  int64     `json:"in_bytes"`
	OutBytes int64     `json:"out_bytes"`
	HMean    float64   `json:"hmean"` // Event duration harmonic mean.
	P50      float64   `json:"p50"`   // Event duration nth percentiles.
	P75      float64   `json:"p75"`
	P95      float64   `json:"p95"`
	P99      float64   `json:"p99"`
	P999     float64   `json:"p999"`
	Long5p   float64   `json:"long_5p"`  // Average of the longest 5% event durations.
	Short5p  float64   `json:"short_5p"` // Average of the shortest 5% event durations.
	Max      float64   `json:"max"`      // Highest event duration.
	Min      float64   `json:"min"`      // Lowest event duration.
	StdDev   float64   `json:"stddev"`   // Standard deviation.

}

Varz outputs unitdb stats on the monitoring port at /varz.

Source Files

View all Source files

Directories

Path Synopsis
examples
memdb
sample
simple
Package hash provides a minimal-memory AnchorHash consistent-hash implementation for Go.
 Package hash provides a minimal-memory AnchorHash consistent-hash implementation for Go.
common
internal
internal/config
internal/db
internal/db/unitdb
internal/message
internal/message/security
internal/net
internal/net/listener
internal/pkg/collection
internal/pkg/crypto
internal/pkg/encoding
internal/pkg/hash
Package ringhash implementats a consistent ring hash: https://en.wikipedia.org/wiki/Consistent_hashing
 Package ringhash implementats a consistent ring hash: https://en.wikipedia.org/wiki/Consistent_hashing
internal/pkg/log
internal/pkg/metrics
internal/pkg/stats
Package Meter yields summarized data describing a series of timed events.
 Package Meter yields summarized data describing a series of timed events.
internal/pkg/uid
internal/store
internal/types
proto
Package __schema is a generated protocol buffer package.
 Package __schema is a generated protocol buffer package.
utp

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.