README
¶
memdb 
The memdb is blazing fast specialized in memory key-value store for time-series database. The in-memory key-value data store persist entries into a WAL for immediate durability. The Write Ahead Log (WAL) retains memdb data when the db restarts. The WAL ensures data is durable in case of an unexpected failure.
About memdb
Key characteristics
- 100% Go
- Optimized for fast lookups and writes
- All DB methods are safe for concurrent use by multiple goroutines.
Quick Start
To build memdb from source code use go get command.
go get -u github.com/unit-io/unitdb/memdb
Usage
Detailed API documentation is available using the go.dev service.
Make use of the client by importing it in your Go client source code. For example,
import "github.com/unit-io/unitdb/memdb"
The memdb supports Get, Set, Delete operations. It also supports batch operations.
Samples are available in the examples directory for reference.
Opening a database
To open or create a new database, use the memdb.Open() function:
package main
import (
"log"
"github.com/unit-io/unitdb/memdb"
)
func main() {
// Opening a database.
// Open DB with reset flag to to skip recovery from log
db, err := memdb.Open(memdb.WithLogFilePath("unitdb"), memdb.WithLogReset())
if err != nil {
log.Fatal(err)
return
}
defer db.Close()
}
Writing to a database
Store a message
Use DB.Put() function to insert a new key-value pair. Note, if key exists then it overrides key-value when the writes happen within same timeID (see timeRecordInterval option) or it insert a new key-value pair in different timeID.
if timeID, err := db.Put(1, []byte("msg 1")); err != nil {
log.Fatal(err)
return
}
Read message
Use DB.Get() function to read inserted value. It gets entry from most recent timeID for the provided key.
if val, err := db.Get(1); err == nil {
log.Printf("%s ", val)
}
Lookup message
Use DB.Lookup() function to lookup entry for faster read.
timeID, err := db.Put(1, []byte("msg 1"))
if err != nil {
log.Fatal(err)
return
}
if val, err := db.Lookup(timeID, 1); err == nil {
log.Printf("%s ", val)
}
Deleting a message
use DB.Delete() function to delete a key-value pair.
if err := db.Delete(1); err != nil {
fmt.Println("error: ", err)
}
Batch operation
Use batch operation to bulk insert records into memdb or bulk delete records from memdb. See examples under examples/memdb folder.
Writing to a batch
Use Batch.Put() to insert a new key-value or Batch.Delete() to delete a key-value from DB.
db.Batch(func(b *memdb.Batch, completed <-chan struct{}) error {
for i := 1; i <= 10; i++ {
val := []byte(fmt.Sprintf("msg.%2d", i))
b.Put(uint64(i), val)
}
return nil
})
Contributing
If you'd like to contribute, please fork the repository and use a feature branch. Pull requests are welcome.
Licensing
This project is licensed under Apache-2.0 License.
Documentation
¶
Index ¶
- func ResponseHandler(w http.ResponseWriter, r *http.Request, data []byte)
- type Batch
- type DB
- func (db *DB) All(f func(timeID int64, keys []uint64) (bool, error)) (err error)
- func (db *DB) Batch(fn func(*Batch, <-chan struct{}) error) error
- func (db *DB) BlockIterator(f func(timeID int64, keys []uint64) (bool, error)) (err error)
- func (db *DB) Close() error
- func (db *DB) Delete(key uint64) error
- func (db *DB) Free(timeID int64) error
- func (db *DB) Get(key uint64) ([]byte, error)
- func (db *DB) HandleVarz(w http.ResponseWriter, r *http.Request)
- func (db *DB) Keys() []uint64
- func (db *DB) Lookup(timeID int64, key uint64) ([]byte, error)
- func (db *DB) NewBatch() *Batch
- func (db *DB) Put(key uint64, data []byte) (int64, error)
- func (db *DB) Size() int64
- func (db *DB) Varz() (*Varz, error)
- type Meter
- type Options
- type Varz
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ResponseHandler ¶ added in v0.1.1
func ResponseHandler(w http.ResponseWriter, r *http.Request, data []byte)
ResponseHandler handles responses for monitoring routes.
Types ¶
type Batch ¶ added in v0.1.1
type Batch struct {
// contains filtered or unexported fields
}
Batch is a write batch.
func (*Batch) Abort ¶ added in v0.1.1
Abort aborts batch or perform cleanup operation on batch complete.
func (*Batch) Commit ¶ added in v0.1.1
Commit commits changes to the DB. In batch operation commit is managed and client is not allowed to call Commit. On Commit complete batch operation signal to the caller if the batch is fully committed to DB.
type DB ¶
type DB struct {
// contains filtered or unexported fields
}
DB represents an SSD-optimized mem store.
func (*DB) Batch ¶ added in v0.1.1
Batch executes a function within the context of a read-write managed transaction. If no error is returned from the function then the transaction is written. If an error is returned then the entire transaction is rolled back. Any error that is returned from the function or returned from the write is returned from the Batch() method.
Attempting to manually commit or rollback within the function will cause a panic.
func (*DB) BlockIterator ¶ added in v0.1.1
BlockIterator iterates all time blocks from DB committed to the WAL.
func (*DB) Delete ¶ added in v0.1.1
Delete deletes entry from the DB. It writes deleted key into new time block to persist record into the WAL. If all entries are deleted from a time block then the time block is released from the WAL.
func (*DB) HandleVarz ¶ added in v0.1.1
func (db *DB) HandleVarz(w http.ResponseWriter, r *http.Request)
HandleVarz will process HTTP requests for unitdb stats information.
func (*DB) NewBatch ¶ added in v0.1.1
NewBatch returns unmanaged Batch so caller can perform Put, Write, Commit, Abort to the Batch.
type Meter ¶ added in v0.1.1
type Meter struct {
Metrics metrics.Metrics
TimeSeries metrics.TimeSeries
Gets metrics.Counter
Puts metrics.Counter
Syncs metrics.Counter
Recovers metrics.Counter
Dels metrics.Counter
}
Meter meter provides various db statistics.
func NewMeter ¶ added in v0.1.1
func NewMeter() *Meter
NewMeter provide meter to capture statistics.
func (*Meter) UnregisterAll ¶ added in v0.1.1
func (m *Meter) UnregisterAll()
UnregisterAll unregister all metrics from meter.
type Options ¶ added in v0.1.1
type Options interface {
// contains filtered or unexported methods
}
Options it contains configurable options and flags for DB.
func WithBufferSize ¶ added in v0.1.1
WithBufferSize sets max size of buffer to use for buffer pooling.
func WithDefaultOptions ¶ added in v0.1.1
func WithDefaultOptions() Options
WithDefaultOptions will open DB with some default values.
func WithLogFilePath ¶ added in v0.1.1
WithLogFilePath sets database directory for storing logs.
func WithLogInterval ¶ added in v0.1.1
WithLogInterval sets interval for a time block. Block is pushed to the queue to write it to the log file.
func WithLogReset ¶ added in v0.1.1
func WithLogReset() Options
WithLogReset flag to skip recovery on DB open and reset WAL.
func WithMemdbSize ¶ added in v0.1.1
WithMemdbSize sets max size of DB.
func WithTimeBlockInterval ¶ added in v0.1.1
WithTimeBlockInterval sets interval for a time block. Block is pushed to the queue to write it to the log file.
type Varz ¶ added in v0.1.1
type Varz struct {
Start time.Time `json:"start"`
Now time.Time `json:"now"`
Uptime string `json:"uptime"`
Count int64 `json:"count"`
Gets int64 `json:"gets"`
Puts int64 `json:"puts"`
Syncs int64 `json:"syncs"`
Recovers int64 `json:"recovers"`
Dels int64 `json:"Dels"`
HMean float64 `json:"hmean"` // Event duration harmonic mean.
P50 float64 `json:"p50"` // Event duration nth percentiles.
P75 float64 `json:"p75"`
P95 float64 `json:"p95"`
P99 float64 `json:"p99"`
P999 float64 `json:"p999"`
Long5p float64 `json:"long_5p"` // Average of the longest 5% event durations.
Short5p float64 `json:"short_5p"` // Average of the shortest 5% event durations.
Max float64 `json:"max"` // Highest event duration.
Min float64 `json:"min"` // Lowest event duration.
StdDev float64 `json:"stddev"` // Standard deviation.
}
Varz outputs memdb stats on the monitoring port at /varz.
Source Files