bitcask

package module
v0.3.10 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 18, 2020 License: MIT Imports: 18 Imported by: 47

README

bitcask

CodeCov Go Report Card codebeat badge GoDoc GitHub license Sourcegraph TODOs

A high performance Key/Value store written in Go with a predictable read/write performance and high throughput. Uses a Bitcask on-disk layout (LSM+WAL) similar to Riak

For a more feature-complete Redis-compatible server, distributed key/value store have a look at Bitraft which uses this library as its backend. Use Bitcask as a starting point or if you want to embed in your application, use Bitraft if you need a complete server/client solution with high availability with a Redis-compatible API.

Features

  • Embeddable (import "github.com/prologic/bitcask")
  • Builtin CLI (bitcask)
  • Builtin Redis-compatible server (bitcaskd)
  • Predictable read/write performance
  • Low latency
  • High throughput (See: Performance )

Development

$ git clone https://github.com/prologic/bitcask.git
$ make

Install

$ go get github.com/prologic/bitcask

Usage (library)

Install the package into your project:

$ go get github.com/prologic/bitcask

package main

import (
	"log"
	"github.com/prologic/bitcask"
)

func main() {
    db, _ := bitcask.Open("/tmp/db")
    defer db.Close()
    db.Put([]byte("Hello"), []byte("World"))
    val, _ := db.Get([]byte("Hello"))
    log.Printf(string(val))
}

See the godoc for further documentation and other examples.

Usage (tool)

$ bitcask -p /tmp/db set Hello World
$ bitcask -p /tmp/db get Hello
World

Usage (server)

There is also a builtin very simple Redis-compatible server called bitcaskd:

$ ./bitcaskd ./tmp
INFO[0000] starting bitcaskd v0.0.7@146f777              bind=":6379" path=./tmp

Example session:

$ telnet localhost 6379
Trying ::1...
Connected to localhost.
Escape character is '^]'.
SET foo bar
+OK
GET foo
$3
bar
DEL foo
:1
GET foo
$-1
PING
+PONG
QUIT
+OK
Connection closed by foreign host.

Docker

You can also use the Bitcask Docker Image:

$ docker pull prologic/bitcask
$ docker run -d -p 6379:6379 prologic/bitcask

Performance

Benchmarks run on a 11" Macbook with a 1.4Ghz Intel Core i7:

$ make bench
...
goos: darwin
goarch: amd64
pkg: github.com/prologic/bitcask

BenchmarkGet/128B-4         	  316515	      3263 ns/op	  39.22 MB/s	     160 B/op	       1 allocs/op
BenchmarkGet/256B-4         	  382551	      3204 ns/op	  79.90 MB/s	     288 B/op	       1 allocs/op
BenchmarkGet/512B-4         	  357216	      3835 ns/op	 133.51 MB/s	     576 B/op	       1 allocs/op
BenchmarkGet/1K-4           	  274958	      4429 ns/op	 231.20 MB/s	    1152 B/op	       1 allocs/op
BenchmarkGet/2K-4           	  227764	      5013 ns/op	 408.55 MB/s	    2304 B/op	       1 allocs/op
BenchmarkGet/4K-4           	  187557	      5534 ns/op	 740.15 MB/s	    4864 B/op	       1 allocs/op
BenchmarkGet/8K-4           	  153546	      7652 ns/op	1070.56 MB/s	    9472 B/op	       1 allocs/op
BenchmarkGet/16K-4          	  115549	     10272 ns/op	1594.95 MB/s	   18432 B/op	       1 allocs/op
BenchmarkGet/32K-4          	   69592	     16405 ns/op	1997.39 MB/s	   40960 B/op	       1 allocs/op

BenchmarkPut/128BNoSync-4   	  123519	     11094 ns/op	  11.54 MB/s	      49 B/op	       2 allocs/op
BenchmarkPut/256BNoSync-4   	   84662	     13398 ns/op	  19.11 MB/s	      50 B/op	       2 allocs/op
BenchmarkPut/1KNoSync-4     	   46345	     24855 ns/op	  41.20 MB/s	      58 B/op	       2 allocs/op
BenchmarkPut/2KNoSync-4     	   28820	     43817 ns/op	  46.74 MB/s	      68 B/op	       2 allocs/op
BenchmarkPut/4KNoSync-4     	   13976	     90059 ns/op	  45.48 MB/s	      89 B/op	       2 allocs/op
BenchmarkPut/8KNoSync-4     	    7852	    155101 ns/op	  52.82 MB/s	     130 B/op	       2 allocs/op
BenchmarkPut/16KNoSync-4    	    4848	    238113 ns/op	  68.81 MB/s	     226 B/op	       2 allocs/op
BenchmarkPut/32KNoSync-4    	    2564	    391483 ns/op	  83.70 MB/s	     377 B/op	       3 allocs/op

BenchmarkPut/128BSync-4     	     260	   4611273 ns/op	   0.03 MB/s	      48 B/op	       2 allocs/op
BenchmarkPut/256BSync-4     	     265	   4665506 ns/op	   0.05 MB/s	      48 B/op	       2 allocs/op
BenchmarkPut/1KSync-4       	     256	   4757334 ns/op	   0.22 MB/s	      48 B/op	       2 allocs/op
BenchmarkPut/2KSync-4       	     255	   4996788 ns/op	   0.41 MB/s	      92 B/op	       2 allocs/op
BenchmarkPut/4KSync-4       	     222	   5136481 ns/op	   0.80 MB/s	      98 B/op	       2 allocs/op
BenchmarkPut/8KSync-4       	     223	   5530824 ns/op	   1.48 MB/s	      99 B/op	       2 allocs/op
BenchmarkPut/16KSync-4      	     213	   5717880 ns/op	   2.87 MB/s	     202 B/op	       2 allocs/op
BenchmarkPut/32KSync-4      	     211	   5835948 ns/op	   5.61 MB/s	     355 B/op	       3 allocs/op

BenchmarkScan-4             	  568696	      2036 ns/op	     392 B/op	      33 allocs/op
PASS

For 128B values:

  • ~300,000 reads/sec
  • ~90,000 writes/sec
  • ~490,000 scans/sec

The full benchmark above shows linear performance as you increase key/value sizes.

Support

Support the ongoing development of Bitcask!

Sponser

Stargazers over time

Stargazers over time

Contributors

Thank you to all those that have contributed to this project, battle-tested it, used it in their own projects or products, fixed bugs, improved performance and even fix tiny typos in documentation! Thank you and keep contributing!

You can find an AUTHORS file where we keep a list of contributors to the project. If you contriibute a PR please consider adding your name there. There is also Github's own Contributors statistics.

  • bitraft -- A Distributed Key/Value store (using Raft) with a Redis compatible protocol.
  • bitcaskfs -- A FUSE filesystem for mounting a Bitcask database.
  • bitcask-bench -- A benchmarking tool comparing Bitcask and several other Go key/value libraries.

License

bitcask is licensed under the term of the MIT License

Documentation

Overview

Package bitcask implements a high-performance key-value store based on a WAL and LSM.

Example 
_, _ = Open("path/to/db")

Output:
Example (WithOptions) 
opts := []Option{
	WithMaxKeySize(1024),
	WithMaxValueSize(4096),
}
_, _ = Open("path/to/db", opts...)

Output:

Index

Examples

Constants

View Source 
const (
	// DefaultDirFileModeBeforeUmask is the default os.FileMode used when creating directories
	DefaultDirFileModeBeforeUmask = os.FileMode(0700)

	// DefaultFileFileModeBeforeUmask is the default os.FileMode used when creating files
	DefaultFileFileModeBeforeUmask = os.FileMode(0600)

	// DefaultMaxDatafileSize is the default maximum datafile size in bytes
	DefaultMaxDatafileSize = 1 << 20 // 1MB

	// DefaultMaxKeySize is the default maximum key size in bytes
	DefaultMaxKeySize = uint32(64) // 64 bytes

	// DefaultMaxValueSize is the default value size in bytes
	DefaultMaxValueSize = uint64(1 << 16) // 65KB

	// DefaultSync is the default file synchronization action
	DefaultSync = false
)

Variables

View Source 
var (
	// ErrKeyNotFound is the error returned when a key is not found
	ErrKeyNotFound = errors.New("error: key not found")

	// ErrKeyTooLarge is the error returned for a key that exceeds the
	// maximum allowed key size (configured with WithMaxKeySize).
	ErrKeyTooLarge = errors.New("error: key too large")

	// ErrEmptyKey is the error returned for a value with an empty key.
	ErrEmptyKey = errors.New("error: empty key")

	// ErrValueTooLarge is the error returned for a value that exceeds the
	// maximum allowed value size (configured with WithMaxValueSize).
	ErrValueTooLarge = errors.New("error: value too large")

	// ErrChecksumFailed is the error returned if a key/value retrieved does
	// not match its CRC checksum
	ErrChecksumFailed = errors.New("error: checksum failed")

	// ErrDatabaseLocked is the error returned if the database is locked
	// (typically opened by another process)
	ErrDatabaseLocked = errors.New("error: database locked")

	// ErrMergeInProgress is the error returned if merge is called when already a merge
	// is in progress
	ErrMergeInProgress = errors.New("error: merge already in progress")
)

Functions

This section is empty.

Types

type Bitcask 

type Bitcask struct {
	*flock.Flock
	// contains filtered or unexported fields
}

Bitcask is a struct that represents a on-disk LSM and WAL data structure and in-memory hash of key/value pairs as per the Bitcask paper and seen in the Riak database.

func Open 

func Open(path string, options ...Option) (*Bitcask, error)

Open opens the database at the given path with optional options. Options can be provided with the `WithXXX` functions that provide configuration options as functions.

func (*Bitcask) Backup added in v0.3.10 

func (b *Bitcask) Backup(path string) error

Backup copies db directory to given path it creates path if it does not exist

func (*Bitcask) Close 

func (b *Bitcask) Close() error

Close closes the database and removes the lock. It is important to call Close() as this is the only way to cleanup the lock held by the open database.

func (*Bitcask) Delete 

func (b *Bitcask) Delete(key []byte) error

Delete deletes the named key. If the key doesn't exist or an I/O error occurs the error is returned.

func (*Bitcask) DeleteAll added in v0.3.6 

func (b *Bitcask) DeleteAll() (err error)

DeleteAll deletes all the keys. If an I/O error occurs the error is returned.

func (*Bitcask) Fold 

func (b *Bitcask) Fold(f func(key []byte) error) (err error)

Fold iterates over all keys in the database calling the function `f` for each key. If the function returns an error, no further keys are processed and the error returned.

func (*Bitcask) Get 

func (b *Bitcask) Get(key []byte) ([]byte, error)

Get fetches value for given key

func (*Bitcask) Has 

func (b *Bitcask) Has(key []byte) bool

Has returns true if the key exists in the database, false otherwise.

func (*Bitcask) Keys 

func (b *Bitcask) Keys() chan []byte

Keys returns all keys in the database as a channel of keys

func (*Bitcask) Len 

func (b *Bitcask) Len() int

Len returns the total number of keys in the database

func (*Bitcask) Merge added in v0.2.2 

func (b *Bitcask) Merge() error

Merge merges all datafiles in the database. Old keys are squashed and deleted keys removes. Duplicate key/value pairs are also removed. Call this function periodically to reclaim disk space.

func (*Bitcask) Put 

func (b *Bitcask) Put(key, value []byte) error

Put stores the key and value in the database.

func (*Bitcask) Reclaimable added in v0.3.10 

func (b *Bitcask) Reclaimable() int64

Reclaimable returns space that can be reclaimed

func (*Bitcask) Reopen added in v0.3.6 

func (b *Bitcask) Reopen() error

func (*Bitcask) Scan 

func (b *Bitcask) Scan(prefix []byte, f func(key []byte) error) (err error)

Scan performs a prefix scan of keys matching the given prefix and calling the function `f` with the keys found. If the function returns an error no further keys are processed and the first error returned.

func (*Bitcask) Stats added in v0.2.2 

func (b *Bitcask) Stats() (stats Stats, err error)

Stats returns statistics about the database including the number of data files, keys and overall size on disk of the data

func (*Bitcask) Sync 

func (b *Bitcask) Sync() error

Sync flushes all buffers to disk ensuring all data is written

type Option 

type Option func(*config.Config) error

Option is a function that takes a config struct and modifies it

func WithAutoRecovery added in v0.3.6 

func WithAutoRecovery(enabled bool) Option

WithAutoRecovery sets auto recovery of data and index file recreation. IMPORTANT: This flag MUST BE used only if a proper backup was made of all the existing datafiles.

func WithDirFileModeBeforeUmask added in v0.3.7 

func WithDirFileModeBeforeUmask(mode os.FileMode) Option

WithDirFileModeBeforeUmask sets the FileMode used for each new file created.

func WithFileFileModeBeforeUmask added in v0.3.7 

func WithFileFileModeBeforeUmask(mode os.FileMode) Option

WithFileFileModeBeforeUmask sets the FileMode used for each new file created.

func WithMaxDatafileSize 

func WithMaxDatafileSize(size int) Option

WithMaxDatafileSize sets the maximum datafile size option

func WithMaxKeySize 

func WithMaxKeySize(size uint32) Option

WithMaxKeySize sets the maximum key size option

func WithMaxValueSize 

func WithMaxValueSize(size uint64) Option

WithMaxValueSize sets the maximum value size option

func WithSync added in v0.3.3 

func WithSync(sync bool) Option

WithSync causes Sync() to be called on every key/value written increasing durability and safety at the expense of performance

type Stats added in v0.2.2 

type Stats struct {
	Datafiles int
	Keys      int
	Size      int64
}

Stats is a struct returned by Stats() on an open Bitcask instance

Source Files

View all Source files

Directories

Path Synopsis
cmd
bitcask
bitcaskd
config
data
data/codec
index
metadata
mocks

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.