scdb

Documentation

Index

• type Store
  • func New(path string, maxKeys *uint64, redundantBlocks *uint16, poolCapacity *uint64, ...) (*Store, error)
  • func (s *Store) Clear() error
  • func (s *Store) Close() error
  • func (s *Store) Compact() error
  • func (s *Store) Delete(k []byte) error
  • func (s *Store) Get(k []byte) ([]byte, error)
  • func (s *Store) Search(term []byte, skip uint64, limit uint64) ([]buffers.KeyValuePair, error)
  • func (s *Store) Set(k []byte, v []byte, ttl *uint64) error

Examples

• New
• Store.Delete
• Store.Get
• Store.Search
• Store.Set

Types

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store is a key-value store that persists key-value pairs to disk

Store behaves like a HashMap that saves keys and value as byte arrays on disk. It allows for specifying how long each key-value pair should be kept for i.e. the time-to-live in seconds. If None is provided, they last indefinitely.

func New

func New(path string, maxKeys *uint64, redundantBlocks *uint16, poolCapacity *uint64, compactionInterval *uint32, isSearchEnabled bool) (*Store, error)

New creates a new Store at the given path The Store has a number of configurations that are passed into this New function

• `storePath` - required: The path to a directory where scdb should store its data

• `maxKeys` - default: 1 million: The maximum number of key-value pairs to store in store

• `redundantBlocks` - default: 1: The store has an index to hold all the keys. This index is split into a fixed number of blocks basing on the virtual memory page size and the total number of keys to be held i.e. `max_keys`. Sometimes, there may be hash collision errors as the store's current stored keys approach `max_keys`. The closer it gets, the more it becomes likely see those errors. Adding redundant blocks helps mitigate this. Just be careful to not add too many (i.e. more than 2) since the higher the number of these blocks, the slower the store becomes.

• `poolCapacity` - default: 5: The number of buffers to hold in memory as cache's for the store. Each buffer has the size equal to the virtual memory's page size, usually 4096 bytes. Increasing this number will speed this store up but of course, the machine has a limited RAM. When this number increases to a value that clogs the RAM, performance suddenly degrades, and keeps getting worse from there on.

• `compactionInterval` - default 3600s (1 hour): The interval at which the store is compacted to remove dangling keys. Dangling keys result from either getting expired or being deleted. When a `delete` operation is done, the actual key-value pair is just marked as `deleted` but is not removed. Something similar happens when a key-value is updated. A new key-value pair is created and the old one is left unindexed. Compaction is important because it reclaims this space and reduces the size of the database file.

• `isSearchEnabled` - default false: Whether the search capability of the store is enabled. Note that when search is enabled, `set`, `delete`, `clear`, `compact` operations become slower.

Example

var maxKeys uint64 = 1_000_000
var redundantBlocks uint16 = 1
var poolCapacity uint64 = 10
var compactionInterval uint32 = 1_800

store, err := New(
	"testdb",
	&maxKeys,
	&redundantBlocks,
	&poolCapacity,
	&compactionInterval,
	true)
if err != nil {
	log.Fatalf("error opening store: %s", err)
}
defer func() {
	_ = store.Close()
}()

func (*Store) Clear

func (s *Store) Clear() error

Clear removes all data in the store

func (*Store) Close

func (s *Store) Close() error

Close frees up any resources occupied by store. After this, the store is unusable. You have to re-instantiate it or just run into some crazy errors

func (*Store) Compact

func (s *Store) Compact() error

Compact manually removes dangling key-value pairs in the database file

Dangling keys result from either getting expired or being deleted. When a Store.Delete operation is done, the actual key-value pair is just marked as `deleted` but is not removed.

Something similar happens when a key-value is updated. A new key-value pair is created and the old one is left un-indexed. Compaction is important because it reclaims this space and reduces the size of the database file.

This is done automatically for you at the set `compactionInterval` but you may wish to do it manually for some reason.

This is a very expensive operation so use it sparingly.

func (*Store) Delete

func (s *Store) Delete(k []byte) error

Delete removes the key-value for the given key

Example

store, err := New("testdb", nil, nil, nil, nil, false)
if err != nil {
	log.Fatalf("error opening store: %s", err)
}
defer func() {
	_ = store.Close()
}()

err = store.Delete([]byte("foo"))
if err != nil {
	log.Fatalf("error deleting key: %s", err)
}

func (*Store) Get

func (s *Store) Get(k []byte) ([]byte, error)

Get returns the value corresponding to the given key

Example

store, err := New("testdb", nil, nil, nil, nil, false)
if err != nil {
	log.Fatalf("error opening store: %s", err)
}
defer func() {
	_ = store.Close()
}()

err = store.Set([]byte("foo"), []byte("bar"), nil)
if err != nil {
	log.Fatalf("error setting key value: %s", err)
}

value, err := store.Get([]byte("foo"))
if err != nil {
	log.Fatalf("error getting key: %s", err)
}

fmt.Printf("%s", value)

Output:

bar

func (*Store) Search

func (s *Store) Search(term []byte, skip uint64, limit uint64) ([]buffers.KeyValuePair, error)

Search searches for unexpired keys that start with the given search term

It skips the first `skip` (default: 0) number of results and returns not more than `limit` (default: 0) number of items. This is to avoid using up more memory than can be handled by the host machine.

If `limit` is 0, all items are returned since it would make no sense for someone to search for zero items.

returns a list of pairs of key-value i.e. `buffers.KeyValuePair`

Example

store, err := New("testdb", nil, nil, nil, nil, true)
if err != nil {
	log.Fatalf("error opening store: %s", err)
}
defer func() {
	_ = store.Close()
}()

data := []buffers.KeyValuePair{
	{K: []byte("hi"), V: []byte("ooliyo")},
	{K: []byte("high"), V: []byte("haiguru")},
	{K: []byte("hind"), V: []byte("enyuma")},
	{K: []byte("hill"), V: []byte("akasozi")},
	{K: []byte("him"), V: []byte("ogwo")},
}

for _, rec := range data {
	err = store.Set(rec.K, rec.V, nil)
	if err != nil {
		log.Fatalf("error setting key value: %s", err)
	}
}

// without pagination
kvs, err := store.Search([]byte("hi"), 0, 0)
if err != nil {
	log.Fatalf("error searching 'hi': %s", err)
}

fmt.Printf("\nno pagination: %v", kvs)

// with pagination: get last three
kvs, err = store.Search([]byte("hi"), 2, 3)
if err != nil {
	log.Fatalf("error searching (paginated) 'hi': %s", err)
}

fmt.Printf("\nskip 2, limit 3: %v", kvs)

Output:

no pagination: [hi: ooliyo high: haiguru hind: enyuma hill: akasozi him: ogwo]
skip 2, limit 3: [hind: enyuma hill: akasozi him: ogwo]

func (*Store) Set

func (s *Store) Set(k []byte, v []byte, ttl *uint64) error

Set sets the given key value in the store This is used to insert or update any key-value pair in the store

Example

store, err := New("testdb", nil, nil, nil, nil, false)
if err != nil {
	log.Fatalf("error opening store: %s", err)
}
defer func() {
	_ = store.Close()
}()

err = store.Set([]byte("foo"), []byte("bar"), nil)
if err != nil {
	log.Fatalf("error setting key value without ttl: %s", err)
}

ttl := uint64(3_600)
err = store.Set([]byte("fake"), []byte("bear"), &ttl)
if err != nil {
	log.Fatalf("error setting key value with ttl: %s", err)
}