kvbtree

Documentation

Overview

Package kvbtree

Index

• type KVBTreeBucket
  • func NewKVMemBucket(clientID, bucketID string) *KVBTreeBucket
  • func NewKVMemBucketFromMap(clientID, bucketID string, data map[string][]byte) *KVBTreeBucket
  • func (bucket *KVBTreeBucket) Close() (err error)
  • func (bucket *KVBTreeBucket) Cursor(ctx context.Context) (buckets.IBucketCursor, error)
  • func (bucket *KVBTreeBucket) Delete(key string) error
  • func (bucket *KVBTreeBucket) Export() map[string][]byte
  • func (bucket *KVBTreeBucket) Get(key string) (val []byte, err error)
  • func (bucket *KVBTreeBucket) GetMultiple(keys []string) (docs map[string][]byte, err error)
  • func (bucket *KVBTreeBucket) ID() string
  • func (bucket *KVBTreeBucket) Info() (info *buckets.BucketStoreInfo)
  • func (bucket *KVBTreeBucket) Set(key string, doc []byte) error
  • func (bucket *KVBTreeBucket) SetMultiple(docs map[string][]byte) (err error)
• type KVBTreeCursor
  • func NewKVCursor(ctx context.Context, bucket buckets.IBucket, ...) *KVBTreeCursor
  • func (cursor *KVBTreeCursor) BucketID() string
  • func (cursor *KVBTreeCursor) Context() context.Context
  • func (cursor *KVBTreeCursor) First() (key string, value []byte, valid bool)
  • func (cursor *KVBTreeCursor) Last() (key string, value []byte, valid bool)
  • func (cursor *KVBTreeCursor) Next() (key string, value []byte, valid bool)
  • func (cursor *KVBTreeCursor) NextN(steps uint) (docs map[string][]byte, itemsRemaining bool)
  • func (cursor *KVBTreeCursor) Prev() (key string, value []byte, valid bool)
  • func (cursor *KVBTreeCursor) PrevN(steps uint) (docs map[string][]byte, itemsRemaining bool)
  • func (cursor *KVBTreeCursor) Release()
  • func (cursor *KVBTreeCursor) Seek(searchKey string) (key string, value []byte, valid bool)
• type KVBTreeStore
  • func NewKVStore(clientID, storePath string) (store *KVBTreeStore)
  • func (store *KVBTreeStore) Close() error
  • func (store *KVBTreeStore) Export() map[string]map[string][]byte
  • func (store *KVBTreeStore) GetBucket(bucketID string) (bucket buckets.IBucket)
  • func (store *KVBTreeStore) Open() error
  • func (store *KVBTreeStore) SetWriteDelay(delay time.Duration)

Types

type KVBTreeBucket

type KVBTreeBucket struct {
	BucketID string `json:"bucketID"`
	ClientID string `json:"clientID"`
	// contains filtered or unexported fields
}

KVBTreeBucket is an in-memory bucket for the KVBTreeBucket

func NewKVMemBucket

func NewKVMemBucket(clientID, bucketID string) *KVBTreeBucket

func NewKVMemBucketFromMap

func NewKVMemBucketFromMap(clientID, bucketID string, data map[string][]byte) *KVBTreeBucket

NewKVMemBucketFromMap creates a new KVBTreeBucket from a map with bucket data Intended for loading a saved store.

func (*KVBTreeBucket) Close

func (bucket *KVBTreeBucket) Close() (err error)

Close the bucket and release its resources commit is not used as this store doesn't handle transactions. This decreases the refCount and detects an error if below 0

func (*KVBTreeBucket) Cursor

func (bucket *KVBTreeBucket) Cursor(
	ctx context.Context) (buckets.IBucketCursor, error)

Cursor returns a new cursor for iterating the bucket. The cursor MUST be closed after use to release its memory.

This implementation is brute force. It generates a sorted list of key/values for use by the cursor. The cursor makes a shallow copy of the store. Store mutations are not reflected in the cursor.

This should be fast enough for many use-cases. 100K records takes around 27msec on an i5@2.9GHz

This returns a cursor with Next() and Prev() iterators

func (*KVBTreeBucket) Delete

func (bucket *KVBTreeBucket) Delete(key string) error

Delete a document from the bucket Also succeeds if the document doesn't exist

func (*KVBTreeBucket) Export

func (bucket *KVBTreeBucket) Export() map[string][]byte

Export returns a shallow copy of the bucket content

func (*KVBTreeBucket) Get

func (bucket *KVBTreeBucket) Get(key string) (val []byte, err error)

Get an object by its ID returns an error if the key does not exist.

func (*KVBTreeBucket) GetMultiple

func (bucket *KVBTreeBucket) GetMultiple(keys []string) (docs map[string][]byte, err error)

GetMultiple returns a batch of documents for the given key The document can be any text.

func (*KVBTreeBucket) ID

func (bucket *KVBTreeBucket) ID() string

func (*KVBTreeBucket) Info

func (bucket *KVBTreeBucket) Info() (info *buckets.BucketStoreInfo)

Info returns the bucket info

func (*KVBTreeBucket) Set

func (bucket *KVBTreeBucket) Set(key string, doc []byte) error

Set writes a document to the store. If the document exists it is replaced. This will store a copy of doc

A background process periodically checks the change count. When increased:
1. Lock the store while copying the index. Unlock when done.
2. Stream the in-memory json documents to a temp file.
3. If success, move the temp file to the store file using the OS atomic move operation.

func (*KVBTreeBucket) SetMultiple

func (bucket *KVBTreeBucket) SetMultiple(docs map[string][]byte) (err error)

SetMultiple writes a batch of key-values Values are copied

type KVBTreeCursor

type KVBTreeCursor struct {
	// contains filtered or unexported fields
}

func NewKVCursor

func NewKVCursor(ctx context.Context, bucket buckets.IBucket, kvIter btree.MapIter[string, []byte]) *KVBTreeCursor

NewKVCursor create a new bucket cursor for the KV store. Cursor.Close() must be called to release the resources.

bucket is the bucket holding the data
orderedKeys is a snapshot of the keys in ascending order

func NewKVCursor(bucket bucketstore.IBucket, orderedKeys []string, kvbtree btree.Map[string, []byte]) *KVBTreeCursor {

func (*KVBTreeCursor) BucketID

func (cursor *KVBTreeCursor) BucketID() string

func (*KVBTreeCursor) Context

func (cursor *KVBTreeCursor) Context() context.Context

Context returns the cursor application context

func (*KVBTreeCursor) First

func (cursor *KVBTreeCursor) First() (key string, value []byte, valid bool)

First moves the cursor to the first item

func (*KVBTreeCursor) Last

func (cursor *KVBTreeCursor) Last() (key string, value []byte, valid bool)

Last moves the cursor to the last item

func (*KVBTreeCursor) Next

func (cursor *KVBTreeCursor) Next() (key string, value []byte, valid bool)

Next increases the cursor position and return the next key and value If the end is reached the returned key is empty

func (*KVBTreeCursor) NextN

func (cursor *KVBTreeCursor) NextN(steps uint) (docs map[string][]byte, itemsRemaining bool)

NextN increases the cursor position N times and return the encountered key-value pairs

func (*KVBTreeCursor) Prev

func (cursor *KVBTreeCursor) Prev() (key string, value []byte, valid bool)

Prev decreases the cursor position and return the previous key and value If the head is reached the returned key is empty

func (*KVBTreeCursor) PrevN

func (cursor *KVBTreeCursor) PrevN(steps uint) (docs map[string][]byte, itemsRemaining bool)

PrevN decreases the cursor position N times and return the encountered key-value pairs

func (*KVBTreeCursor) Release

func (cursor *KVBTreeCursor) Release()

Release the cursor capability

func (*KVBTreeCursor) Seek

func (cursor *KVBTreeCursor) Seek(searchKey string) (key string, value []byte, valid bool)

Seek positions the cursor at the given searchKey. This implementation is brute force. It generates a sorted list of orderedKeys for use by the cursor. This should still be fast enough for most cases. (test shows around 500msec for 1 million orderedKeys).

BucketID to seach for. Returns and error if the bucket is not found
key is the starting point. If key doesn't exist, the next closest key will be used.

This returns a cursor with Next() and Prev() iterators

type KVBTreeStore

type KVBTreeStore struct {
	// contains filtered or unexported fields
}

KVBTreeStore is an embedded, file backed, in-memory, lightweight, and very fast key-value bucket store. This is intended for simple use-cases with up to 100K records. Interestingly, this simple brute-force store using maps is faster than anything else I've tested and even scales up to 1M records. Pretty much all you need for basic databases.

Changes are periodically persisted to file in the background.

Limitations:

• No transaction support (basic usage, remember)
• Changes are periodically (default 3 second) written to disk

Improvements for future considerations:

• Append-only to reduce disk writes for larger databases

--- about jsonpath --- This was experimental because of the W3C WoT recommendation, and seems to work well. However this is shelved as the Hub has no use-case for it and the other stores don't support it.

Query: jsonPath: `$[?(@.properties.title.name=="title1")]`

Approx 1.8 msec with a dataset of 1K records (1 result)
Approx 23 msec with a dataset of 10K records (1 result)
Approx 260 msec with a dataset of 100K records (1 result)

A good overview of jsonpath implementations can be found here: > https://cburgmer.github.io/json-path-comparison/ Two good options for jsonpath queries:

> github.com/ohler55/ojg/jp
> github.com/PaesslerAG/jsonpath

Note that future implementations of this service can change the storage media used while maintaining API compatibility.

func NewKVStore

func NewKVStore(clientID, storePath string) (store *KVBTreeStore)

NewKVStore creates a store instance and load it with saved documents. Run Connect to start the background loop and Stop to end it.

SenderID service or user for debugging and logging
storeFile path to storage file or "" for in-memory only

func (*KVBTreeStore) Close

func (store *KVBTreeStore) Close() error

Close the store and stop the background update. If any changes are remaining then write to disk now.

func (*KVBTreeStore) Export

func (store *KVBTreeStore) Export() map[string]map[string][]byte

export returns a map of the given bucket this copies the keys and values into a new map

func (*KVBTreeStore) GetBucket

func (store *KVBTreeStore) GetBucket(bucketID string) (bucket buckets.IBucket)

GetBucket returns a bucket and creates it if it doesn't exist

func (*KVBTreeStore) Open

func (store *KVBTreeStore) Open() error

Open the store and start the background loop for saving changes

func (*KVBTreeStore) SetWriteDelay

func (store *KVBTreeStore) SetWriteDelay(delay time.Duration)

SetWriteDelay sets the delay for writing after a change