walletdb

README

walletdb

Package walletdb provides a namespaced database interface for bchwallet.

A wallet essentially consists of a multitude of stored data such as private and public keys, key derivation bits, pay-to-script-hash scripts, and various metadata. One of the issues with many wallets is they are tightly integrated. Designing a wallet with loosely coupled components that provide specific functionality is ideal, however it presents a challenge in regards to data storage since each component needs to store its own data without knowing the internals of other components or breaking atomicity.

This package solves this issue by providing a namespaced database interface that is intended to be used by the main wallet daemon. This allows the potential for any backend database type with a suitable driver. Each component, which will typically be a package, can then implement various functionality such as address management, voting pools, and colored coin metadata in their own namespace without having to worry about conflicts with other packages even though they are sharing the same database that is managed by the wallet.

A suite of tests is provided to ensure proper functionality. See test_coverage.txt for the gocov coverage report. Alternatively, if you are running a POSIX OS, you can run the cov_report.sh script for a real-time report. Package walletdb is licensed under the copyfree ISC license.

This interfaces provided by this package were heavily inspired by the excellent boltdb project at https://github.com/boltdb/bolt by Ben B. Johnson.

Feature Overview

• Key/value store
• Namespace support
  • Allows multiple packages to have their own area in the database without worrying about conflicts
• Read-only and read-write transactions with both manual and managed modes
• Nested buckets
• Supports registration of backend databases
• Comprehensive test coverage

Documentation

[] (http://godoc.org/github.com/gcash/bchwallet/walletdb)

Full go doc style documentation for the project can be viewed online without installing this package by using the GoDoc site here: http://godoc.org/github.com/gcash/bchwallet/walletdb

You can also view the documentation locally once the package is installed with the godoc tool by running godoc -http=":6060" and pointing your browser to http://localhost:6060/pkg/github.com/gcash/bchwallet/walletdb

Installation

$ go get github.com/gcash/bchwallet/walletdb

Examples

• [Basic Usage Example] (http://godoc.org/github.com/gcash/bchwallet/walletdb#example-package--BasicUsage) Demonstrates creating a new database, getting a namespace from it, and using a managed read-write transaction against the namespace to store and retrieve data.

License

Package walletdb is licensed under the copyfree ISC License.

Documentation

Overview

Package walletdb provides a namespaced database interface for bchwallet.

Overview

A wallet essentially consists of a multitude of stored data such as private and public keys, key derivation bits, pay-to-script-hash scripts, and various metadata. One of the issues with many wallets is they are tightly integrated. Designing a wallet with loosely coupled components that provide specific functionality is ideal, however it presents a challenge in regards to data storage since each component needs to store its own data without knowing the internals of other components or breaking atomicity.

This package solves this issue by providing a pluggable driver, namespaced database interface that is intended to be used by the main wallet daemon. This allows the potential for any backend database type with a suitable driver. Each component, which will typically be a package, can then implement various functionality such as address management, voting pools, and colored coin metadata in their own namespace without having to worry about conflicts with other packages even though they are sharing the same database that is managed by the wallet.

A quick overview of the features walletdb provides are as follows:

• Key/value store
• Namespace support
• Allows multiple packages to have their own area in the database without worrying about conflicts
• Read-only and read-write transactions with both manual and managed modes
• Nested buckets
• Supports registration of backend databases
• Comprehensive test coverage

Database

The main entry point is the DB interface. It exposes functionality for creating, retrieving, and removing namespaces. It is obtained via the Create and Open functions which take a database type string that identifies the specific database driver (backend) to use as well as arguments specific to the specified driver.

Namespaces

The Namespace interface is an abstraction that provides facilities for obtaining transactions (the Tx interface) that are the basis of all database reads and writes. Unlike some database interfaces that support reading and writing without transactions, this interface requires transactions even when only reading or writing a single key.

The Begin function provides an unmanaged transaction while the View and Update functions provide a managed transaction. These are described in more detail below.

Transactions

The Tx interface provides facilities for rolling back or commiting changes that took place while the transaction was active. It also provides the root bucket under which all keys, values, and nested buckets are stored. A transaction can either be read-only or read-write and managed or unmanaged.

Managed versus Unmanaged Transactions

A managed transaction is one where the caller provides a function to execute within the context of the transaction and the commit or rollback is handled automatically depending on whether or not the provided function returns an error. Attempting to manually call Rollback or Commit on the managed transaction will result in a panic.

An unmanaged transaction, on the other hand, requires the caller to manually call Commit or Rollback when they are finished with it. Leaving transactions open for long periods of time can have several adverse effects, so it is recommended that managed transactions are used instead.

Buckets

The Bucket interface provides the ability to manipulate key/value pairs and nested buckets as well as iterate through them.

The Get, Put, and Delete functions work with key/value pairs, while the Bucket, CreateBucket, CreateBucketIfNotExists, and DeleteBucket functions work with buckets. The ForEach function allows the caller to provide a function to be called with each key/value pair and nested bucket in the current bucket.

Root Bucket

As discussed above, all of the functions which are used to manipulate key/value pairs and nested buckets exist on the Bucket interface. The root bucket is the upper-most bucket in a namespace under which data is stored and is created at the same time as the namespace. Use the RootBucket function on the Tx interface to retrieve it.

Nested Buckets

The CreateBucket and CreateBucketIfNotExists functions on the Bucket interface provide the ability to create an arbitrary number of nested buckets. It is a good idea to avoid a lot of buckets with little data in them as it could lead to poor page utilization depending on the specific driver in use.

Example (BasicUsage)

This example demonstrates creating a new database, getting a namespace from it, and using a managed read-write transaction against the namespace to store and retrieve data.

package main

import (
	"bytes"
	"fmt"
	"os"
	"path/filepath"

	"github.com/gcash/bchwallet/walletdb"
	_ "github.com/gcash/bchwallet/walletdb/bdb"
)

func main() {
	// This example assumes the bdb (bolt db) driver is imported.
	//
	// import (
	// 	"github.com/gcash/bchwallet/walletdb"
	// 	_ "github.com/gcash/bchwallet/walletdb/bdb"
	// )

	// Create a database and schedule it to be closed and removed on exit.
	// Typically you wouldn't want to remove the database right away like
	// this, but it's done here in the example to ensure the example cleans
	// up after itself.
	dbPath := filepath.Join(os.TempDir(), "exampleusage.db")
	db, err := walletdb.Create("bdb", dbPath, true)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer os.Remove(dbPath)
	defer db.Close()

	// Get or create a bucket in the database as needed.  This bucket
	// is what is typically passed to specific sub-packages so they have
	// their own area to work in without worrying about conflicting keys.
	bucketKey := []byte("walletsubpackage")
	err = walletdb.Update(db, func(tx walletdb.ReadWriteTx) error {
		bucket := tx.ReadWriteBucket(bucketKey)
		if bucket == nil {
			_, err = tx.CreateTopLevelBucket(bucketKey)
			if err != nil {
				return err
			}
		}
		return nil
	})
	if err != nil {
		fmt.Println(err)
		return
	}

	// Use the Update function of the namespace to perform a managed
	// read-write transaction.  The transaction will automatically be rolled
	// back if the supplied inner function returns a non-nil error.
	err = walletdb.Update(db, func(tx walletdb.ReadWriteTx) error {
		// All data is stored against the root bucket of the namespace,
		// or nested buckets of the root bucket.  It's not really
		// necessary to store it in a separate variable like this, but
		// it has been done here for the purposes of the example to
		// illustrate.
		rootBucket := tx.ReadWriteBucket(bucketKey)

		// Store a key/value pair directly in the root bucket.
		key := []byte("mykey")
		value := []byte("myvalue")
		if err := rootBucket.Put(key, value); err != nil {
			return err
		}

		// Read the key back and ensure it matches.
		if !bytes.Equal(rootBucket.Get(key), value) {
			return fmt.Errorf("unexpected value for key '%s'", key)
		}

		// Create a new nested bucket under the root bucket.
		nestedBucketKey := []byte("mybucket")
		nestedBucket, err := rootBucket.CreateBucket(nestedBucketKey)
		if err != nil {
			return err
		}

		// The key from above that was set in the root bucket does not
		// exist in this new nested bucket.
		if nestedBucket.Get(key) != nil {
			return fmt.Errorf("key '%s' is not expected nil", key)
		}

		return nil
	})
	if err != nil {
		fmt.Println(err)
		return
	}

}

Index

• Variables
• func Batch(db DB, f func(tx ReadWriteTx) error) error
• func BucketIsEmpty(bucket ReadBucket) bool
• func RegisterDriver(driver Driver) error
• func SupportedDrivers() []string
• func Update(db DB, f func(tx ReadWriteTx) error) error
• func View(db DB, f func(tx ReadTx) error) error
• type BatchDB
• type DB
  • func Create(dbType string, args ...interface{}) (DB, error)
  • func Open(dbType string, args ...interface{}) (DB, error)
• type Driver
• type ReadBucket
• type ReadCursor
• type ReadTx
• type ReadWriteBucket
• type ReadWriteCursor
• type ReadWriteTx

Examples

• Package (BasicUsage)
• Create
• DB (CreateTopLevelBucket)

Variables

var (
	// ErrDbUnknownType is returned when there is no driver registered for
	// the specified database type.
	ErrDbUnknownType = errors.New("unknown database type")

	// ErrDbDoesNotExist is returned when open is called for a database that
	// does not exist.
	ErrDbDoesNotExist = errors.New("database does not exist")

	// ErrDbExists is returned when create is called for a database that
	// already exists.
	ErrDbExists = errors.New("database already exists")

	// ErrDbNotOpen is returned when a database instance is accessed before
	// it is opened or after it is closed.
	ErrDbNotOpen = errors.New("database not open")

	// ErrDbAlreadyOpen is returned when open is called on a database that
	// is already open.
	ErrDbAlreadyOpen = errors.New("database already open")

	// ErrInvalid is returned if the specified database is not valid.
	ErrInvalid = errors.New("invalid database")
)

Errors that the various database functions may return.

var (
	// ErrTxClosed is returned when attempting to commit or rollback a
	// transaction that has already had one of those operations performed.
	ErrTxClosed = errors.New("tx closed")

	// ErrTxNotWritable is returned when an operation that requires write
	// access to the database is attempted against a read-only transaction.
	ErrTxNotWritable = errors.New("tx not writable")
)

Errors that can occur when beginning or committing a transaction.

var (
	// ErrBucketNotFound is returned when trying to access a bucket that has
	// not been created yet.
	ErrBucketNotFound = errors.New("bucket not found")

	// ErrBucketExists is returned when creating a bucket that already exists.
	ErrBucketExists = errors.New("bucket already exists")

	// ErrBucketNameRequired is returned when creating a bucket with a blank name.
	ErrBucketNameRequired = errors.New("bucket name required")

	// ErrKeyRequired is returned when inserting a zero-length key.
	ErrKeyRequired = errors.New("key required")

	// ErrKeyTooLarge is returned when inserting a key that is larger than MaxKeySize.
	ErrKeyTooLarge = errors.New("key too large")

	// ErrValueTooLarge is returned when inserting a value that is larger than MaxValueSize.
	ErrValueTooLarge = errors.New("value too large")

	// ErrIncompatibleValue is returned when trying create or delete a
	// bucket on an existing non-bucket key or when trying to create or
	// delete a non-bucket key on an existing bucket key.
	ErrIncompatibleValue = errors.New("incompatible value")
)

Errors that can occur when putting or deleting a value or bucket.

var (
	// ErrDbTypeRegistered is returned when two different database drivers
	// attempt to register with the name database type.
	ErrDbTypeRegistered = errors.New("database type already registered")
)

Errors that can occur during driver registration.

Functions

func Batch

func Batch(db DB, f func(tx ReadWriteTx) error) error

Batch opens a database read/write transaction and executes the function f with the transaction passed as a parameter. After f exits, if f did not error, the transaction is committed. Otherwise, if f did error, the transaction is rolled back. If the rollback fails, the original error returned by f is still returned. If the commit fails, the commit error is returned.

Batch is only useful when there are multiple goroutines calling it.

func BucketIsEmpty

func BucketIsEmpty(bucket ReadBucket) bool

BucketIsEmpty returns whether the bucket is empty, that is, whether there are no key/value pairs or nested buckets.

func RegisterDriver

func RegisterDriver(driver Driver) error

RegisterDriver adds a backend database driver to available interfaces. ErrDbTypeRegistered will be retruned if the database type for the driver has already been registered.

func SupportedDrivers

func SupportedDrivers() []string

SupportedDrivers returns a slice of strings that represent the database drivers that have been registered and are therefore supported.

func Update

func Update(db DB, f func(tx ReadWriteTx) error) error

Update opens a database read/write transaction and executes the function f with the transaction passed as a parameter. After f exits, if f did not error, the transaction is committed. Otherwise, if f did error, the transaction is rolled back. If the rollback fails, the original error returned by f is still returned. If the commit fails, the commit error is returned.

func View

func View(db DB, f func(tx ReadTx) error) error

View opens a database read transaction and executes the function f with the transaction passed as a parameter. After f exits, the transaction is rolled back. If f errors, its error is returned, not a rollback error (if any occur).

Types

type BatchDB

type BatchDB interface {
	DB

	// Batch is similar to the package-level Update method, but it will
	// attempt to optismitcally combine the invocation of several
	// transaction functions into a single db write transaction.
	Batch(func(tx ReadWriteTx) error) error
}

BatchDB is a special version of the main DB interface that allos the caller to specify write transactions that should be combine dtoegether if multiple goroutines are calling the Batch method.

type DB

type DB interface {
	// BeginReadTx opens a database read transaction.
	BeginReadTx() (ReadTx, error)

	// BeginReadWriteTx opens a database read+write transaction.
	BeginReadWriteTx() (ReadWriteTx, error)

	// Copy writes a copy of the database to the provided writer.  This
	// call will start a read-only transaction to perform all operations.
	Copy(w io.Writer) error

	// Close cleanly shuts down the database and syncs all data.
	Close() error
}

DB represents an ACID database. All database access is performed through read or read+write transactions.

Example (CreateTopLevelBucket)

This example demonstrates creating a new top level bucket.

package main

import (
	"fmt"
	"os"
	"path/filepath"

	"github.com/gcash/bchwallet/walletdb"
	_ "github.com/gcash/bchwallet/walletdb/bdb"
)

// exampleNum is used as a counter in the exampleLoadDB function to provided
// a unique database name for each example.
var exampleNum = 0

// exampleLoadDB is used in the examples to elide the setup code.
func exampleLoadDB() (walletdb.DB, func(), error) {
	dbName := fmt.Sprintf("exampleload%d.db", exampleNum)
	dbPath := filepath.Join(os.TempDir(), dbName)
	db, err := walletdb.Create("bdb", dbPath, true)
	if err != nil {
		return nil, nil, err
	}
	teardownFunc := func() {
		db.Close()
		os.Remove(dbPath)
	}
	exampleNum++

	return db, teardownFunc, err
}

func main() {
	// Load a database for the purposes of this example and schedule it to
	// be closed and removed on exit. See the Create example for more
	// details on what this step is doing.
	db, teardownFunc, err := exampleLoadDB()
	if err != nil {
		fmt.Println(err)
		return
	}
	defer teardownFunc()

	dbtx, err := db.BeginReadWriteTx()
	if err != nil {
		fmt.Println(err)
		return
	}
	defer dbtx.Commit()

	// Get or create a bucket in the database as needed.  This bucket
	// is what is typically passed to specific sub-packages so they have
	// their own area to work in without worrying about conflicting keys.
	bucketKey := []byte("walletsubpackage")
	bucket, err := dbtx.CreateTopLevelBucket(bucketKey)
	if err != nil {
		fmt.Println(err)
		return
	}

	// Prevent unused error.
	_ = bucket

}

func Create

func Create(dbType string, args ...interface{}) (DB, error)

Create intializes and opens a database for the specified type. The arguments are specific to the database type driver. See the documentation for the database driver for further details.

ErrDbUnknownType will be returned if the the database type is not registered.

Example

This example demonstrates creating a new database.

package main

import (
	"fmt"
	"os"
	"path/filepath"

	"github.com/gcash/bchwallet/walletdb"
	_ "github.com/gcash/bchwallet/walletdb/bdb"
)

func main() {
	// This example assumes the bdb (bolt db) driver is imported.
	//
	// import (
	// 	"github.com/gcash/bchwallet/walletdb"
	// 	_ "github.com/gcash/bchwallet/walletdb/bdb"
	// )

	// Create a database and schedule it to be closed and removed on exit.
	// Typically you wouldn't want to remove the database right away like
	// this, but it's done here in the example to ensure the example cleans
	// up after itself.
	dbPath := filepath.Join(os.TempDir(), "examplecreate.db")
	db, err := walletdb.Create("bdb", dbPath, true)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer os.Remove(dbPath)
	defer db.Close()

}

func Open

func Open(dbType string, args ...interface{}) (DB, error)

Open opens an existing database for the specified type. The arguments are specific to the database type driver. See the documentation for the database driver for further details.

ErrDbUnknownType will be returned if the the database type is not registered.

type Driver

type Driver struct {
	// DbType is the identifier used to uniquely identify a specific
	// database driver.  There can be only one driver with the same name.
	DbType string

	// Create is the function that will be invoked with all user-specified
	// arguments to create the database.  This function must return
	// ErrDbExists if the database already exists.
	Create func(args ...interface{}) (DB, error)

	// Open is the function that will be invoked with all user-specified
	// arguments to open the database.  This function must return
	// ErrDbDoesNotExist if the database has not already been created.
	Open func(args ...interface{}) (DB, error)
}

Driver defines a structure for backend drivers to use when they registered themselves as a backend which implements the Db interface.

type ReadBucket

type ReadBucket interface {
	// NestedReadBucket retrieves a nested bucket with the given key.
	// Returns nil if the bucket does not exist.
	NestedReadBucket(key []byte) ReadBucket

	// ForEach invokes the passed function with every key/value pair in
	// the bucket.  This includes nested buckets, in which case the value
	// is nil, but it does not include the key/value pairs within those
	// nested buckets.
	//
	// NOTE: The values returned by this function are only valid during a
	// transaction.  Attempting to access them after a transaction has ended
	// results in undefined behavior.  This constraint prevents additional
	// data copies and allows support for memory-mapped database
	// implementations.
	ForEach(func(k, v []byte) error) error

	// Get returns the value for the given key.  Returns nil if the key does
	// not exist in this bucket (or nested buckets).
	//
	// NOTE: The value returned by this function is only valid during a
	// transaction.  Attempting to access it after a transaction has ended
	// results in undefined behavior.  This constraint prevents additional
	// data copies and allows support for memory-mapped database
	// implementations.
	Get(key []byte) []byte

	ReadCursor() ReadCursor
}

ReadBucket represents a bucket (a hierarchical structure within the database) that is only allowed to perform read operations.

type ReadCursor

type ReadCursor interface {
	// First positions the cursor at the first key/value pair and returns
	// the pair.
	First() (key, value []byte)

	// Last positions the cursor at the last key/value pair and returns the
	// pair.
	Last() (key, value []byte)

	// Next moves the cursor one key/value pair forward and returns the new
	// pair.
	Next() (key, value []byte)

	// Prev moves the cursor one key/value pair backward and returns the new
	// pair.
	Prev() (key, value []byte)

	// Seek positions the cursor at the passed seek key.  If the key does
	// not exist, the cursor is moved to the next key after seek.  Returns
	// the new pair.
	Seek(seek []byte) (key, value []byte)
}

ReadCursor represents a bucket cursor that can be positioned at the start or end of the bucket's key/value pairs and iterate over pairs in the bucket. This type is only allowed to perform database read operations.

type ReadTx

type ReadTx interface {
	// ReadBucket opens the root bucket for read only access.  If the bucket
	// described by the key does not exist, nil is returned.
	ReadBucket(key []byte) ReadBucket

	// Rollback closes the transaction, discarding changes (if any) if the
	// database was modified by a write transaction.
	Rollback() error
}

ReadTx represents a database transaction that can only be used for reads. If a database update must occur, use a ReadWriteTx.

type ReadWriteBucket

type ReadWriteBucket interface {
	ReadBucket

	// NestedReadWriteBucket retrieves a nested bucket with the given key.
	// Returns nil if the bucket does not exist.
	NestedReadWriteBucket(key []byte) ReadWriteBucket

	// CreateBucket creates and returns a new nested bucket with the given
	// key.  Returns ErrBucketExists if the bucket already exists,
	// ErrBucketNameRequired if the key is empty, or ErrIncompatibleValue
	// if the key value is otherwise invalid for the particular database
	// implementation.  Other errors are possible depending on the
	// implementation.
	CreateBucket(key []byte) (ReadWriteBucket, error)

	// CreateBucketIfNotExists creates and returns a new nested bucket with
	// the given key if it does not already exist.  Returns
	// ErrBucketNameRequired if the key is empty or ErrIncompatibleValue
	// if the key value is otherwise invalid for the particular database
	// backend.  Other errors are possible depending on the implementation.
	CreateBucketIfNotExists(key []byte) (ReadWriteBucket, error)

	// DeleteNestedBucket removes a nested bucket with the given key.
	// Returns ErrTxNotWritable if attempted against a read-only transaction
	// and ErrBucketNotFound if the specified bucket does not exist.
	DeleteNestedBucket(key []byte) error

	// Put saves the specified key/value pair to the bucket.  Keys that do
	// not already exist are added and keys that already exist are
	// overwritten.  Returns ErrTxNotWritable if attempted against a
	// read-only transaction.
	Put(key, value []byte) error

	// Delete removes the specified key from the bucket.  Deleting a key
	// that does not exist does not return an error.  Returns
	// ErrTxNotWritable if attempted against a read-only transaction.
	Delete(key []byte) error

	// Cursor returns a new cursor, allowing for iteration over the bucket's
	// key/value pairs and nested buckets in forward or backward order.
	ReadWriteCursor() ReadWriteCursor

	// Tx returns the bucket's transaction.
	Tx() ReadWriteTx

	// NextSequence returns an autoincrementing integer for the bucket.
	NextSequence() (uint64, error)

	// SetSequence updates the sequence number for the bucket.
	SetSequence(v uint64) error

	// Sequence returns the current integer for the bucket without
	// incrementing it.
	Sequence() uint64
}

ReadWriteBucket represents a bucket (a hierarchical structure within the database) that is allowed to perform both read and write operations.

type ReadWriteCursor

type ReadWriteCursor interface {
	ReadCursor

	// Delete removes the current key/value pair the cursor is at without
	// invalidating the cursor.  Returns ErrIncompatibleValue if attempted
	// when the cursor points to a nested bucket.
	Delete() error
}

ReadWriteCursor represents a bucket cursor that can be positioned at the start or end of the bucket's key/value pairs and iterate over pairs in the bucket. This abstraction is allowed to perform both database read and write operations.

type ReadWriteTx

type ReadWriteTx interface {
	ReadTx

	// ReadWriteBucket opens the root bucket for read/write access.  If the
	// bucket described by the key does not exist, nil is returned.
	ReadWriteBucket(key []byte) ReadWriteBucket

	// CreateTopLevelBucket creates the top level bucket for a key if it
	// does not exist.  The newly-created bucket it returned.
	CreateTopLevelBucket(key []byte) (ReadWriteBucket, error)

	// DeleteTopLevelBucket deletes the top level bucket for a key.  This
	// errors if the bucket can not be found or the key keys a single value
	// instead of a bucket.
	DeleteTopLevelBucket(key []byte) error

	// Commit commits all changes that have been on the transaction's root
	// buckets and all of their sub-buckets to persistent storage.
	Commit() error

	// OnCommit takes a function closure that will be executed when the
	// transaction successfully gets committed.
	OnCommit(func())
}

ReadWriteTx represents a database transaction that can be used for both reads and writes. When only reads are necessary, consider using a ReadTx instead.