walletdb

README

walletdb

[] (https://travis-ci.org/btcsuite/btcwallet)

Package walletdb provides a namespaced database interface for btcwallet.

A wallet essentially consists of a multitude of stored data such as private and public keys, key derivation bits, pay-to-script-hash scripts, and various metadata. One of the issues with many wallets is they are tightly integrated. Designing a wallet with loosely coupled components that provide specific functionality is ideal, however it presents a challenge in regards to data storage since each component needs to store its own data without knowing the internals of other components or breaking atomicity.

This package solves this issue by providing a namespaced database interface that is intended to be used by the main wallet daemon. This allows the potential for any backend database type with a suitable driver. Each component, which will typically be a package, can then implement various functionality such as address management, voting pools, and colored coin metadata in their own namespace without having to worry about conflicts with other packages even though they are sharing the same database that is managed by the wallet.

A suite of tests is provided to ensure proper functionality. See test_coverage.txt for the gocov coverage report. Alternatively, if you are running a POSIX OS, you can run the cov_report.sh script for a real-time report. Package walletdb is licensed under the copyfree ISC license.

This interfaces provided by this package were heavily inspired by the excellent boltdb project at https://github.com/boltdb/bolt by Ben B. Johnson.

Feature Overview

• Key/value store
• Namespace support
  • Allows multiple packages to have their own area in the database without worrying about conflicts
• Read-only and read-write transactions with both manual and managed modes
• Nested buckets
• Supports registration of backend databases
• Comprehensive test coverage

Documentation

[] (http://godoc.org/github.com/cybriq/p9/walletmain/walletdb)

Full go doc style documentation for the project can be viewed online without installing this package by using the GoDoc site here: http://godoc.org/github.com/cybriq/p9/walletmain/walletdb

You can also view the documentation locally once the package is installed with the godoc tool by running godoc -http=":6060" and pointing your browser to http://localhost:6060/pkg/github.com/cybriq/p9/walletmain/walletdb

Installation

$ go get github.com/cybriq/p9/walletmain/walletdb

Examples

• [Basic Usage Example] (http://godoc.org/github.com/cybriq/p9/walletmain/walletdb#example-package--BasicUsage)
  Demonstrates creating a new database, getting a namespace from it, and using a managed read-write transaction against the namespace to store and retrieve data.

License

Package walletdb is licensed under the copyfree ISC License.

Documentation

Overview

Package walletdb provides a namespaced database interface for btcwallet.

Overview

A wallet essentially consists of a multitude of stored data such as private and public keys, key derivation bits, pay-to-script-hash scripts, and various metadata. One of the issues with many wallets is they are tightly integrated. Designing a wallet with loosely coupled components that provide specific functionality is ideal, however it presents a challenge in regards to data storage since each component needs to store its own data without knowing the internals of other components or breaking atomicity.

This package solves this issue by providing a pluggable driver, namespaced database interface that is intended to be used by the main wallet daemon. This allows the potential for any backend database type with a suitable driver. Each component, which will typically be a package, can then implement various functionality such as address management, voting pools, and colored coin metadata in their own namespace without having to worry about conflicts with other packages even though they are sharing the same database that is managed by the wallet.

A quick overview of the features walletdb provides are as follows:

• Key/value store

• Namespace support

• Allows multiple packages to have their own area in the database without worrying about conflicts

• Read-only and read-write transactions with both manual and managed modes

• Nested buckets

• Supports registration of backend databases

• Comprehensive test coverage

Database

The main entry point is the DB interface. It exposes functionality for creating, retrieving, and removing namespaces. It is obtained via the Create and Open functions which take a database type string that identifies the specific database driver (backend) to use as well as arguments specific to the specified driver.

Namespaces

The Namespace interface is an abstraction that provides facilities for obtaining transactions (the Tx interface) that are the basis of all database reads and writes. Unlike some database interfaces that support reading and writing without transactions, this interface requires transactions even when only reading or writing a single key.

The Begin function provides an unmanaged transaction while the View and Update functions provide a managed transaction. These are described in more detail below.

Transactions

The Tx interface provides facilities for rolling back or commiting changes that took place while the transaction was active. It also provides the root bucket under which all keys, values, and nested buckets are stored. A transaction can either be read-only or read-write and managed or unmanaged.

Managed versus Unmanaged Transactions

A managed transaction is one where the caller provides a function to execute within the context of the transaction and the commit or rollback is handled automatically depending on whether or not the provided function returns an error. Attempting to manually call Rollback or Commit on the managed transaction will result in a panic.

An unmanaged transaction, on the other hand, requires the caller to manually call Commit or Rollback when they are finished with it. Leaving transactions open for long periods of time can have several adverse effects, so it is recommended that managed transactions are used instead.

Buckets

The Bucket interface provides the ability to manipulate key/value pairs and nested buckets as well as iterate through them.

The Get, Put, and Delete functions work with key/value pairs, while the Bucket, CreateBucket, CreateBucketIfNotExists, and DeleteBucket functions work with buckets. The ForEach function allows the caller to provide a function to be called with each key/value pair and nested bucket in the current bucket.

Root Bucket

As discussed above, all of the functions which are used to manipulate key/value pairs and nested buckets exist on the Bucket interface. The root bucket is the upper-most bucket in a namespace under which data is stored and is created at the same time as the namespace. Use the RootBucket function on the Tx interface to retrieve it.

Nested Buckets

The CreateBucket and CreateBucketIfNotExists functions on the Bucket interface provide the ability to create an arbitrary number of nested buckets. It is a good idea to avoid a lot of buckets with little data in them as it could lead to poor page utilization depending on the specific driver in use.

Example (BasicUsage)

This example demonstrates creating a new database, getting a namespace from it, and using a managed read-write transaction against the namespace to store and retrieve data.

// This example assumes the bdb (bolt db) driver is imported.
//
// import (
// 	"github.com/cybriq/p9/cmd/wallet/db"
// 	_ "github.com/cybriq/p9/cmd/wallet/db/bdb"
// )
//
// Create a database and schedule it to be closed and removed on exit. Typically you wouldn't want to remove the
// database right away like this, but it's done here in the example to ensure the example cleans up after itself.
dbPath := filepath.Join(os.TempDir(), "exampleusage.db")
db, e := walletdb.Create("bdb", dbPath)
if E.Chk(e) {
	return
}
defer func() {
	if e = os.Remove(dbPath); walletdb.E.Chk(e) {
	}
}()
defer func() {
	if e = db.Close(); walletdb.E.Chk(e) {
	}
}()
// Get or create a bucket in the database as needed. This bucket is what is typically passed to specific
// sub-packages so they have their own area to work in without worrying about conflicting keys.
bucketKey := []byte("walletsubpackage")
e = walletdb.Update(
	db, func(tx walletdb.ReadWriteTx) (e error) {
		bucket := tx.ReadWriteBucket(bucketKey)
		if bucket == nil {
			_, e = tx.CreateTopLevelBucket(bucketKey)
			if e != nil {
				return e
			}
		}
		return nil
	},
)
if E.Chk(e) {
	return
}
// Use the Update function of the namespace to perform a managed read-write transaction. The transaction will
// automatically be rolled back if the supplied inner function returns a non-nil error.
e = walletdb.Update(
	db, func(tx walletdb.ReadWriteTx) (e error) {
		// All data is stored against the root bucket of the namespace, or nested buckets of the root bucket. It's not
		// really necessary to store it in a separate variable like this, but it has been done here for the purposes of
		// the example to illustrate.
		rootBucket := tx.ReadWriteBucket(bucketKey)
		// Store a key/value pair directly in the root bucket.
		key := []byte("mykey")
		value := []byte("myvalue")
		if e = rootBucket.Put(key, value); E.Chk(e) {
			return e
		}
		// Read the key back and ensure it matches.
		if !bytes.Equal(rootBucket.Get(key), value) {
			return fmt.Errorf("unexpected value for key '%s'", key)
		}
		// Create a new nested bucket under the root bucket.
		nestedBucketKey := []byte("mybucket")
		nestedBucket, e := rootBucket.CreateBucket(nestedBucketKey)
		if e != nil {
			return e
		}
		// The key from above that was set in the root bucket does not exist in this new nested bucket.
		if nestedBucket.Get(key) != nil {
			return fmt.Errorf("key '%s' is not expected nil", key)
		}
		return nil
	},
)
if E.Chk(e) {
	return
}

Index

• Variables
• func BucketIsEmpty(bucket ReadBucket) bool
• func RegisterDriver(driver Driver) (e error)
• func SupportedDrivers() []string
• func Update(db DB, f func(tx ReadWriteTx) error) (e error)
• func View(db DB, f func(tx ReadTx) error) (e error)
• type DB
  • func Create(dbType string, args ...interface{}) (DB, error)
  • func Open(dbType string, args ...interface{}) (DB, error)
• type Driver
• type ReadBucket
• type ReadCursor
• type ReadTx
• type ReadWriteBucket
• type ReadWriteCursor
• type ReadWriteTx

Examples

• Package (BasicUsage)
• Create
• DB (CreateTopLevelBucket)

Variables

var (
	// ErrDbUnknownType is returned when there is no driver registered for the specified database type.
	ErrDbUnknownType = errors.New("unknown database type")
	// ErrDbDoesNotExist is returned when open is called for a database that does not exist.
	ErrDbDoesNotExist = errors.New("database does not exist")
	// ErrDbExists is returned when create is called for a database that already exists.
	ErrDbExists = errors.New("database already exists")
	// ErrDbNotOpen is returned when a database instance is accessed before it is opened or after it is closed.
	ErrDbNotOpen = errors.New("database not open")
	// ErrDbAlreadyOpen is returned when open is called on a database that is already open.
	ErrDbAlreadyOpen = errors.New("database already open")
	// ErrInvalid is returned if the specified database is not valid.
	ErrInvalid = errors.New("invalid database")
)

Errors that the various database functions may return.

var (
	// ErrTxClosed is returned when attempting to commit or rollback a transaction that has already had one of those
	// operations performed.
	ErrTxClosed = errors.New("tx closed")
	// ErrTxNotWritable is returned when an operation that requires write access to the database is attempted against a
	// read-only transaction.
	ErrTxNotWritable = errors.New("tx not writable")
)

Errors that can occur when beginning or committing a transaction.

var (
	// ErrBucketNotFound is returned when trying to access a bucket that has
	// not been created yet.
	ErrBucketNotFound = errors.New("bucket not found")
	// ErrBucketExists is returned when creating a bucket that already exists.
	ErrBucketExists = errors.New("bucket already exists")
	// ErrBucketNameRequired is returned when creating a bucket with a blank name.
	ErrBucketNameRequired = errors.New("bucket name required")
	// ErrKeyRequired is returned when inserting a zero-length key.
	ErrKeyRequired = errors.New("key required")
	// ErrKeyTooLarge is returned when inserting a key that is larger than MaxKeySize.
	ErrKeyTooLarge = errors.New("key too large")
	// ErrValueTooLarge is returned when inserting a value that is larger than MaxValueSize.
	ErrValueTooLarge = errors.New("value too large")
	// ErrIncompatibleValue is returned when trying create or delete a bucket on an existing non-bucket key or when
	// trying to create or delete a non-bucket key on an existing bucket key.
	ErrIncompatibleValue = errors.New("incompatible value")
)

Errors that can occur when putting or deleting a value or bucket.

var (
	// ErrDbTypeRegistered is returned when two different database drivers attempt to register with the name database
	// type.
	ErrDbTypeRegistered = errors.New("database type already registered")
)

Errors that can occur during driver registration.

var F, E, W, I, D, T log.LevelPrinter = log.GetLogPrinterSet(subsystem)

Functions

func BucketIsEmpty

func BucketIsEmpty(bucket ReadBucket) bool

BucketIsEmpty returns whether the bucket is empty, that is, whether there are no key/value pairs or nested buckets.

func RegisterDriver

func RegisterDriver(driver Driver) (e error)

RegisterDriver adds a backend database driver to available interfaces. ErrDbTypeRegistered will be retruned if the database type for the driver has already been registered.

func SupportedDrivers

func SupportedDrivers() []string

SupportedDrivers returns a slice of strings that represent the database drivers that have been registered and are therefore supported.

func Update

func Update(db DB, f func(tx ReadWriteTx) error) (e error)

Update opens a database read/write transaction and executes the function f with the transaction passed as a parameter.

After f exits, if f did not error, the transaction is committed.

Otherwise, if f did error, the transaction is rolled back.

If the rollback fails, the original error returned by f is still returned.

If the commit fails, the commit error is returned.

func View

func View(db DB, f func(tx ReadTx) error) (e error)

View opens a database read transaction and executes the function f with the transaction passed as a parameter. After f exits, the transaction is rolled back. If f errors, its error is returned, not a rollback error (if any occur).

Types

type DB

type DB interface {
	// BeginReadTx opens a database read transaction.
	BeginReadTx() (ReadTx, error)
	// BeginReadWriteTx opens a database read+write transaction.
	BeginReadWriteTx() (ReadWriteTx, error)
	// Copy writes a copy of the database to the provided writer. This call will start a read-only transaction to
	// perform all operations.
	Copy(w io.Writer) error
	// Close cleanly shuts down the database and syncs all data.
	Close() error
}

DB represents an ACID database. All database access is performed through read or read+write transactions.

Example (CreateTopLevelBucket)

This example demonstrates creating a new top level bucket.

// Load a database for the purposes of this example and schedule it to be closed and removed on exit. See the Create
// example for more details on what this step is doing.
db, teardownFunc, e := exampleLoadDB()
if E.Chk(e) {
	return
}
defer teardownFunc()
dbtx, e := db.BeginReadWriteTx()
if E.Chk(e) {
	return
}
defer func() {
	e = dbtx.Commit()
	if E.Chk(e) {
		return
	}
}()
// Get or create a bucket in the database as needed. This bucket is what is typically passed to specific
// sub-packages so they have their own area to work in without worrying about conflicting keys.
bucketKey := []byte("walletsubpackage")
bucket, e := dbtx.CreateTopLevelBucket(bucketKey)
if E.Chk(e) {
	return
}
// Prevent unused error.
_ = bucket

func Create

func Create(dbType string, args ...interface{}) (DB, error)

Create intializes and opens a database for the specified type. The arguments are specific to the database type driver. See the documentation for the database driver for further details.

ErrDbUnknownType will be returned if the the database type is not registered.

Example

This example demonstrates creating a new database.

// This example assumes the bdb (bolt db) driver is imported.
//
// import (
// 	"github.com/cybriq/p9/cmd/wallet/db"
// 	_ "github.com/cybriq/p9/cmd/wallet/db/bdb"
// )
//
// Create a database and schedule it to be closed and removed on exit. Typically you wouldn't want to remove the
// database right away like this, but it's done here in the example to ensure the example cleans up after itself.
dbPath := filepath.Join(os.TempDir(), "examplecreate.db")
db, e := walletdb.Create("bdb", dbPath)
if E.Chk(e) {
	return
}
defer func() {
	if e := os.Remove(dbPath); walletdb.E.Chk(e) {
	}
}()
defer func() {
	if e := db.Close(); walletdb.E.Chk(e) {
	}
}()

func Open

func Open(dbType string, args ...interface{}) (DB, error)

Open opens an existing database for the specified type. The arguments are specific to the database type driver. See the documentation for the database driver for further details.

ErrDbUnknownType will be returned if the the database type is not registered.

type Driver

type Driver struct {
	// DbType is the identifier used to uniquely identify a specific database driver. There can be only one driver with
	// the same name.
	DbType string
	// Create is the function that will be invoked with all user-specified arguments to create the database. This
	// function must return ErrDbExists if the database already exists.
	Create func(args ...interface{}) (DB, error)
	// Open is the function that will be invoked with all user-specified arguments to open the database. This function
	// must return ErrDbDoesNotExist if the database has not already been created.
	Open func(args ...interface{}) (DB, error)
}

Driver defines a structure for backend drivers to use when they registered themselves as a backend which implements the Db interface.

type ReadBucket

type ReadBucket interface {
	// NestedReadBucket retrieves a nested bucket with the given key. Returns nil if the bucket does not exist.
	NestedReadBucket(key []byte) ReadBucket
	// ForEach invokes the passed function with every key/value pair in the bucket. This includes nested buckets, in
	// which case the value is nil, but it does not include the key/value pairs within those nested buckets.
	//
	// NOTE: The values returned by this function are only valid during a transaction. Attempting to access them after a
	// transaction has ended results in undefined behavior. This constraint prevents additional data copies and allows
	// support for memory-mapped database implementations.
	ForEach(func(k, v []byte) error) error
	// Get returns the value for the given key. Returns nil if the key does not exist in this bucket (or nested
	// buckets).
	//
	// NOTE: The value returned by this function is only valid during a transaction. Attempting to access it after a
	// transaction has ended results in undefined behavior. This constraint prevents additional data copies and allows
	// support for memory-mapped database implementations.
	Get(key []byte) []byte
	ReadCursor() ReadCursor
}

ReadBucket represents a bucket (a hierarchical structure within the database) that is only allowed to perform read operations.

type ReadCursor

type ReadCursor interface {
	// First positions the cursor at the first key/value pair and returns the pair.
	First() (key, value []byte)
	// Last positions the cursor at the last key/value pair and returns the
	// pair.
	Last() (key, value []byte)
	// Next moves the cursor one key/value pair forward and returns the new
	// pair.
	Next() (key, value []byte)
	// Prev moves the cursor one key/value pair backward and returns the new
	// pair.
	Prev() (key, value []byte)
	// Seek positions the cursor at the passed seek key. If the key does not exist, the cursor is moved to the next key
	// after seek. Returns the new pair.
	Seek(seek []byte) (key, value []byte)
}

ReadCursor represents a bucket cursor that can be positioned at the start or end of the bucket's key/value pairs and iterate over pairs in the bucket. This type is only allowed to perform database read operations.

type ReadTx

type ReadTx interface {
	// ReadBucket opens the root bucket for read only access. If the bucket described by the key does not exist, nil is
	// returned.
	ReadBucket(key []byte) ReadBucket
	// Rollback closes the transaction, discarding changes (if any) if the database was modified by a write transaction.
	Rollback() error
}

ReadTx represents a database transaction that can only be used for reads. If a database update must occur, use a ReadWriteTx.

type ReadWriteBucket

type ReadWriteBucket interface {
	ReadBucket
	// NestedReadWriteBucket retrieves a nested bucket with the given key. Returns nil if the bucket does not exist.
	NestedReadWriteBucket(key []byte) ReadWriteBucket
	// CreateBucket creates and returns a new nested bucket with the given key. Returns ErrBucketExists if the bucket
	// already exists, ErrBucketNameRequired if the key is empty, or ErrIncompatibleValue if the key value is otherwise
	// invalid for the particular database implementation. Other errors are possible depending on the implementation.
	CreateBucket(key []byte) (ReadWriteBucket, error)
	// CreateBucketIfNotExists creates and returns a new nested bucket with the given key if it does not already exist.
	// Returns ErrBucketNameRequired if the key is empty or ErrIncompatibleValue if the key value is otherwise invalid
	// for the particular database backend. Other errors are possible depending on the implementation.
	CreateBucketIfNotExists(key []byte) (ReadWriteBucket, error)
	// DeleteNestedBucket removes a nested bucket with the given key. Returns ErrTxNotWritable if attempted against a
	// read-only transaction and ErrBucketNotFound if the specified bucket does not exist.
	DeleteNestedBucket(key []byte) error
	// Put saves the specified key/value pair to the bucket. Keys that do not already exist are added and keys that
	// already exist are overwritten. Returns ErrTxNotWritable if attempted against a read-only transaction.
	Put(key, value []byte) error
	// Delete removes the specified key from the bucket. Deleting a key that does not exist does not return an error.
	// Returns ErrTxNotWritable if attempted against a read-only transaction.
	Delete(key []byte) error

	// ReadWriteCursor returns a new cursor, allowing for iteration over the bucket's key/value pairs and nested buckets
	// in forward or backward order.
	ReadWriteCursor() ReadWriteCursor
}

ReadWriteBucket represents a bucket (a hierarchical structure within the database) that is allowed to perform both read and write operations.

type ReadWriteCursor

type ReadWriteCursor interface {
	ReadCursor
	// Delete removes the current key/value pair the cursor is at without invalidating the cursor. Returns
	// ErrIncompatibleValue if attempted when the cursor points to a nested bucket.
	Delete() error
}

ReadWriteCursor represents a bucket cursor that can be positioned at the start or end of the bucket's key/value pairs and iterate over pairs in the bucket. This abstraction is allowed to perform both database read and write operations.

type ReadWriteTx

type ReadWriteTx interface {
	ReadTx
	// ReadWriteBucket opens the root bucket for read/write access. If the bucket described by the key does not exist,
	// nil is returned.
	ReadWriteBucket(key []byte) ReadWriteBucket
	// CreateTopLevelBucket creates the top level bucket for a key if it does not exist. The newly-created bucket it
	// returned.
	CreateTopLevelBucket(key []byte) (ReadWriteBucket, error)
	// DeleteTopLevelBucket deletes the top level bucket for a key. This errors if the bucket can not be found or the
	// key keys a single value instead of a bucket.
	DeleteTopLevelBucket(key []byte) error
	// Commit commits all changes that have been on the transaction's root buckets and all of their sub-buckets to
	// persistent storage.
	Commit() error
}

ReadWriteTx represents a database transaction that can be used for both reads and writes. When only reads are necessary, consider using a ReadTx instead.