walletdb

README

walletdb

Package walletdb provides a namespaced database interface for dcrwallet.

A wallet essentially consists of a multitude of stored data such as private and public keys, key derivation bits, pay-to-script-hash scripts, and various metadata. One of the issues with many wallets is they are tightly integrated. Designing a wallet with loosely coupled components that provide specific functionality is ideal, however it presents a challenge in regards to data storage since each component needs to store its own data without knowing the internals of other components or breaking atomicity.

This package solves this issue by providing a namespaced database interface that is intended to be used by the main wallet daemon. This allows the potential for any backend database type with a suitable driver. Each component, which will typically be a package, can then implement various functionality such as address management, voting pools, and colored coin metadata in their own namespace without having to worry about conflicts with other packages even though they are sharing the same database that is managed by the wallet.

A suite of tests is provided to ensure proper functionality. See test_coverage.txt for the gocov coverage report. Alternatively, if you are running a POSIX OS, you can run the cov_report.sh script for a real-time report. Package walletdb is licensed under the copyfree ISC license.

This interfaces provided by this package were heavily inspired by the excellent boltdb project at https://github.com/boltdb/bolt by Ben B. Johnson.

Feature Overview

• Key/value store
• Namespace support
  • Allows multiple packages to have their own area in the database without worrying about conflicts
• Read-only and read-write transactions with both manual and managed modes
• Nested buckets
• Supports registration of backend databases
• Comprehensive test coverage

Documentation

[] (http://godoc.org/github.com/decred/dcrwallet/walletdb)

Full go doc style documentation for the project can be viewed online without installing this package by using the GoDoc site here: http://godoc.org/github.com/decred/dcrwallet/walletdb

You can also view the documentation locally once the package is installed with the godoc tool by running godoc -http=":6060" and pointing your browser to http://localhost:6060/pkg/github.com/decred/dcrwallet/walletdb

Installation

$ go get github.com/decred/dcrwallet/walletdb

Examples

• [Basic Usage Example] (http://godoc.org/github.com/decred/dcrwallet/walletdb#example-package--BasicUsage)
  Demonstrates creating a new database, getting a namespace from it, and using a managed read-write transaction against the namespace to store and retrieve data.

License

Package walletdb is licensed under the copyfree ISC License.

Documentation

Overview

Package walletdb provides a namespaced database interface for dcrwallet.

Overview

A wallet essentially consists of a multitude of stored data such as private and public keys, key derivation bits, pay-to-script-hash scripts, and various metadata. One of the issues with many wallets is they are tightly integrated. Designing a wallet with loosely coupled components that provide specific functionality is ideal, however it presents a challenge in regards to data storage since each component needs to store its own data without knowing the internals of other components or breaking atomicity.

This package solves this issue by providing a pluggable driver, namespaced database interface that is intended to be used by the main wallet daemon. This allows the potential for any backend database type with a suitable driver. Each component, which will typically be a package, can then implement various functionality such as address management, voting pools, and colored coin metadata in their own namespace without having to worry about conflicts with other packages even though they are sharing the same database that is managed by the wallet.

A quick overview of the features walletdb provides are as follows:

• Key/value store
• Namespace support
• Allows multiple packages to have their own area in the database without worrying about conflicts
• Read-only and read-write transactions with both manual and managed modes
• Nested buckets
• Supports registration of backend databases
• Comprehensive test coverage

Database

The main entry point is the DB interface. It exposes functionality for creating, retrieving, and removing namespaces. It is obtained via the Create and Open functions which take a database type string that identifies the specific database driver (backend) to use as well as arguments specific to the specified driver.

Namespaces

The Namespace interface is an abstraction that provides facilities for obtaining transactions (the Tx interface) that are the basis of all database reads and writes. Unlike some database interfaces that support reading and writing without transactions, this interface requires transactions even when only reading or writing a single key.

The Begin function provides an unmanaged transaction while the View and Update functions provide a managed transaction. These are described in more detail below.

Transactions

The Tx interface provides facilities for rolling back or commiting changes that took place while the transaction was active. It also provides the root bucket under which all keys, values, and nested buckets are stored. A transaction can either be read-only or read-write and managed or unmanaged.

Managed versus Unmanaged Transactions

A managed transaction is one where the caller provides a function to execute within the context of the transaction and the commit or rollback is handled automatically depending on whether or not the provided function returns an error. Attempting to manually call Rollback or Commit on the managed transaction will result in a panic.

An unmanaged transaction, on the other hand, requires the caller to manually call Commit or Rollback when they are finished with it. Leaving transactions open for long periods of time can have several adverse effects, so it is recommended that managed transactions are used instead.

Buckets

The Bucket interface provides the ability to manipulate key/value pairs and nested buckets as well as iterate through them.

The Get, Put, and Delete functions work with key/value pairs, while the Bucket, CreateBucket, CreateBucketIfNotExists, and DeleteBucket functions work with buckets. The ForEach function allows the caller to provide a function to be called with each key/value pair and nested bucket in the current bucket.

Root Bucket

As discussed above, all of the functions which are used to manipulate key/value pairs and nested buckets exist on the Bucket interface. The root bucket is the upper-most bucket in a namespace under which data is stored and is created at the same time as the namespace. Use the RootBucket function on the Tx interface to retrieve it.

Nested Buckets

The CreateBucket and CreateBucketIfNotExists functions on the Bucket interface provide the ability to create an arbitrary number of nested buckets. It is a good idea to avoid a lot of buckets with little data in them as it could lead to poor page utilization depending on the specific driver in use.

Example (BasicUsage)

This example demonstrates creating a new database, getting a namespace from it, and using a managed read-write transaction against the namespace to store and retrieve data.

package main

import (
	"bytes"
	"fmt"
	"os"
	"path/filepath"

	"github.com/decred/dcrwallet/walletdb"
	_ "github.com/decred/dcrwallet/walletdb/bdb"
)

func main() {
	// This example assumes the bdb (bolt db) driver is imported.
	//
	// import (
	// 	"github.com/btcsuite/btcwallet/walletdb"
	// 	_ "github.com/btcsuite/btcwallet/walletdb/bdb"
	// )

	// Create a database and schedule it to be closed and removed on exit.
	// Typically you wouldn't want to remove the database right away like
	// this, but it's done here in the example to ensure the example cleans
	// up after itself.
	dbPath := filepath.Join(os.TempDir(), "exampleusage.db")
	db, err := walletdb.Create("bdb", dbPath)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer os.Remove(dbPath)
	defer db.Close()

	// Get or create a namespace in the database as needed.  This namespace
	// is what is typically passed to specific sub-packages so they have
	// their own area to work in without worrying about conflicting keys.
	namespaceKey := []byte("walletsubpackage")
	namespace, err := db.Namespace(namespaceKey)
	if err != nil {
		fmt.Println(err)
		return
	}

	// Use the Update function of the namespace to perform a managed
	// read-write transaction.  The transaction will automatically be rolled
	// back if the supplied inner function returns a non-nil error.
	err = namespace.Update(func(tx walletdb.Tx) error {
		// All data is stored against the root bucket of the namespace,
		// or nested buckets of the root bucket.  It's not really
		// necessary to store it in a separate variable like this, but
		// it has been done here for the purposes of the example to
		// illustrate.
		rootBucket := tx.RootBucket()

		// Store a key/value pair directly in the root bucket.
		key := []byte("mykey")
		value := []byte("myvalue")
		if err := rootBucket.Put(key, value); err != nil {
			return err
		}

		// Read the key back and ensure it matches.
		if !bytes.Equal(rootBucket.Get(key), value) {
			return fmt.Errorf("unexpected value for key '%s'", key)
		}

		// Create a new nested bucket under the root bucket.
		nestedBucketKey := []byte("mybucket")
		nestedBucket, err := rootBucket.CreateBucket(nestedBucketKey)
		if err != nil {
			return err
		}

		// The key from above that was set in the root bucket does not
		// exist in this new nested bucket.
		if nestedBucket.Get(key) != nil {
			return fmt.Errorf("key '%s' is not expected nil", key)
		}

		return nil
	})
	if err != nil {
		fmt.Println(err)
		return
	}

}

Index

• Variables
• func NamespaceIsEmpty(namespace Namespace) (bool, error)
• func RegisterDriver(driver Driver) error
• func SupportedDrivers() []string
• type Bucket
• type Cursor
• type DB
  • func Create(dbType string, args ...interface{}) (DB, error)
  • func Open(dbType string, args ...interface{}) (DB, error)
• type Driver
• type Namespace
• type Tx

Examples

• Package (BasicUsage)
• Create
• DB (Namespace)

Variables

var (
	// ErrDbUnknownType is returned when there is no driver registered for
	// the specified database type.
	ErrDbUnknownType = errors.New("unknown database type")

	// ErrDbDoesNotExist is returned when open is called for a database that
	// does not exist.
	ErrDbDoesNotExist = errors.New("database does not exist")

	// ErrDbExists is returned when create is called for a database that
	// already exists.
	ErrDbExists = errors.New("database already exists")

	// ErrDbNotOpen is returned when a database instance is accessed before
	// it is opened or after it is closed.
	ErrDbNotOpen = errors.New("database not open")

	// ErrDbAlreadyOpen is returned when open is called on a database that
	// is already open.
	ErrDbAlreadyOpen = errors.New("database already open")

	// ErrInvalid is returned if the specified database is not valid.
	ErrInvalid = errors.New("invalid database")
)

Errors that the various database functions may return.

var (
	// ErrTxClosed is returned when attempting to commit or rollback a
	// transaction that has already had one of those operations performed.
	ErrTxClosed = errors.New("tx closed")

	// ErrTxNotWritable is returned when an operation that requires write
	// access to the database is attempted against a read-only transaction.
	ErrTxNotWritable = errors.New("tx not writable")
)

Errors that can occur when beginning or committing a transaction.

var (
	// ErrBucketNotFound is returned when trying to access a bucket that has
	// not been created yet.
	ErrBucketNotFound = errors.New("bucket not found")

	// ErrBucketExists is returned when creating a bucket that already exists.
	ErrBucketExists = errors.New("bucket already exists")

	// ErrBucketNameRequired is returned when creating a bucket with a blank name.
	ErrBucketNameRequired = errors.New("bucket name required")

	// ErrKeyRequired is returned when inserting a zero-length key.
	ErrKeyRequired = errors.New("key required")

	// ErrKeyTooLarge is returned when inserting a key that is larger than MaxKeySize.
	ErrKeyTooLarge = errors.New("key too large")

	// ErrValueTooLarge is returned when inserting a value that is larger than MaxValueSize.
	ErrValueTooLarge = errors.New("value too large")

	// ErrIncompatibleValue is returned when trying create or delete a
	// bucket on an existing non-bucket key or when trying to create or
	// delete a non-bucket key on an existing bucket key.
	ErrIncompatibleValue = errors.New("incompatible value")
)

Errors that can occur when putting or deleting a value or bucket.

var (
	// ErrDbTypeRegistered is returned when two different database drivers
	// attempt to register with the name database type.
	ErrDbTypeRegistered = errors.New("database type already registered")
)

Errors that can occur during driver registration.

Functions

func NamespaceIsEmpty

func NamespaceIsEmpty(namespace Namespace) (bool, error)

NamespaceIsEmpty returns whether the namespace is empty, that is, whether there are no key/value pairs or nested buckets.

func RegisterDriver

func RegisterDriver(driver Driver) error

RegisterDriver adds a backend database driver to available interfaces. ErrDbTypeRegistered will be retruned if the database type for the driver has already been registered.

func SupportedDrivers

func SupportedDrivers() []string

SupportedDrivers returns a slice of strings that represent the database drivers that have been registered and are therefore supported.

Types

type Bucket

type Bucket interface {
	// Bucket retrieves a nested bucket with the given key.  Returns nil if
	// the bucket does not exist.
	Bucket(key []byte) Bucket

	// CreateBucket creates and returns a new nested bucket with the given
	// key.  Returns ErrBucketExists if the bucket already exists,
	// ErrBucketNameRequired if the key is empty, or ErrIncompatibleValue
	// if the key value is otherwise invalid for the particular database
	// implementation.  Other errors are possible depending on the
	// implementation.
	CreateBucket(key []byte) (Bucket, error)

	// CreateBucketIfNotExists creates and returns a new nested bucket with
	// the given key if it does not already exist.  Returns
	// ErrBucketNameRequired if the key is empty or ErrIncompatibleValue
	// if the key value is otherwise invalid for the particular database
	// backend.  Other errors are possible depending on the implementation.
	CreateBucketIfNotExists(key []byte) (Bucket, error)

	// DeleteBucket removes a nested bucket with the given key.  Returns
	// ErrTxNotWritable if attempted against a read-only transaction and
	// ErrBucketNotFound if the specified bucket does not exist.
	DeleteBucket(key []byte) error

	// ForEach invokes the passed function with every key/value pair in
	// the bucket.  This includes nested buckets, in which case the value
	// is nil, but it does not include the key/value pairs within those
	// nested buckets.
	//
	// NOTE: The values returned by this function are only valid during a
	// transaction.  Attempting to access them after a transaction has ended
	// results in undefined behavior.  This constraint prevents additional
	// data copies and allows support for memory-mapped database
	// implementations.
	ForEach(func(k, v []byte) error) error

	// Writable returns whether or not the bucket is writable.
	Writable() bool

	// Put saves the specified key/value pair to the bucket.  Keys that do
	// not already exist are added and keys that already exist are
	// overwritten.  Returns ErrTxNotWritable if attempted against a
	// read-only transaction.
	Put(key, value []byte) error

	// Get returns the value for the given key.  Returns nil if the key does
	// not exist in this bucket (or nested buckets).
	//
	// NOTE: The value returned by this function is only valid during a
	// transaction.  Attempting to access it after a transaction has ended
	// results in undefined behavior.  This constraint prevents additional
	// data copies and allows support for memory-mapped database
	// implementations.
	Get(key []byte) []byte

	// Delete removes the specified key from the bucket.  Deleting a key
	// that does not exist does not return an error.  Returns
	// ErrTxNotWritable if attempted against a read-only transaction.
	Delete(key []byte) error

	// Cursor returns a new cursor, allowing for iteration over the bucket's
	// key/value pairs and nested buckets in forward or backward order.
	Cursor() Cursor
}

Bucket represents a collection of key/value pairs.

type Cursor

type Cursor interface {
	// Bucket returns the bucket the cursor was created for.
	Bucket() Bucket

	// Delete removes the current key/value pair the cursor is at without
	// invalidating the cursor.  Returns ErrTxNotWritable if attempted on a
	// read-only transaction, or ErrIncompatibleValue if attempted when the
	// cursor points to a nested bucket.
	Delete() error

	// First positions the cursor at the first key/value pair and returns
	// the pair.
	First() (key, value []byte)

	// Last positions the cursor at the last key/value pair and returns the
	// pair.
	Last() (key, value []byte)

	// Next moves the cursor one key/value pair forward and returns the new
	// pair.
	Next() (key, value []byte)

	// Prev moves the cursor one key/value pair backward and returns the new
	// pair.
	Prev() (key, value []byte)

	// Seek positions the cursor at the passed seek key.  If the key does
	// not exist, the cursor is moved to the next key after seek.  Returns
	// the new pair.
	Seek(seek []byte) (key, value []byte)
}

Cursor represents a cursor over key/value pairs and nested buckets of a bucket.

Note that open cursors are not tracked on bucket changes and any modifications to the bucket, with the exception of Cursor.Delete, invalidate the cursor. After invalidation, the cursor must be repositioned, or the keys and values returned may be unpredictable.

type DB

type DB interface {
	// Namespace returns a Namespace interface for the provided key.  See
	// the Namespace interface documentation for more details.  Attempting
	// to access a Namespace on a database that is not open yet or has been
	// closed will result in ErrDbNotOpen.  Namespaces are created in the
	// database on first access.
	Namespace(key []byte) (Namespace, error)

	// DeleteNamespace deletes the namespace for the passed key.
	// ErrBucketNotFound will be returned if the namespace does not exist.
	DeleteNamespace(key []byte) error

	// Copy writes a copy of the database to the provided writer.  This
	// call will start a read-only transaction to perform all operations.
	Copy(w io.Writer) error

	// Close cleanly shuts down the database and syncs all data.
	Close() error
}

DB represents a collection of namespaces which are persisted. All database access is performed through transactions which are obtained through the specific Namespace.

Example (Namespace)

This example demonstrates creating a new namespace.

package main

import (
	"fmt"
	"os"
	"path/filepath"

	"github.com/decred/dcrwallet/walletdb"
	_ "github.com/decred/dcrwallet/walletdb/bdb"
)

// exampleNum is used as a counter in the exampleLoadDB function to provided
// a unique database name for each example.
var exampleNum = 0

// exampleLoadDB is used in the examples to elide the setup code.
func exampleLoadDB() (walletdb.DB, func(), error) {
	dbName := fmt.Sprintf("exampleload%d.db", exampleNum)
	dbPath := filepath.Join(os.TempDir(), dbName)
	db, err := walletdb.Create("bdb", dbPath)
	if err != nil {
		return nil, nil, err
	}
	teardownFunc := func() {
		db.Close()
		os.Remove(dbPath)
	}
	exampleNum++

	return db, teardownFunc, err
}

func main() {
	// Load a database for the purposes of this example and schedule it to
	// be closed and removed on exit. See the Create example for more
	// details on what this step is doing.
	db, teardownFunc, err := exampleLoadDB()
	if err != nil {
		fmt.Println(err)
		return
	}
	defer teardownFunc()

	// Get or create a namespace in the database as needed.  This namespace
	// is what is typically passed to specific sub-packages so they have
	// their own area to work in without worrying about conflicting keys.
	namespaceKey := []byte("walletsubpackage")
	namespace, err := db.Namespace(namespaceKey)
	if err != nil {
		fmt.Println(err)
		return
	}

	// Prevent unused error.  Ordinarily the namespace would be used at this
	// point to start a managed or manual transaction.
	_ = namespace

}

func Create

func Create(dbType string, args ...interface{}) (DB, error)

Create intializes and opens a database for the specified type. The arguments are specific to the database type driver. See the documentation for the database driver for further details.

ErrDbUnknownType will be returned if the the database type is not registered.

Example

This example demonstrates creating a new database.

package main

import (
	"fmt"
	"os"
	"path/filepath"

	"github.com/decred/dcrwallet/walletdb"
	_ "github.com/decred/dcrwallet/walletdb/bdb"
)

func main() {
	// This example assumes the bdb (bolt db) driver is imported.
	//
	// import (
	// 	"github.com/btcsuite/btcwallet/walletdb"
	// 	_ "github.com/btcsuite/btcwallet/walletdb/bdb"
	// )

	// Create a database and schedule it to be closed and removed on exit.
	// Typically you wouldn't want to remove the database right away like
	// this, but it's done here in the example to ensure the example cleans
	// up after itself.
	dbPath := filepath.Join(os.TempDir(), "examplecreate.db")
	db, err := walletdb.Create("bdb", dbPath)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer os.Remove(dbPath)
	defer db.Close()

}

func Open

func Open(dbType string, args ...interface{}) (DB, error)

Open opens an existing database for the specified type. The arguments are specific to the database type driver. See the documentation for the database driver for further details.

ErrDbUnknownType will be returned if the the database type is not registered.

type Driver

type Driver struct {
	// DbType is the identifier used to uniquely identify a specific
	// database driver.  There can be only one driver with the same name.
	DbType string

	// Create is the function that will be invoked with all user-specified
	// arguments to create the database.  This function must return
	// ErrDbExists if the database already exists.
	Create func(args ...interface{}) (DB, error)

	// Open is the function that will be invoked with all user-specified
	// arguments to open the database.  This function must return
	// ErrDbDoesNotExist if the database has not already been created.
	Open func(args ...interface{}) (DB, error)
}

Driver defines a structure for backend drivers to use when they registered themselves as a backend which implements the Db interface.

type Namespace

type Namespace interface {
	// Begin starts a transaction which is either read-only or read-write
	// depending on the specified flag.  Multiple read-only transactions
	// can be started simultaneously while only a single read-write
	// transaction can be started at a time.  The call will block when
	// starting a read-write transaction when one is already open.
	//
	// NOTE: The transaction must be closed by calling Rollback or Commit on
	// it when it is no longer needed.  Failure to do so can result in
	// unclaimed memory depending on the specific database implementation.
	Begin(writable bool) (Tx, error)

	// View invokes the passed function in the context of a managed
	// read-only transaction.  Any errors returned from the user-supplied
	// function are returned from this function.
	//
	// Calling Rollback on the transaction passed to the user-supplied
	// function will result in a panic.
	View(fn func(Tx) error) error

	// Update invokes the passed function in the context of a managed
	// read-write transaction.  Any errors returned from the user-supplied
	// function will cause the transaction to be rolled back and are
	// returned from this function.  Otherwise, the transaction is commited
	// when the user-supplied function returns a nil error.
	//
	// Calling Rollback on the transaction passed to the user-supplied
	// function will result in a panic.
	Update(fn func(Tx) error) error
}

Namespace represents a database namespace that is inteded to support the concept of a single entity that controls the opening, creating, and closing of a database while providing other entities their own namespace to work in.

type Tx

type Tx interface {
	// RootBucket returns the top-most bucket for the namespace the
	// transaction was created from.
	RootBucket() Bucket

	// Commit commits all changes that have been made through the root
	// bucket and all of its sub-buckets to persistent storage.
	Commit() error

	// Rollback undoes all changes that have been made to the root bucket
	// and all of its sub-buckets.
	Rollback() error
}

Tx represents a database transaction. It can either by read-only or read-write. The transaction provides a root bucket against which all read and writes occur.

As would be expected with a transaction, no changes will be saved to the database until it has been committed. The transaction will only provide a view of the database at the time it was created. Transactions should not be long running operations.