memory

package
v0.0.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 12, 2022 License: MIT Imports: 16 Imported by: 0

README

Bhojpur Cache - In-Memory Storage Engine

It is a key/value database storage engine inspired by [Howard Chu's][hyc_symas] [LMDB project][lmdb]. The goal of the project is to provide a simple, fast, and reliable in-memory database storage engine for such projects that do not require full-fledged database server functionality, such as: PostgreSQL or MySQL.

Since the Bhojpur Cache in-memory database storage engine is meant to be used as a low-level piece of functionality, thence simplicity is the key. The Database APIs will be small and only focus on getting values and setting values.

Getting Started

Installing

To start using Bhojpur Cache in-memory database storage engine, install Go and run go get:

$ go get github.com/bhojpur/cache/pkg/memory...

It will retrieve the in-memory storage database engine library and install the Bhojpur Cache command line utility into your $GOBIN path.

Opening an In-Memory Database

The top-level object in a Bhojpur Cache in-memory database storage engine is a DB. It is represented as a single file on your data storage volume and represents a consistent snapshot of your in-memory data.

To open your in-memory database, simply use the memory.Open() function:

package main

import (
	"log"

	memory "github.com/bhojpur/cache/pkg/memory"
)

func main() {
	// Open the my.db data file in your current directory.
	// It will be created, if the file doesn't exist.
	db, err := memory.Open("my.db", 0600, nil)
	if err != nil {
		log.Fatal(err)
	}
	defer db.Close()

	...
}

Please note that Bhojpur Cache in-memory database storage engine obtains a file lock on the data file so multiple processes cannot open the same database at the same time. Opening an already open Bhojpur Cache in-memory database file will cause it to hang until the other processes closes it. To prevent an indefinite wait time, you can pass a timeout option to the Open() function:

db, err := memory.Open("my.db", 0600, &memory.Options{Timeout: 1 * time.Second})
In-Memory Transactions

The Bhojpur Cache in-memory database storage engine allows only one read-write transaction at a time, but allows as many read-only transactions as you want at a time. Each transaction has a consistent view of the data as it existed when the transaction started.

Individual transactions and all objects created from them (e.g. buckets, keys) are not thread safe. To work with data in multiple goroutines you must start a transaction for each one or use locking to ensure only one goroutine accesses a transaction at a time. Creating a transaction from the DB is thread safe.

The read-only transactions and read-write transactions should not depend on one another and generally shouldn't be opened simultaneously in the same goroutine. It can cause a deadlock as the read-write transaction needs to periodically re-map the data file, but it cannot do so while a read-only transaction is open.

Read-write Transactions

To start a read-write transaction, you can use the DB.Update() function:

err := db.Update(func(tx *memory.Tx) error {
	...
	return nil
})

Inside the closure, you have a consistent view of the database. You commit the transaction by returning nil at the end. You can also rollback the transaction at any point by returning an error. All database operations are allowed inside a read-write transaction.

Always check the return error as it will report any disk failures that can cause your transaction to remain incomplete. If you return an error within your closure it will be passed through.

Read-only Transactions

To start a read-only transaction, you can use the DB.View() function:

err := db.View(func(tx *memory.Tx) error {
	...
	return nil
})

You also get a consistent view of the database within this closure. However, no mutating operations are allowed within a read-only transaction. You can only retrieve the buckets, retrieve values, and copy the database within a read-only transaction.

Batch read-write Transactions

Each DB.Update() operation waits for the storage disk volumes to commit the writes. This overhead can be minimized by combining multiple updates with the DB.Batch() function:

err := db.Batch(func(tx *memory.Tx) error {
	...
	return nil
})

The concurrent Batch calls are opportunistically combined into larger transactions. A Batch is only useful when there are multiple goroutines calling it.

The trade-off is that Batch can call the given function multiple times, if parts of the transaction fail. The function must be idempotent and side effects must take effect only after a successful return from DB.Batch().

For example: do not display messages from inside the function, instead set variables in the enclosing scope:

var id uint64
err := db.Batch(func(tx *memory.Tx) error {
	// Find last key in bucket, decode as bigendian uint64, increment
	// by one, encode back to []byte, and add new key.
	...
	id = newValue
	return nil
})
if err != nil {
	return ...
}
fmt.Println("Allocated ID %d", id)
Managing transactions manually

The DB.View() and DB.Update() functions are wrappers around the DB.Begin() function. These helper functions will start the transaction, execute a function, then safely close your transaction if an error is returned. It is a recommended way to use Bhojpur Cache in-memory database transactions.

However, sometimes you may want to manually start and end your transactions. You can use the DB.Begin() function directly, but please be sure to close the transaction.

// Start a writable transaction.
tx, err := db.Begin(true)
if err != nil {
    return err
}
defer tx.Rollback()

// Use the transaction...
_, err := tx.CreateBucket([]byte("MyBucket"))
if err != nil {
    return err
}

// Commit the transaction and check for error.
if err := tx.Commit(); err != nil {
    return err
}

The first argument to DB.Begin() is a boolean stating, if the transaction should be writable.

Using Buckets

The Bucket are collections of key/value pairs within the database. All keys in a bucket must be unique. You can create a Bucket using the DB.CreateBucket() function:

db.Update(func(tx *memory.Tx) error {
	b, err := tx.CreateBucket([]byte("MyBucket"))
	if err != nil {
		return fmt.Errorf("create bucket: %s", err)
	}
	return nil
})

You can also create a Bucket only if it doesn't exist by using the Tx.CreateBucketIfNotExists() function. It's a common pattern to call this function for all your top-level buckets after you open your database so that you can guarantee they exist for future transactions.

To delete a Bucket, simply call the Tx.DeleteBucket() function.

Using key/value Pairs

To save a key/value pair to a Bucket, use the Bucket.Put() function:

db.Update(func(tx *memory.Tx) error {
	b := tx.Bucket([]byte("MyBucket"))
	err := b.Put([]byte("answer"), []byte("42"))
	return err
})

It will set the value of the "answer" key to "42" in the MyBucket bucket. To retrieve this value, we can use the Bucket.Get() function:

db.View(func(tx *memory.Tx) error {
	b := tx.Bucket([]byte("MyBucket"))
	v := b.Get([]byte("answer"))
	fmt.Printf("The answer is: %s\n", v)
	return nil
})

The Get() function does not return any error, because its operation is guaranteed to work (unless there is some kind of system failure). If the key exists, then it will return its byte slice value. If it doesn't exist, then it will return nil. It is important to note that you can have a zero-length value set to a key which is different than the key not existing.

Use the Bucket.Delete() function to delete a key from the Bucket.

Please note that values returned from the Get() are only valid, while the transaction is open. If you need to use a value outside of the transaction then you must use copy() to copy it to another byte slice.

Auto-incrementing integer for the bucket

By using the NextSequence() function, you can let Bhojpur Cache in-memory database storage engine determine a sequence, which can be used as the unique identifier for your key/value pairs. See the example below.

// CreateUser saves u to the In-Memory database. The new user ID is set on u once the data is persisted.
func (s *Store) CreateUser(u *User) error {
    return s.db.Update(func(tx *memory.Tx) error {
        // Retrieve the users Bucket.
        // This should be created when the In-Memory database is first opened.
        b := tx.Bucket([]byte("users"))

        // Generate ID for the user.
        // This returns an error only if the Tx is closed or not writeable.
        // That can't happen in an Update() call so I ignore the error check.
        id, _ := b.NextSequence()
        u.ID = int(id)

        // Marshal user data into bytes.
        buf, err := json.Marshal(u)
        if err != nil {
            return err
        }

        // Persist bytes to users Bucket.
        return b.Put(itob(u.ID), buf)
    })
}

// itob returns an 8-byte big endian representation of v.
func itob(v int) []byte {
    b := make([]byte, 8)
    binary.BigEndian.PutUint64(b, uint64(v))
    return b
}

type User struct {
    ID int
    ...
}
Iterating over Keys

The Bhojpur Cache in-memory database storage engine stores its keys in byte-sorted order within a Bucket. It makes sequential iteration over these keys extremely fast. To iterate over the keys, we'll use a Cursor:

db.View(func(tx *memory.Tx) error {
	// Assuming that Bucket exists and has keys
	b := tx.Bucket([]byte("MyBucket"))

	c := b.Cursor()

	for k, v := c.First(); k != nil; k, v = c.Next() {
		fmt.Printf("key=%s, value=%s\n", k, v)
	}

	return nil
})

The Cursor allows you to move to a specific point in the list of keys and move forward or backward through the keys one at a time.

The following functions are available on the Cursor object:

First()  Move to the first key.
Last()   Move to the last key.
Seek()   Move to a specific key.
Next()   Move to the next key.
Prev()   Move to the previous key.

Each of those functions has a return signature of (key []byte, value []byte). When you have iterated to the end of the Cursor, then Next() will return a nil key. You must seek to a position using First(), Last(), or Seek() before calling Next() or Prev(). If you do not seek to a position, then these functions will return a nil key.

During iteration, if the key is non-nil but the value is nil, that means the key refers to a Bucket rather than a value. Use Bucket.Bucket() to access the sub-bucket.

Prefix Scans

To iterate over a key prefix, you can combine Seek() and bytes.HasPrefix():

db.View(func(tx *memory.Tx) error {
	// Assuming that Bucket exists and has keys
	c := tx.Bucket([]byte("MyBucket")).Cursor()

	prefix := []byte("1234")
	for k, v := c.Seek(prefix); k != nil && bytes.HasPrefix(k, prefix); k, v = c.Next() {
		fmt.Printf("key=%s, value=%s\n", k, v)
	}

	return nil
})
Range Scans

Another common use case is scanning over a range such as, a time range. If you use a sortable time encoding, such as RFC3339, then you can query a specific date range like this:

db.View(func(tx *memory.Tx) error {
	// Assume our events bucket exists and has RFC3339 encoded time keys.
	c := tx.Bucket([]byte("Events")).Cursor()

	// Our time range spans the 2010's decade.
	min := []byte("2010-01-01T00:00:00Z")
	max := []byte("2020-01-01T00:00:00Z")

	// Iterate over the 2010's.
	for k, v := c.Seek(min); k != nil && bytes.Compare(k, max) <= 0; k, v = c.Next() {
		fmt.Printf("%s: %s\n", k, v)
	}

	return nil
})

Note that, while RFC3339 is sortable, the Go implementation of RFC3339Nano does not use a fixed number of digits after the decimal point and is therefore not sortable.

ForEach()

You can also use the function ForEach(), if you know you'll be iterating over all the keys in a Bucket:

db.View(func(tx *memory.Tx) error {
	// Assume that Bucket exists and has keys
	b := tx.Bucket([]byte("MyBucket"))

	b.ForEach(func(k, v []byte) error {
		fmt.Printf("key=%s, value=%s\n", k, v)
		return nil
	})
	return nil
})

Please note that keys and values in a ForEach() call are only valid, while the transaction is open. If you need to use a key or value outside of the transaction, you must use copy() to copy it to another byte slice.

Nested Buckets

You can also store a Bucket in a key to create nested buckets. The API is the same as the Bucket Management API on the DB object:

func (*Bucket) CreateBucket(key []byte) (*Bucket, error)
func (*Bucket) CreateBucketIfNotExists(key []byte) (*Bucket, error)
func (*Bucket) DeleteBucket(key []byte) error

For example, you had a multi-tenant software application, where the root-level bucket was the Account bucket. Inside of this bucket, there was a sequence of accounts, which themselves are buckets. And, inside the sequence bucket, you could have many more buckets pertaining to the Account itself (e.g., Users, Notes, etc) isolating the information into logical groupings.


// createUser creates a new user in the given account.
func createUser(accountID int, u *User) error {
    // Start the In-Memory database transaction.
    tx, err := db.Begin(true)
    if err != nil {
        return err
    }
    defer tx.Rollback()

    // Retrieve the root Bucket for the account.
    // Assume this has already been created when the account was set up.
    root := tx.Bucket([]byte(strconv.FormatUint(accountID, 10)))

    // Setup the users Bucket.
    bkt, err := root.CreateBucketIfNotExists([]byte("USERS"))
    if err != nil {
        return err
    }

    // Generate an ID for the new User.
    userID, err := bkt.NextSequence()
    if err != nil {
        return err
    }
    u.ID = userID

    // Marshal and save the encoded User.
    if buf, err := json.Marshal(u); err != nil {
        return err
    } else if err := bkt.Put([]byte(strconv.FormatUint(u.ID, 10)), buf); err != nil {
        return err
    }

    // Commit the In-Memory database transaction.
    if err := tx.Commit(); err != nil {
        return err
    }

    return nil
}
In-Memory Database Backups

The Bhojpur Cache in-memory database storage engine stores a single file so it's easy to backup. You can use Tx.WriteTo() function to write a consistent view of the in-memory database to a writer. If you call this from a read-only transaction, it will perform a hot backup and not block your other database reads and writes.

By default, it will use a regular file handle which will utilize the operating system's page cache.

A common use case is to Backup-over-HTTP so that you could use tools like cURL to do the in-memory database backups:

func BackupHandleFunc(w http.ResponseWriter, req *http.Request) {
	err := db.View(func(tx *memory.Tx) error {
		w.Header().Set("Content-Type", "application/octet-stream")
		w.Header().Set("Content-Disposition", `attachment; filename="my.db"`)
		w.Header().Set("Content-Length", strconv.Itoa(int(tx.Size())))
		_, err := tx.WriteTo(w)
		return err
	})
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
	}
}

Then, you can backup the data using this command:

$ curl http://localhost/backup > my.db

Or, you can open a web-browser and point to http://localhost/backup. It will download the in-memory data snapshot automatically.

If you want to backup to another file you can use the Tx.CopyFile() helper function.

Statistics

The Bhojpur Cache in-memory database keeps a running count of many of the internal operations as it performs so that you can better understand what's going on. By grabbing an in-memory data snapshot of these stats at two points in time, we can analyze what operations were performed during that time range.

For example, we could start a goroutine to log the stats every 10 seconds:

go func() {
	// Grab the initial stats.
	prev := memdb.Stats()

	for {
		// Wait for 10s.
		time.Sleep(10 * time.Second)

		// Grab the current stats and diff them.
		stats := memdb.Stats()
		diff := stats.Sub(&prev)

		// Encode stats to JSON and print to STDERR.
		json.NewEncoder(os.Stderr).Encode(diff)

		// Save stats for the next loop.
		prev = stats
	}
}()

It's also useful to pipe these stats to a service, such as: statsd, for monitoring or to provide an HTTP endpoint that would perform a fixed-length sample.

Read-only Mode

Sometimes it is useful to create a shared, read-only Bhojpur Cache in-memory database. To do this, set the Options.ReadOnly flag when opening your in-memory database. The read-only mode uses a shared lock to allow multiple processes to read from the in-memory database, but it will block any processes from opening the database file in read-write mode.

db, err := memory.Open("my.db", 0666, &memory.Options{ReadOnly: true})
if err != nil {
	log.Fatal(err)
}
Mobile Platform usage (e.g., Android / iOS)

The Bhojpur Cache in-memory database storage engine is able to run on mobile devices by leveraging the binding feature of the GoMobile tool. Create a struct that will contain your Bhojpur Cache in-memory database storage logic and a reference to a *memory.DB by initializing constructor that takes in a file path where the database file will be stored. Neither the Android nor iOS require extra permissions or cleanup from using this method.

func NewCacheDB(filepath string) *CacheDB {
	db, err := memory.Open(filepath+"/demo.db", 0600, nil)
	if err != nil {
		log.Fatal(err)
	}

	return &CacheDB{db}
}

type CacheDB struct {
	db *memory.DB
	...
}

func (b *CacheDB) Path() string {
	return b.db.Path()
}

func (b *CacheDB) Close() {
	b.db.Close()
}

The database logic should be defined as methods on this wrapper struct.

To initialize this struct from the native language (both the mobile platforms now sync their local storage to the Cloud. These snippets disable that functionality for the database file):

Android Platform
String path;
if (android.os.Build.VERSION.SDK_INT >=android.os.Build.VERSION_CODES.LOLLIPOP){
    path = getNoBackupFilesDir().getAbsolutePath();
} else{
    path = getFilesDir().getAbsolutePath();
}
Cachemobiledemo.CacheDB cacheDB = Cachemobiledemo.NewCacheDB(path)
iOS Platform
- (void)demo {
    NSString* path = [NSSearchPathForDirectoriesInDomains(NSLibraryDirectory,
                                                          NSUserDomainMask,
                                                          YES) objectAtIndex:0];
	GoCachemobiledemoCacheDB * demo = GoCachemobiledemoNewCacheDB(path);
	[self addSkipBackupAttributeToItemAtPath:demo.path];
	//Some DB Logic would go here
	[demo close];
}

- (BOOL)addSkipBackupAttributeToItemAtPath:(NSString *) filePathString
{
    NSURL* URL= [NSURL fileURLWithPath: filePathString];
    assert([[NSFileManager defaultManager] fileExistsAtPath: [URL path]]);

    NSError *error = nil;
    BOOL success = [URL setResourceValue: [NSNumber numberWithBool: YES]
                                  forKey: NSURLIsExcludedFromBackupKey error: &error];
    if(!success){
        NSLog(@"Error excluding %@ from backup %@", [URL lastPathComponent], error);
    }
    return success;
}

Comparing with other Database Systems

PostgreSQL, MySQL, & other relational databases

Relational databases structure data into rows and are only accessible through the use of SQL. This approach provides flexibility in how you store and query your data, but also incurs overhead in parsing and planning SQL statements. The Bhojpur Cache in-memory database engine accesses all data by a byte slice key. This Bhojpur Cache memory database fast to read and write data by key, but provides no built-in support for joining values together.

Most relational databases (with the exception of SQLite) are standalone servers that run separately from your application. This gives your systems flexibility to connect multiple application servers to a single database server, but also adds overhead in serializing and transporting data over the network. The Bhojpur Cache in-memory database engine runs as a library included in your application, so all data access has to go through your application's process. This brings data closer to your application, but limits multi-process access to the data.

LevelDB, RocksDB

The LevelDB and its derivatives (e.g., RocksDB, HyperLevelDB) are similar to the Bhojpur Cache in-memory database storage engine in that they are libraries bundled into the application, however, their underlying structure is a log-structured merge-tree (LSM tree). An LSM tree optimizes random writes by using a write ahead log and multi-tiered, sorted files, called SSTables. The Bhojpur Cache in-memory database storage engine uses a B+tree internally and only a single file. Both the approaches have some trade-offs.

If you require a high random write throughput (>10,000 w/sec) or you need to use spinning disk drives, then LevelDB could be a good choice. If your application is read-heavy or does a lot of range scans then Bhojpur Cache in-memory database storage engine could be a good choice.

Another important consideration is that LevelDB does not have transactions. It supports batch writing of key/values pairs and it supports read snapshots but it will not give you the ability to do a compare-and-swap operation safely. The Bhojpur Cache in-memory database storage engine supports fully serializable ACID transactions.

LMDB

The Bhojpur Cache in-memory databse storage engine was originally a port of LMDB so it is architecturally similar. Both use a B+tree, have ACID semantics with fully serializable transactions, and support lock-free MVCC using a single writer and multiple readers.

The two projects have somewhat diverged. LMDB heavily focuses on raw performance while Bhojpur Cache in-memory database storage engine has focused on simplicity and ease of use. For example, LMDB allows several unsafe actions, such as: direct writes for the sake of performance. The Bhojpur Cache in-memory database storage engine opts to disallow actions which can leave the database in a corrupted state. The only exception to this in Bhojpur Cache in-memory database storage engine is DB.NoSync.

There are also a few differences in API. LMDB requires a maximum mmap size when opening an mdb_env whereas Bhojpur Cache in-memory database storage engine will handle incremental mmap resizing automatically. LMDB overloads the getter and setter functions with multiple flags, whereas Bhojpur Cache in-memory database splits these specialized cases into their own functions.

Caveats & Limitations

It's important to pick the right tool for the job and Bhojpur Cache in-memory database storage engine is no exception. Here are a few things to note when evaluating and using it:

  • Bhojpur Cache in-memory database storage engine is good for read intensive workloads. Sequential write performance is also fast but random writes can be slow. You can use DB.Batch() or add a write-ahead log to help mitigate this issue.

  • Bhojpur Cache in-memory database storage engine uses a B+tree internally, so there can be a lot of random page access. The solid-state drive (SSD) provide a significant performance boost over spinning disk drives.

  • Try to avoid long running read transactions. Bhojpur Cache in-memory database storage engine uses copy-on-write so old pages cannot be reclaimed while an old transaction is using them.

  • Byte slices returned from Bhojpur Cache in-memory database storage engine are only valid during a transaction. Once the transaction has been committed or rolled back then the memory they point to can be reused by a new page or can be unmapped from virtual memory and you'll see an unexpected fault address panic when accessing it.

  • Bhojpur Cache in-memory database storage engine uses an exclusive write lock on the database file so it cannot be shared by multiple processes.

  • Be careful while using Bucket.FillPercent. Setting a high fill percent for the Buckets that have random inserts will cause your database to have very poor page utilization.

  • In general, use larger buckets. Smaller buckets causes poor memory page utilization once they become larger than the page size (typically 4KB).

  • Bulk loading a lot of random writes into a new Bucket could be slow as the page will not split until the transaction is committed. Randomly inserting more than 100,000 key/value pairs into a single new Bucket in a single transaction is not advised.

  • Bhojpur Cache in-memory database storage engine uses a memory-mapped file, so the underlying operating system handles the caching of the data. Typically, the OS will cache as much of the file as it can in the memory and will release the memory as needed to other processes. This means that Bhojpur Cache storage engine can show very high memory usage when working with large databases. However, this is expected and the OS will release memory as needed. Bhojpur Cache in-memory storage engine can handle databases much larger than the available physical RAM, provided its memory-map fits in the process virtual address space. It may be problematic on 32-bits systems.

  • The data structures in the Bhojpur Cache in-memory database are memory mapped so the data file will be endian specific. This means that you cannot copy a Bhojpur Cache database file from a little endian machine to a big endian machine and have it work. For most users this is not a concern since most modern CPUs are little endian.

  • Because of the way pages are laid out on disk, the Bhojpur Cache in-memory database storage engine cannot truncate data files and return free pages back to the disk. Instead, Bhojpur Cache in-memory database storage engine maintains a free list of unused pages within its data file. These free pages can be reused by later transactions. This works well for many use cases as the databases generally tend to grow. However, it's important to note that deleting large chunks of data will not allow you to reclaim that space on disk.

Reading the Source Code

The Bhojpur Cache in-memory database storage engine is a relatively small code base (<3KLOC) for an embedded, serializable, transactional key/value database so it can be a good starting point for people interested in how databases work.

The best places to start are the main entry points into Bhojpur Cache in-memory database storage engine:

  • Open() - Initializes the reference to the database. It's responsible for creating the database if it doesn't exist, obtaining an exclusive lock on the file, reading the meta pages, and memory-mapping the file.

  • DB.Begin() - Starts a read-only or read-write transaction depending on the value of the writable argument. This requires briefly obtaining the meta lock to keep track of open transactions. Only one read-write transaction can exist at a time so the rwlock is acquired during the life of a read-write transaction.

  • Bucket.Put() - Writes a key/value pair into a Bucket. After validating the arguments, a cursor is used to traverse the B+tree to the page and position where they key & value will be written. Once the position is found, the bucket materializes the underlying page and the page's parent pages into memory as "nodes". These nodes are where mutations occur during read-write transactions. These changes get flushed to disk during commit.

  • Bucket.Get() - Retrieves a key/value pair from a Bucket. This uses a cursor to move to the page & position of a key/value pair. During a read-only transaction, the key and value data is returned as a direct reference to the underlying mmap file so there's no allocation overhead. For the read-write transactions, this data may reference the mmap file or one of the in-memory node values.

  • Cursor - This object is simply for traversing the B+tree of on-disk pages or in-memory nodes. It can seek to a specific key, move to the first or last value, or it can move forward or backward. The cursor handles the movement up and down the B+tree transparently to the end user.

  • Tx.Commit() - Converts the in-memory dirty nodes and the list of free pages into pages to be written to disk. Writing to disk then occurs in two phases. First, the dirty pages are written to disk and an fsync() occurs. Secondly, a new meta page with an incremented transaction ID is written and another fsync() occurs. This two phase write ensures that partially written data pages are ignored in the event of a crash since the meta page pointing to them is never written. Partially written meta pages are invalidated, because they are written with a checksum.

Documentation

Index

Examples

Constants

View Source 
const (
	// MaxKeySize is the maximum length of a key, in bytes.
	MaxKeySize = 32768

	// MaxValueSize is the maximum length of a value, in bytes.
	MaxValueSize = (1 << 31) - 2
)
View Source 
const (
	DefaultMaxBatchSize  int = 1000
	DefaultMaxBatchDelay     = 10 * time.Millisecond
	DefaultAllocSize         = 16 * 1024 * 1024
)

Default values if not set in a DB instance.

View Source 
const (
	// FreelistArrayType indicates backend freelist type is array
	FreelistArrayType = FreelistType("array")
	// FreelistMapType indicates backend freelist type is hashmap
	FreelistMapType = FreelistType("hashmap")
)
View Source 
const DefaultFillPercent = 0.5

DefaultFillPercent is the percentage that split pages are filled. This value can be changed by setting Bucket.FillPercent.

View Source 
const IgnoreNoSync = runtime.GOOS == "openbsd"

IgnoreNoSync specifies whether the NoSync field of a DB is ignored when syncing changes to a file. This is required as some operating systems, such as OpenBSD, do not have a unified buffer cache (UBC) and writes must be synchronized using the msync(2) syscall.

Variables

View Source 
var (
	// ErrDatabaseNotOpen is returned when a DB instance is accessed before it
	// is opened or after it is closed.
	ErrDatabaseNotOpen = errors.New("in-memory database not open")

	// ErrDatabaseOpen is returned when opening a database that is
	// already open.
	ErrDatabaseOpen = errors.New("in-memory database already open")

	// ErrInvalid is returned when both meta pages on a database are invalid.
	// This typically occurs when a file is not a Bhojpur Cache in-memory database.
	ErrInvalid = errors.New("invalid in-memory database")

	// ErrVersionMismatch is returned when the data file was created with a
	// different version of Bhojpur Cache.
	ErrVersionMismatch = errors.New("in-memory database storage engine version mismatch")

	// ErrChecksum is returned when either meta page checksum does not match.
	ErrChecksum = errors.New("checksum error")

	// ErrTimeout is returned when a database cannot obtain an exclusive lock
	// on the data file after the timeout passed to Open().
	ErrTimeout = errors.New("timeout")
)

These errors can be returned when opening or calling methods on a DB.

View Source 
var (
	// ErrTxNotWritable is returned when performing a write operation on a
	// read-only transaction.
	ErrTxNotWritable = errors.New("tx not writable")

	// ErrTxClosed is returned when committing or rolling back a transaction
	// that has already been committed or rolled back.
	ErrTxClosed = errors.New("tx closed")

	// ErrDatabaseReadOnly is returned when a mutating transaction is started on a
	// read-only database.
	ErrDatabaseReadOnly = errors.New("in-memory database is in read-only mode")
)

These errors can occur when beginning or committing a Tx.

View Source 
var (
	// ErrBucketNotFound is returned when trying to access a bucket that has
	// not been created yet.
	ErrBucketNotFound = errors.New("in-memory bucket not found")

	// ErrBucketExists is returned when creating a bucket that already exists.
	ErrBucketExists = errors.New("in-memory bucket already exists")

	// ErrBucketNameRequired is returned when creating a bucket with a blank name.
	ErrBucketNameRequired = errors.New("in-memory bucket name required")

	// ErrKeyRequired is returned when inserting a zero-length key.
	ErrKeyRequired = errors.New("key required")

	// ErrKeyTooLarge is returned when inserting a key that is larger than MaxKeySize.
	ErrKeyTooLarge = errors.New("key too large")

	// ErrValueTooLarge is returned when inserting a value that is larger than MaxValueSize.
	ErrValueTooLarge = errors.New("value too large")

	// ErrIncompatibleValue is returned when trying create or delete a bucket
	// on an existing non-bucket key or when trying to create or delete a
	// non-bucket key on an existing bucket key.
	ErrIncompatibleValue = errors.New("incompatible value")
)

These errors can occur when putting or deleting a value or a bucket.

View Source 
var DefaultOptions = &Options{
	Timeout:      0,
	NoGrowSync:   false,
	FreelistType: FreelistArrayType,
}

DefaultOptions represent the options used if nil options are passed into Open(). No timeout is used which will cause Bhojpur Cache to wait indefinitely for a lock.

Functions

func Compact 

func Compact(dst, src *DB, txMaxSize int64) error

Compact will create a copy of the source DB and in the destination DB. This may reclaim space that the source database no longer has use for. txMaxSize can be used to limit the transactions size of this process and may trigger intermittent commits. A value of zero will ignore transaction sizes.

Types

type Bucket 

type Bucket struct {

	// Sets the threshold for filling nodes when they split. By default,
	// the bucket will fill to 50% but it can be useful to increase this
	// amount if you know that your write workloads are mostly append-only.
	//
	// This is non-persisted across transactions so it must be set in every Tx.
	FillPercent float64
	// contains filtered or unexported fields
}

Bucket represents a collection of key/value pairs inside the database.

func (*Bucket) Bucket 

func (b *Bucket) Bucket(name []byte) *Bucket

Bucket retrieves a nested bucket by name. Returns nil if the bucket does not exist. The bucket instance is only valid for the lifetime of the transaction.

func (*Bucket) CreateBucket 

func (b *Bucket) CreateBucket(key []byte) (*Bucket, error)

CreateBucket creates a new bucket at the given key and returns the new bucket. Returns an error if the key already exists, if the bucket name is blank, or if the bucket name is too long. The bucket instance is only valid for the lifetime of the transaction.

func (*Bucket) CreateBucketIfNotExists 

func (b *Bucket) CreateBucketIfNotExists(key []byte) (*Bucket, error)

CreateBucketIfNotExists creates a new bucket if it doesn't already exist and returns a reference to it. Returns an error if the bucket name is blank, or if the bucket name is too long. The bucket instance is only valid for the lifetime of the transaction.

func (*Bucket) Cursor 

func (b *Bucket) Cursor() *Cursor

Cursor creates a cursor associated with the bucket. The cursor is only valid as long as the transaction is open. Do not use a cursor after the transaction is closed.

func (*Bucket) Delete 

func (b *Bucket) Delete(key []byte) error

Delete removes a key from the bucket. If the key does not exist then nothing is done and a nil error is returned. Returns an error if the bucket was created from a read-only transaction.

Example 
// Open the database.
db, err := memcache.Open(tempfile(), 0666, nil)
if err != nil {
	log.Fatal(err)
}
defer os.Remove(db.Path())

// Start a write transaction.
if err := db.Update(func(tx *memcache.Tx) error {
	// Create a bucket.
	b, err := tx.CreateBucket([]byte("widgets"))
	if err != nil {
		return err
	}

	// Set the value "bar" for the key "foo".
	if err := b.Put([]byte("foo"), []byte("bar")); err != nil {
		return err
	}

	// Retrieve the key back from the database and verify it.
	value := b.Get([]byte("foo"))
	fmt.Printf("The value of 'foo' was: %s\n", value)

	return nil
}); err != nil {
	log.Fatal(err)
}

// Delete the key in a different write transaction.
if err := db.Update(func(tx *memcache.Tx) error {
	return tx.Bucket([]byte("widgets")).Delete([]byte("foo"))
}); err != nil {
	log.Fatal(err)
}

// Retrieve the key again.
if err := db.View(func(tx *memcache.Tx) error {
	value := tx.Bucket([]byte("widgets")).Get([]byte("foo"))
	if value == nil {
		fmt.Printf("The value of 'foo' is now: nil\n")
	}
	return nil
}); err != nil {
	log.Fatal(err)
}

// Close database to release file lock.
if err := db.Close(); err != nil {
	log.Fatal(err)
}

Output:

The value of 'foo' was: bar
The value of 'foo' is now: nil

func (*Bucket) DeleteBucket 

func (b *Bucket) DeleteBucket(key []byte) error

DeleteBucket deletes a bucket at the given key. Returns an error if the bucket does not exists, or if the key represents a non-bucket value.

func (*Bucket) ForEach 

func (b *Bucket) ForEach(fn func(k, v []byte) error) error

ForEach executes a function for each key/value pair in a bucket. If the provided function returns an error then the iteration is stopped and the error is returned to the caller. The provided function must not modify the bucket; this will result in undefined behavior.

Example 
// Open the database.
db, err := memcache.Open(tempfile(), 0666, nil)
if err != nil {
	log.Fatal(err)
}
defer os.Remove(db.Path())

// Insert data into a bucket.
if err := db.Update(func(tx *memcache.Tx) error {
	b, err := tx.CreateBucket([]byte("animals"))
	if err != nil {
		return err
	}

	if err := b.Put([]byte("dog"), []byte("fun")); err != nil {
		return err
	}
	if err := b.Put([]byte("cat"), []byte("lame")); err != nil {
		return err
	}
	if err := b.Put([]byte("liger"), []byte("awesome")); err != nil {
		return err
	}

	// Iterate over items in sorted key order.
	if err := b.ForEach(func(k, v []byte) error {
		fmt.Printf("A %s is %s.\n", k, v)
		return nil
	}); err != nil {
		return err
	}

	return nil
}); err != nil {
	log.Fatal(err)
}

// Close database to release file lock.
if err := db.Close(); err != nil {
	log.Fatal(err)
}

Output:

A cat is lame.
A dog is fun.
A liger is awesome.

func (*Bucket) Get 

func (b *Bucket) Get(key []byte) []byte

Get retrieves the value for a key in the bucket. Returns a nil value if the key does not exist or if the key is a nested bucket. The returned value is only valid for the life of the transaction.

func (*Bucket) NextSequence 

func (b *Bucket) NextSequence() (uint64, error)

NextSequence returns an autoincrementing integer for the bucket.

func (*Bucket) Put 

func (b *Bucket) Put(key []byte, value []byte) error

Put sets the value for a key in the bucket. If the key exist then its previous value will be overwritten. Supplied value must remain valid for the life of the transaction. Returns an error if the bucket was created from a read-only transaction, if the key is blank, if the key is too large, or if the value is too large.

Example 
// Open the database.
db, err := memcache.Open(tempfile(), 0666, nil)
if err != nil {
	log.Fatal(err)
}
defer os.Remove(db.Path())

// Start a write transaction.
if err := db.Update(func(tx *memcache.Tx) error {
	// Create a bucket.
	b, err := tx.CreateBucket([]byte("widgets"))
	if err != nil {
		return err
	}

	// Set the value "bar" for the key "foo".
	if err := b.Put([]byte("foo"), []byte("bar")); err != nil {
		return err
	}
	return nil
}); err != nil {
	log.Fatal(err)
}

// Read value back in a different read-only transaction.
if err := db.View(func(tx *memcache.Tx) error {
	value := tx.Bucket([]byte("widgets")).Get([]byte("foo"))
	fmt.Printf("The value of 'foo' is: %s\n", value)
	return nil
}); err != nil {
	log.Fatal(err)
}

// Close database to release file lock.
if err := db.Close(); err != nil {
	log.Fatal(err)
}

Output:

The value of 'foo' is: bar

func (*Bucket) Root 

func (b *Bucket) Root() pgid

Root returns the root of the bucket.

func (*Bucket) Sequence 

func (b *Bucket) Sequence() uint64

Sequence returns the current integer for the bucket without incrementing it.

func (*Bucket) SetSequence 

func (b *Bucket) SetSequence(v uint64) error

SetSequence updates the sequence number for the bucket.

func (*Bucket) Stats 

func (b *Bucket) Stats() BucketStats

Stat returns stats on a bucket.

func (*Bucket) Tx 

func (b *Bucket) Tx() *Tx

Tx returns the tx of the bucket.

func (*Bucket) Writable 

func (b *Bucket) Writable() bool

Writable returns whether the bucket is writable.

type BucketStats 

type BucketStats struct {
	// Page count statistics.
	BranchPageN     int // number of logical branch pages
	BranchOverflowN int // number of physical branch overflow pages
	LeafPageN       int // number of logical leaf pages
	LeafOverflowN   int // number of physical leaf overflow pages

	// Tree statistics.
	KeyN  int // number of keys/value pairs
	Depth int // number of levels in B+tree

	// Page size utilization.
	BranchAlloc int // bytes allocated for physical branch pages
	BranchInuse int // bytes actually used for branch data
	LeafAlloc   int // bytes allocated for physical leaf pages
	LeafInuse   int // bytes actually used for leaf data

	// Bucket statistics
	BucketN           int // total number of buckets including the top bucket
	InlineBucketN     int // total number on inlined buckets
	InlineBucketInuse int // bytes used for inlined buckets (also accounted for in LeafInuse)
}

BucketStats records statistics about resources used by a bucket.

func (*BucketStats) Add 

func (s *BucketStats) Add(other BucketStats)

type Cursor 

type Cursor struct {
	// contains filtered or unexported fields
}

Cursor represents an iterator that can traverse over all key/value pairs in a bucket in sorted order.

Cursors see nested buckets with value == nil.

Cursors can be obtained from a transaction and are valid as long as the transaction is open.

Keys and values returned from the cursor are only valid for the life of the transaction.

Changing data while traversing with a cursor may cause it to be invalidated and return unexpected keys and/or values. You must reposition your cursor after mutating data.

Example 
// Open the database.
db, err := memcache.Open(tempfile(), 0666, nil)
if err != nil {
	log.Fatal(err)
}
defer os.Remove(db.Path())

// Start a read-write transaction.
if err := db.Update(func(tx *memcache.Tx) error {
	// Create a new bucket.
	b, err := tx.CreateBucket([]byte("animals"))
	if err != nil {
		return err
	}

	// Insert data into a bucket.
	if err := b.Put([]byte("dog"), []byte("fun")); err != nil {
		log.Fatal(err)
	}
	if err := b.Put([]byte("cat"), []byte("lame")); err != nil {
		log.Fatal(err)
	}
	if err := b.Put([]byte("liger"), []byte("awesome")); err != nil {
		log.Fatal(err)
	}

	// Create a cursor for iteration.
	c := b.Cursor()

	// Iterate over items in sorted key order. This starts from the
	// first key/value pair and updates the k/v variables to the
	// next key/value on each iteration.
	//
	// The loop finishes at the end of the cursor when a nil key is returned.
	for k, v := c.First(); k != nil; k, v = c.Next() {
		fmt.Printf("A %s is %s.\n", k, v)
	}

	return nil
}); err != nil {
	log.Fatal(err)
}

if err := db.Close(); err != nil {
	log.Fatal(err)
}

Output:

A cat is lame.
A dog is fun.
A liger is awesome.
Example (Reverse) 
// Open the database.
db, err := memcache.Open(tempfile(), 0666, nil)
if err != nil {
	log.Fatal(err)
}
defer os.Remove(db.Path())

// Start a read-write transaction.
if err := db.Update(func(tx *memcache.Tx) error {
	// Create a new bucket.
	b, err := tx.CreateBucket([]byte("animals"))
	if err != nil {
		return err
	}

	// Insert data into a bucket.
	if err := b.Put([]byte("dog"), []byte("fun")); err != nil {
		log.Fatal(err)
	}
	if err := b.Put([]byte("cat"), []byte("lame")); err != nil {
		log.Fatal(err)
	}
	if err := b.Put([]byte("liger"), []byte("awesome")); err != nil {
		log.Fatal(err)
	}

	// Create a cursor for iteration.
	c := b.Cursor()

	// Iterate over items in reverse sorted key order. This starts
	// from the last key/value pair and updates the k/v variables to
	// the previous key/value on each iteration.
	//
	// The loop finishes at the beginning of the cursor when a nil key
	// is returned.
	for k, v := c.Last(); k != nil; k, v = c.Prev() {
		fmt.Printf("A %s is %s.\n", k, v)
	}

	return nil
}); err != nil {
	log.Fatal(err)
}

// Close the database to release the file lock.
if err := db.Close(); err != nil {
	log.Fatal(err)
}

Output:

A liger is awesome.
A dog is fun.
A cat is lame.

func (*Cursor) Bucket 

func (c *Cursor) Bucket() *Bucket

Bucket returns the bucket that this cursor was created from.

func (*Cursor) Delete 

func (c *Cursor) Delete() error

Delete removes the current key/value under the cursor from the bucket. Delete fails if current key/value is a bucket or if the transaction is not writable.

func (*Cursor) First 

func (c *Cursor) First() (key []byte, value []byte)

First moves the cursor to the first item in the bucket and returns its key and value. If the bucket is empty then a nil key and value are returned. The returned key and value are only valid for the life of the transaction.

func (*Cursor) Last 

func (c *Cursor) Last() (key []byte, value []byte)

Last moves the cursor to the last item in the bucket and returns its key and value. If the bucket is empty then a nil key and value are returned. The returned key and value are only valid for the life of the transaction.

func (*Cursor) Next 

func (c *Cursor) Next() (key []byte, value []byte)

Next moves the cursor to the next item in the bucket and returns its key and value. If the cursor is at the end of the bucket then a nil key and value are returned. The returned key and value are only valid for the life of the transaction.

func (*Cursor) Prev 

func (c *Cursor) Prev() (key []byte, value []byte)

Prev moves the cursor to the previous item in the bucket and returns its key and value. If the cursor is at the beginning of the bucket then a nil key and value are returned. The returned key and value are only valid for the life of the transaction.

func (*Cursor) Seek 

func (c *Cursor) Seek(seek []byte) (key []byte, value []byte)

Seek moves the cursor to a given key and returns it. If the key does not exist then the next key is used. If no keys follow, a nil key is returned. The returned key and value are only valid for the life of the transaction.

type DB 

type DB struct {
	// When enabled, the database will perform a Check() after every commit.
	// A panic is issued if the database is in an inconsistent state. This
	// flag has a large performance impact so it should only be used for
	// debugging purposes.
	StrictMode bool

	// Setting the NoSync flag will cause the database to skip fsync()
	// calls after each commit. This can be useful when bulk loading data
	// into a database and you can restart the bulk load in the event of
	// a system failure or database corruption. Do not set this flag for
	// normal use.
	//
	// If the package global IgnoreNoSync constant is true, this value is
	// ignored.  See the comment on that constant for more details.
	//
	// THIS IS UNSAFE. PLEASE USE WITH CAUTION.
	NoSync bool

	// When true, skips syncing freelist to disk. This improves the database
	// write performance under normal operation, but requires a full database
	// re-sync during recovery.
	NoFreelistSync bool

	// FreelistType sets the backend freelist type. There are two options. Array which is simple but endures
	// dramatic performance degradation if database is large and framentation in freelist is common.
	// The alternative one is using hashmap, it is faster in almost all circumstances
	// but it doesn't guarantee that it offers the smallest page id available. In normal case it is safe.
	// The default type is array
	FreelistType FreelistType

	// When true, skips the truncate call when growing the database.
	// Setting this to true is only safe on non-ext3/ext4 systems.
	// Skipping truncation avoids preallocation of hard drive space and
	// bypasses a truncate() and fsync() syscall on remapping.
	NoGrowSync bool

	// If you want to read the entire database fast, you can set MmapFlag to
	// syscall.MAP_POPULATE on Linux 2.6.23+ for sequential read-ahead.
	MmapFlags int

	// MaxBatchSize is the maximum size of a batch. Default value is
	// copied from DefaultMaxBatchSize in Open.
	//
	// If <=0, disables batching.
	//
	// Do not change concurrently with calls to Batch.
	MaxBatchSize int

	// MaxBatchDelay is the maximum delay before a batch starts.
	// Default value is copied from DefaultMaxBatchDelay in Open.
	//
	// If <=0, effectively disables batching.
	//
	// Do not change concurrently with calls to Batch.
	MaxBatchDelay time.Duration

	// AllocSize is the amount of space allocated when the database
	// needs to create new pages. This is done to amortize the cost
	// of truncate() and fsync() when growing the data file.
	AllocSize int

	// Mlock locks database file in memory when set to true.
	// It prevents major page faults, however used memory can't be reclaimed.
	//
	// Supported only on Unix via mlock/munlock syscalls.
	Mlock bool
	// contains filtered or unexported fields
}

DB represents a collection of buckets persisted to a file on disk. All data access is performed through transactions which can be obtained through the DB. All the functions on DB will return a ErrDatabaseNotOpen if accessed before Open() is called.

func Open 

func Open(path string, mode os.FileMode, options *Options) (*DB, error)

Open creates and opens an In-Memory database at the given path. If the file does not exist then it will be created automatically. Passing in nil options will cause Bhojpur Cache to open the database with the default options.

func (*DB) Batch 

func (db *DB) Batch(fn func(*Tx) error) error

Batch calls fn as part of a batch. It behaves similar to Update, except:

1. concurrent Batch calls can be combined into a single Bhojpur Cache transaction.

2. the function passed to Batch may be called multiple times, regardless of whether it returns error or not.

This means that Batch function side effects must be idempotent and take permanent effect only after a successful return is seen in caller.

The maximum batch size and delay can be adjusted with DB.MaxBatchSize and DB.MaxBatchDelay, respectively.

Batch is only useful when there are multiple goroutines calling it.

func (*DB) Begin 

func (db *DB) Begin(writable bool) (*Tx, error)

Begin starts a new transaction. Multiple read-only transactions can be used concurrently but only one write transaction can be used at a time. Starting multiple write transactions will cause the calls to block and be serialized until the current write transaction finishes.

Transactions should not be dependent on one another. Opening a read transaction and a write transaction in the same goroutine can cause the writer to deadlock because the database periodically needs to re-mmap itself as it grows and it cannot do that while a read transaction is open.

If a long running read transaction (for example, a snapshot transaction) is needed, you might want to set DB.InitialMmapSize to a large enough value to avoid potential blocking of write transaction.

IMPORTANT: You must close read-only transactions after you are finished or else the database will not reclaim old pages.

Example 
package main

import (
	"fmt"
	"log"
	"os"

	memcache "github.com/bhojpur/cache/pkg/memory"
)

func main() {
	// Open the database.
	db, err := memcache.Open(tempfile(), 0666, nil)
	if err != nil {
		log.Fatal(err)
	}
	defer os.Remove(db.Path())

	// Create a bucket using a read-write transaction.
	if err = db.Update(func(tx *memcache.Tx) error {
		_, err := tx.CreateBucket([]byte("widgets"))
		return err
	}); err != nil {
		log.Fatal(err)
	}

	// Create several keys in a transaction.
	tx, err := db.Begin(true)
	if err != nil {
		log.Fatal(err)
	}
	b := tx.Bucket([]byte("widgets"))
	if err = b.Put([]byte("john"), []byte("blue")); err != nil {
		log.Fatal(err)
	}
	if err = b.Put([]byte("abby"), []byte("red")); err != nil {
		log.Fatal(err)
	}
	if err = b.Put([]byte("zephyr"), []byte("purple")); err != nil {
		log.Fatal(err)
	}
	if err = tx.Commit(); err != nil {
		log.Fatal(err)
	}

	// Iterate over the values in sorted key order.
	tx, err = db.Begin(false)
	if err != nil {
		log.Fatal(err)
	}
	c := tx.Bucket([]byte("widgets")).Cursor()
	for k, v := c.First(); k != nil; k, v = c.Next() {
		fmt.Printf("%s likes %s\n", k, v)
	}

	if err = tx.Rollback(); err != nil {
		log.Fatal(err)
	}

	if err = db.Close(); err != nil {
		log.Fatal(err)
	}

}

// tempfile returns a temporary file path.
func tempfile() string {
	f, err := os.CreateTemp("", "bhojpur-cache-")
	if err != nil {
		panic(err)
	}
	if err := f.Close(); err != nil {
		panic(err)
	}
	if err := os.Remove(f.Name()); err != nil {
		panic(err)
	}
	return f.Name()
}

Output:

abby likes red
john likes blue
zephyr likes purple

func (*DB) Close 

func (db *DB) Close() error

Close releases all database resources. It will block waiting for any open transactions to finish before closing the database and returning.

func (*DB) GoString 

func (db *DB) GoString() string

GoString returns the Go string representation of the database.

func (*DB) Info 

func (db *DB) Info() *Info

This is for internal access to the raw data bytes from the C cursor, use carefully, or not at all.

func (*DB) IsReadOnly 

func (db *DB) IsReadOnly() bool

func (*DB) Path 

func (db *DB) Path() string

Path returns the path to currently open database file.

func (*DB) Stats 

func (db *DB) Stats() Stats

Stats retrieves ongoing performance stats for the database. This is only updated when a transaction closes.

func (*DB) String 

func (db *DB) String() string

String returns the string representation of the database.

func (*DB) Sync 

func (db *DB) Sync() error

Sync executes fdatasync() against the database file handle.

This is not necessary under normal operation, however, if you use NoSync then it allows you to force the database file to sync against the disk.

func (*DB) Update 

func (db *DB) Update(fn func(*Tx) error) error

Update executes a function within the context of a read-write managed transaction. If no error is returned from the function then the transaction is committed. If an error is returned then the entire transaction is rolled back. Any error that is returned from the function or returned from the commit is returned from the Update() method.

Attempting to manually commit or rollback within the function will cause a panic.

Example 
package main

import (
	"fmt"
	"log"
	"os"

	memcache "github.com/bhojpur/cache/pkg/memory"
)

func main() {
	// Open the database.
	db, err := memcache.Open(tempfile(), 0666, nil)
	if err != nil {
		log.Fatal(err)
	}
	defer os.Remove(db.Path())

	// Execute several commands within a read-write transaction.
	if err := db.Update(func(tx *memcache.Tx) error {
		b, err := tx.CreateBucket([]byte("widgets"))
		if err != nil {
			return err
		}
		if err := b.Put([]byte("foo"), []byte("bar")); err != nil {
			return err
		}
		return nil
	}); err != nil {
		log.Fatal(err)
	}

	// Read the value back from a separate read-only transaction.
	if err := db.View(func(tx *memcache.Tx) error {
		value := tx.Bucket([]byte("widgets")).Get([]byte("foo"))
		fmt.Printf("The value of 'foo' is: %s\n", value)
		return nil
	}); err != nil {
		log.Fatal(err)
	}

	// Close database to release the file lock.
	if err := db.Close(); err != nil {
		log.Fatal(err)
	}

}

// tempfile returns a temporary file path.
func tempfile() string {
	f, err := os.CreateTemp("", "bhojpur-cache-")
	if err != nil {
		panic(err)
	}
	if err := f.Close(); err != nil {
		panic(err)
	}
	if err := os.Remove(f.Name()); err != nil {
		panic(err)
	}
	return f.Name()
}

Output:

The value of 'foo' is: bar

func (*DB) View 

func (db *DB) View(fn func(*Tx) error) error

View executes a function within the context of a managed read-only transaction. Any error that is returned from the function is returned from the View() method.

Attempting to manually rollback within the function will cause a panic.

Example 
package main

import (
	"fmt"
	"log"
	"os"

	memcache "github.com/bhojpur/cache/pkg/memory"
)

func main() {
	// Open the database.
	db, err := memcache.Open(tempfile(), 0666, nil)
	if err != nil {
		log.Fatal(err)
	}
	defer os.Remove(db.Path())

	// Insert data into a bucket.
	if err := db.Update(func(tx *memcache.Tx) error {
		b, err := tx.CreateBucket([]byte("people"))
		if err != nil {
			return err
		}
		if err := b.Put([]byte("john"), []byte("doe")); err != nil {
			return err
		}
		if err := b.Put([]byte("susy"), []byte("que")); err != nil {
			return err
		}
		return nil
	}); err != nil {
		log.Fatal(err)
	}

	// Access data from within a read-only transactional block.
	if err := db.View(func(tx *memcache.Tx) error {
		v := tx.Bucket([]byte("people")).Get([]byte("john"))
		fmt.Printf("John's last name is %s.\n", v)
		return nil
	}); err != nil {
		log.Fatal(err)
	}

	// Close database to release the file lock.
	if err := db.Close(); err != nil {
		log.Fatal(err)
	}

}

// tempfile returns a temporary file path.
func tempfile() string {
	f, err := os.CreateTemp("", "bhojpur-cache-")
	if err != nil {
		panic(err)
	}
	if err := f.Close(); err != nil {
		panic(err)
	}
	if err := os.Remove(f.Name()); err != nil {
		panic(err)
	}
	return f.Name()
}

Output:

John's last name is doe.

type FreelistType 

type FreelistType string

FreelistType is the type of the freelist backend

type Info 

type Info struct {
	Data     uintptr
	PageSize int
}

type Options 

type Options struct {
	// Timeout is the amount of time to wait to obtain a file lock.
	// When set to zero it will wait indefinitely. This option is only
	// available on Darwin and Linux.
	Timeout time.Duration

	// Sets the DB.NoGrowSync flag before memory mapping the file.
	NoGrowSync bool

	// Do not sync freelist to disk. This improves the database write performance
	// under normal operation, but requires a full database re-sync during recovery.
	NoFreelistSync bool

	// FreelistType sets the backend freelist type. There are two options. Array which is simple but endures
	// dramatic performance degradation if database is large and framentation in freelist is common.
	// The alternative one is using hashmap, it is faster in almost all circumstances
	// but it doesn't guarantee that it offers the smallest page id available. In normal case it is safe.
	// The default type is array
	FreelistType FreelistType

	// Open database in read-only mode. Uses flock(..., LOCK_SH |LOCK_NB) to
	// grab a shared lock (UNIX).
	ReadOnly bool

	// Sets the DB.MmapFlags flag before memory mapping the file.
	MmapFlags int

	// InitialMmapSize is the initial mmap size of the database
	// in bytes. Read transactions won't block write transaction
	// if the InitialMmapSize is large enough to hold database mmap
	// size. (See DB.Begin for more information)
	//
	// If <=0, the initial map size is 0.
	// If initialMmapSize is smaller than the previous database size,
	// it takes no effect.
	InitialMmapSize int

	// PageSize overrides the default OS page size.
	PageSize int

	// NoSync sets the initial value of DB.NoSync. Normally this can just be
	// set directly on the DB itself when returned from Open(), but this option
	// is useful in APIs which expose Options but not the underlying DB.
	NoSync bool

	// OpenFile is used to open files. It defaults to os.OpenFile. This option
	// is useful for writing hermetic tests.
	OpenFile func(string, int, os.FileMode) (*os.File, error)

	// Mlock locks database file in memory when set to true.
	// It prevents potential page faults, however
	// used memory can't be reclaimed. (UNIX only)
	Mlock bool
}

Options represents the options that can be set when opening a database.

type PageInfo 

type PageInfo struct {
	ID            int
	Type          string
	Count         int
	OverflowCount int
}

PageInfo represents human readable information about a page.

type Stats 

type Stats struct {
	// Freelist stats
	FreePageN     int // total number of free pages on the freelist
	PendingPageN  int // total number of pending pages on the freelist
	FreeAlloc     int // total bytes allocated in free pages
	FreelistInuse int // total bytes used by the freelist

	// Transaction stats
	TxN     int // total number of started read transactions
	OpenTxN int // number of currently open read transactions

	TxStats TxStats // global, ongoing stats.
}

Stats represents statistics about the database.

func (*Stats) Sub 

func (s *Stats) Sub(other *Stats) Stats

Sub calculates and returns the difference between two sets of database stats. This is useful when obtaining stats at two different points and time and you need the performance counters that occurred within that time span.

type Tx 

type Tx struct {

	// WriteFlag specifies the flag for write-related methods like WriteTo().
	// Tx opens the database file with the specified flag to copy the data.
	//
	// By default, the flag is unset, which works well for mostly in-memory
	// workloads. For databases that are much larger than available RAM,
	// set the flag to syscall.O_DIRECT to avoid trashing the page cache.
	WriteFlag int
	// contains filtered or unexported fields
}

Tx represents a read-only or read/write transaction on the database. Read-only transactions can be used for retrieving values for keys and creating cursors. Read/write transactions can create and remove buckets and create and remove keys.

IMPORTANT: You must commit or rollback transactions when you are done with them. Pages can not be reclaimed by the writer until no more transactions are using them. A long running read transaction can cause the database to quickly grow.

func (*Tx) Bucket 

func (tx *Tx) Bucket(name []byte) *Bucket

Bucket retrieves a bucket by name. Returns nil if the bucket does not exist. The bucket instance is only valid for the lifetime of the transaction.

func (*Tx) Check 

func (tx *Tx) Check() <-chan error

Check performs several consistency checks on the database for this transaction. An error is returned if any inconsistency is found.

It can be safely run concurrently on a writable transaction. However, this incurs a high cost for large databases and databases with a lot of subbuckets because of caching. This overhead can be removed if running on a read-only transaction, however, it is not safe to execute other writer transactions at the same time.

func (*Tx) Commit 

func (tx *Tx) Commit() error

Commit writes all changes to disk and updates the meta page. Returns an error if a disk write error occurs, or if Commit is called on a read-only transaction.

func (*Tx) Copy 

func (tx *Tx) Copy(w io.Writer) error

Copy writes the entire database to a writer. This function exists for backwards compatibility.

Deprecated; Use WriteTo() instead.

func (*Tx) CopyFile 

func (tx *Tx) CopyFile(path string, mode os.FileMode) error

CopyFile copies the entire database to file at the given path. A reader transaction is maintained during the copy so it is safe to continue using the database while a copy is in progress.

Example 
// Open the database.
db, err := memcache.Open(tempfile(), 0666, nil)
if err != nil {
	log.Fatal(err)
}
defer os.Remove(db.Path())

// Create a bucket and a key.
if err := db.Update(func(tx *memcache.Tx) error {
	b, err := tx.CreateBucket([]byte("widgets"))
	if err != nil {
		return err
	}
	if err := b.Put([]byte("foo"), []byte("bar")); err != nil {
		return err
	}
	return nil
}); err != nil {
	log.Fatal(err)
}

// Copy the database to another file.
toFile := tempfile()
if err := db.View(func(tx *memcache.Tx) error {
	return tx.CopyFile(toFile, 0666)
}); err != nil {
	log.Fatal(err)
}
defer os.Remove(toFile)

// Open the cloned database.
db2, err := memcache.Open(toFile, 0666, nil)
if err != nil {
	log.Fatal(err)
}

// Ensure that the key exists in the copy.
if err := db2.View(func(tx *memcache.Tx) error {
	value := tx.Bucket([]byte("widgets")).Get([]byte("foo"))
	fmt.Printf("The value for 'foo' in the clone is: %s\n", value)
	return nil
}); err != nil {
	log.Fatal(err)
}

// Close database to release file lock.
if err := db.Close(); err != nil {
	log.Fatal(err)
}

if err := db2.Close(); err != nil {
	log.Fatal(err)
}

Output:

The value for 'foo' in the clone is: bar

func (*Tx) CreateBucket 

func (tx *Tx) CreateBucket(name []byte) (*Bucket, error)

CreateBucket creates a new bucket. Returns an error if the bucket already exists, if the bucket name is blank, or if the bucket name is too long. The bucket instance is only valid for the lifetime of the transaction.

func (*Tx) CreateBucketIfNotExists 

func (tx *Tx) CreateBucketIfNotExists(name []byte) (*Bucket, error)

CreateBucketIfNotExists creates a new bucket if it doesn't already exist. Returns an error if the bucket name is blank, or if the bucket name is too long. The bucket instance is only valid for the lifetime of the transaction.

func (*Tx) Cursor 

func (tx *Tx) Cursor() *Cursor

Cursor creates a cursor associated with the root bucket. All items in the cursor will return a nil value because all root bucket keys point to buckets. The cursor is only valid as long as the transaction is open. Do not use a cursor after the transaction is closed.

func (*Tx) DB 

func (tx *Tx) DB() *DB

DB returns a reference to the database that created the transaction.

func (*Tx) DeleteBucket 

func (tx *Tx) DeleteBucket(name []byte) error

DeleteBucket deletes a bucket. Returns an error if the bucket cannot be found or if the key represents a non-bucket value.

func (*Tx) ForEach 

func (tx *Tx) ForEach(fn func(name []byte, b *Bucket) error) error

ForEach executes a function for each bucket in the root. If the provided function returns an error then the iteration is stopped and the error is returned to the caller.

func (*Tx) ID 

func (tx *Tx) ID() int

ID returns the transaction id.

func (*Tx) OnCommit 

func (tx *Tx) OnCommit(fn func())

OnCommit adds a handler function to be executed after the transaction successfully commits.

func (*Tx) Page 

func (tx *Tx) Page(id int) (*PageInfo, error)

Page returns page information for a given page number. This is only safe for concurrent use when used by a writable transaction.

func (*Tx) Rollback 

func (tx *Tx) Rollback() error

Rollback closes the transaction and ignores all previous updates. Read-only transactions must be rolled back and not committed.

Example 
// Open the database.
db, err := memcache.Open(tempfile(), 0666, nil)
if err != nil {
	log.Fatal(err)
}
defer os.Remove(db.Path())

// Create a bucket.
if err := db.Update(func(tx *memcache.Tx) error {
	_, err := tx.CreateBucket([]byte("widgets"))
	return err
}); err != nil {
	log.Fatal(err)
}

// Set a value for a key.
if err := db.Update(func(tx *memcache.Tx) error {
	return tx.Bucket([]byte("widgets")).Put([]byte("foo"), []byte("bar"))
}); err != nil {
	log.Fatal(err)
}

// Update the key but rollback the transaction so it never saves.
tx, err := db.Begin(true)
if err != nil {
	log.Fatal(err)
}
b := tx.Bucket([]byte("widgets"))
if err := b.Put([]byte("foo"), []byte("baz")); err != nil {
	log.Fatal(err)
}
if err := tx.Rollback(); err != nil {
	log.Fatal(err)
}

// Ensure that our original value is still set.
if err := db.View(func(tx *memcache.Tx) error {
	value := tx.Bucket([]byte("widgets")).Get([]byte("foo"))
	fmt.Printf("The value for 'foo' is still: %s\n", value)
	return nil
}); err != nil {
	log.Fatal(err)
}

// Close database to release file lock.
if err := db.Close(); err != nil {
	log.Fatal(err)
}

Output:

The value for 'foo' is still: bar

func (*Tx) Size 

func (tx *Tx) Size() int64

Size returns current database size in bytes as seen by this transaction.

func (*Tx) Stats 

func (tx *Tx) Stats() TxStats

Stats retrieves a copy of the current transaction statistics.

func (*Tx) Writable 

func (tx *Tx) Writable() bool

Writable returns whether the transaction can perform write operations.

func (*Tx) WriteTo 

func (tx *Tx) WriteTo(w io.Writer) (n int64, err error)

WriteTo writes the entire database to a writer. If err == nil then exactly tx.Size() bytes will be written into the writer.

type TxStats 

type TxStats struct {
	// Page statistics.
	PageCount int // number of page allocations
	PageAlloc int // total bytes allocated

	// Cursor statistics.
	CursorCount int // number of cursors created

	// Node statistics
	NodeCount int // number of node allocations
	NodeDeref int // number of node dereferences

	// Rebalance statistics.
	Rebalance     int           // number of node rebalances
	RebalanceTime time.Duration // total time spent rebalancing

	// Split/Spill statistics.
	Split     int           // number of nodes split
	Spill     int           // number of nodes spilled
	SpillTime time.Duration // total time spent spilling

	// Write statistics.
	Write     int           // number of writes performed
	WriteTime time.Duration // total time spent writing to disk
}

TxStats represents statistics about the actions performed by the transaction.

func (*TxStats) Sub 

func (s *TxStats) Sub(other *TxStats) TxStats

Sub calculates and returns the difference between two sets of transaction stats. This is useful when obtaining stats at two different points and time and you need the performance counters that occurred within that time span.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.