buntdb

package module
v1.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 7, 2022 License: MIT Imports: 15 Imported by: 0

README

BuntDB
Godoc LICENSE

BuntDB is a low-level, in-memory, key/value store in pure Go. It persists to disk, is ACID compliant, and uses locking for multiple readers and a single writer. It supports custom indexes and geospatial data. It's ideal for projects that need a dependable database and favor speed over data size.

Features

Getting Started

Installing

To start using BuntDB, install Go and run go get:

$ go get -u github.com/tidwall/buntdb

This will retrieve the library.

Opening a database

The primary object in BuntDB is a DB. To open or create your database, use the buntdb.Open() function:

package main

import (
	"log"

	"github.com/tidwall/buntdb"
)

func main() {
	// Open the data.db file. It will be created if it doesn't exist.
	db, err := buntdb.Open("data.db")
	if err != nil {
		log.Fatal(err)
	}
	defer db.Close()

	...
}

It's also possible to open a database that does not persist to disk by using :memory: as the path of the file.

buntdb.Open(":memory:") // Open a file that does not persist to disk.

Transactions

All reads and writes must be performed from inside a transaction. BuntDB can have one write transaction opened at a time, but can have many concurrent read transactions. Each transaction maintains a stable view of the database. In other words, once a transaction has begun, the data for that transaction cannot be changed by other transactions.

Transactions run in a function that exposes a Tx object, which represents the transaction state. While inside a transaction, all database operations should be performed using this object. You should never access the origin DB object while inside a transaction. Doing so may have side-effects, such as blocking your application.

When a transaction fails, it will roll back, and revert all changes that occurred to the database during that transaction. There's a single return value that you can use to close the transaction. For read/write transactions, returning an error this way will force the transaction to roll back. When a read/write transaction succeeds all changes are persisted to disk.

Read-only Transactions

A read-only transaction should be used when you don't need to make changes to the data. The advantage of a read-only transaction is that there can be many running concurrently.

err := db.View(func(tx *buntdb.Tx) error {
	...
	return nil
})
Read/write Transactions

A read/write transaction is used when you need to make changes to your data. There can only be one read/write transaction running at a time. So make sure you close it as soon as you are done with it.

err := db.Update(func(tx *buntdb.Tx) error {
	...
	return nil
})

Setting and getting key/values

To set a value you must open a read/write transaction:

err := db.Update(func(tx *buntdb.Tx) error {
	_, _, err := tx.Set("mykey", "myvalue", nil)
	return err
})

To get the value:

err := db.View(func(tx *buntdb.Tx) error {
	val, err := tx.Get("mykey")
	if err != nil{
		return err
	}
	fmt.Printf("value is %s\n", val)
	return nil
})

Getting non-existent values will cause an ErrNotFound error.

Iterating

All keys/value pairs are ordered in the database by the key. To iterate over the keys:

err := db.View(func(tx *buntdb.Tx) error {
	err := tx.Ascend("", func(key, value string) bool {
		fmt.Printf("key: %s, value: %s\n", key, value)
		return true // continue iteration
	})
	return err
})

There is also AscendGreaterOrEqual, AscendLessThan, AscendRange, AscendEqual, Descend, DescendLessOrEqual, DescendGreaterThan, DescendRange, and DescendEqual. Please see the documentation for more information on these functions.

Custom Indexes

Initially all data is stored in a single B-tree with each item having one key and one value. All of these items are ordered by the key. This is great for quickly getting a value from a key or iterating over the keys. Feel free to peruse the B-tree implementation.

You can also create custom indexes that allow for ordering and iterating over values. A custom index also uses a B-tree, but it's more flexible because it allows for custom ordering.

For example, let's say you want to create an index for ordering names:

db.CreateIndex("names", "*", buntdb.IndexString)

This will create an index named names which stores and sorts all values. The second parameter is a pattern that is used to filter on keys. A * wildcard argument means that we want to accept all keys. IndexString is a built-in function that performs case-insensitive ordering on the values

Now you can add various names:

db.Update(func(tx *buntdb.Tx) error {
	tx.Set("user:0:name", "tom", nil)
	tx.Set("user:1:name", "Randi", nil)
	tx.Set("user:2:name", "jane", nil)
	tx.Set("user:4:name", "Janet", nil)
	tx.Set("user:5:name", "Paula", nil)
	tx.Set("user:6:name", "peter", nil)
	tx.Set("user:7:name", "Terri", nil)
	return nil
})

Finally you can iterate over the index:

db.View(func(tx *buntdb.Tx) error {
	tx.Ascend("names", func(key, val string) bool {
	fmt.Printf(buf, "%s %s\n", key, val)
		return true
	})
	return nil
})

The output should be:

user:2:name jane
user:4:name Janet
user:5:name Paula
user:6:name peter
user:1:name Randi
user:7:name Terri
user:0:name tom

The pattern parameter can be used to filter on keys like this:

db.CreateIndex("names", "user:*", buntdb.IndexString)

Now only items with keys that have the prefix user: will be added to the names index.

Built-in types

Along with IndexString, there is also IndexInt, IndexUint, and IndexFloat. These are built-in types for indexing. You can choose to use these or create your own.

So to create an index that is numerically ordered on an age key, we could use:

db.CreateIndex("ages", "user:*:age", buntdb.IndexInt)

And then add values:

db.Update(func(tx *buntdb.Tx) error {
	tx.Set("user:0:age", "35", nil)
	tx.Set("user:1:age", "49", nil)
	tx.Set("user:2:age", "13", nil)
	tx.Set("user:4:age", "63", nil)
	tx.Set("user:5:age", "8", nil)
	tx.Set("user:6:age", "3", nil)
	tx.Set("user:7:age", "16", nil)
	return nil
})

db.View(func(tx *buntdb.Tx) error {
	tx.Ascend("ages", func(key, val string) bool {
	fmt.Printf(buf, "%s %s\n", key, val)
		return true
	})
	return nil
})

The output should be:

user:6:age 3
user:5:age 8
user:2:age 13
user:7:age 16
user:0:age 35
user:1:age 49
user:4:age 63

Spatial Indexes

BuntDB has support for spatial indexes by storing rectangles in an R-tree. An R-tree is organized in a similar manner as a B-tree, and both are balanced trees. But, an R-tree is special because it can operate on data that is in multiple dimensions. This is super handy for Geospatial applications.

To create a spatial index use the CreateSpatialIndex function:

db.CreateSpatialIndex("fleet", "fleet:*:pos", buntdb.IndexRect)

Then IndexRect is a built-in function that converts rect strings to a format that the R-tree can use. It's easy to use this function out of the box, but you might find it better to create a custom one that renders from a different format, such as Well-known text or GeoJSON.

To add some lon,lat points to the fleet index:

db.Update(func(tx *buntdb.Tx) error {
	tx.Set("fleet:0:pos", "[-115.567 33.532]", nil)
	tx.Set("fleet:1:pos", "[-116.671 35.735]", nil)
	tx.Set("fleet:2:pos", "[-113.902 31.234]", nil)
	return nil
})

And then you can run the Intersects function on the index:

db.View(func(tx *buntdb.Tx) error {
	tx.Intersects("fleet", "[-117 30],[-112 36]", func(key, val string) bool {
		...
		return true
	})
	return nil
})

This will get all three positions.

k-Nearest Neighbors

Use the Nearby function to get all the positions in order of nearest to farthest :

db.View(func(tx *buntdb.Tx) error {
	tx.Nearby("fleet", "[-113 33]", func(key, val string, dist float64) bool {
		...
		return true
	})
	return nil
})
Spatial bracket syntax

The bracket syntax [-117 30],[-112 36] is unique to BuntDB, and it's how the built-in rectangles are processed. But, you are not limited to this syntax. Whatever Rect function you choose to use during CreateSpatialIndex will be used to process the parameter, in this case it's IndexRect.

  • 2D rectangle: [10 15],[20 25] Min XY: "10x15", Max XY: "20x25"

  • 3D rectangle: [10 15 12],[20 25 18] Min XYZ: "10x15x12", Max XYZ: "20x25x18"

  • 2D point: [10 15] XY: "10x15"

  • LonLat point: [-112.2693 33.5123] LatLon: "33.5123 -112.2693"

  • LonLat bounding box: [-112.26 33.51],[-112.18 33.67] Min LatLon: "33.51 -112.26", Max LatLon: "33.67 -112.18"

Notice: The longitude is the Y axis and is on the left, and latitude is the X axis and is on the right.

You can also represent Infinity by using -inf and +inf. For example, you might have the following points ([X Y M] where XY is a point and M is a timestamp):

[3 9 1]
[3 8 2]
[4 8 3]
[4 7 4]
[5 7 5]
[5 6 6]

You can then do a search for all points with M between 2-4 by calling Intersects.

tx.Intersects("points", "[-inf -inf 2],[+inf +inf 4]", func(key, val string) bool {
	println(val)
	return true
})

Which will return:

[3 8 2]
[4 8 3]
[4 7 4]

JSON Indexes

Indexes can be created on individual fields inside JSON documents. BuntDB uses GJSON under the hood.

For example:

package main

import (
	"fmt"

	"github.com/tidwall/buntdb"
)

func main() {
	db, _ := buntdb.Open(":memory:")
	db.CreateIndex("last_name", "*", buntdb.IndexJSON("name.last"))
	db.CreateIndex("age", "*", buntdb.IndexJSON("age"))
	db.Update(func(tx *buntdb.Tx) error {
		tx.Set("1", `{"name":{"first":"Tom","last":"Johnson"},"age":38}`, nil)
		tx.Set("2", `{"name":{"first":"Janet","last":"Prichard"},"age":47}`, nil)
		tx.Set("3", `{"name":{"first":"Carol","last":"Anderson"},"age":52}`, nil)
		tx.Set("4", `{"name":{"first":"Alan","last":"Cooper"},"age":28}`, nil)
		return nil
	})
	db.View(func(tx *buntdb.Tx) error {
		fmt.Println("Order by last name")
		tx.Ascend("last_name", func(key, value string) bool {
			fmt.Printf("%s: %s\n", key, value)
			return true
		})
		fmt.Println("Order by age")
		tx.Ascend("age", func(key, value string) bool {
			fmt.Printf("%s: %s\n", key, value)
			return true
		})
		fmt.Println("Order by age range 30-50")
		tx.AscendRange("age", `{"age":30}`, `{"age":50}`, func(key, value string) bool {
			fmt.Printf("%s: %s\n", key, value)
			return true
		})
		return nil
	})
}

Results:

Order by last name
3: {"name":{"first":"Carol","last":"Anderson"},"age":52}
4: {"name":{"first":"Alan","last":"Cooper"},"age":28}
1: {"name":{"first":"Tom","last":"Johnson"},"age":38}
2: {"name":{"first":"Janet","last":"Prichard"},"age":47}

Order by age
4: {"name":{"first":"Alan","last":"Cooper"},"age":28}
1: {"name":{"first":"Tom","last":"Johnson"},"age":38}
2: {"name":{"first":"Janet","last":"Prichard"},"age":47}
3: {"name":{"first":"Carol","last":"Anderson"},"age":52}

Order by age range 30-50
1: {"name":{"first":"Tom","last":"Johnson"},"age":38}
2: {"name":{"first":"Janet","last":"Prichard"},"age":47}

Multi Value Index

With BuntDB it's possible to join multiple values on a single index. This is similar to a multi column index in a traditional SQL database.

In this example we are creating a multi value index on "name.last" and "age":

db, _ := buntdb.Open(":memory:")
db.CreateIndex("last_name_age", "*", buntdb.IndexJSON("name.last"), buntdb.IndexJSON("age"))
db.Update(func(tx *buntdb.Tx) error {
	tx.Set("1", `{"name":{"first":"Tom","last":"Johnson"},"age":38}`, nil)
	tx.Set("2", `{"name":{"first":"Janet","last":"Prichard"},"age":47}`, nil)
	tx.Set("3", `{"name":{"first":"Carol","last":"Anderson"},"age":52}`, nil)
	tx.Set("4", `{"name":{"first":"Alan","last":"Cooper"},"age":28}`, nil)
	tx.Set("5", `{"name":{"first":"Sam","last":"Anderson"},"age":51}`, nil)
	tx.Set("6", `{"name":{"first":"Melinda","last":"Prichard"},"age":44}`, nil)
	return nil
})
db.View(func(tx *buntdb.Tx) error {
	tx.Ascend("last_name_age", func(key, value string) bool {
		fmt.Printf("%s: %s\n", key, value)
		return true
	})
	return nil
})

// Output:
// 5: {"name":{"first":"Sam","last":"Anderson"},"age":51}
// 3: {"name":{"first":"Carol","last":"Anderson"},"age":52}
// 4: {"name":{"first":"Alan","last":"Cooper"},"age":28}
// 1: {"name":{"first":"Tom","last":"Johnson"},"age":38}
// 6: {"name":{"first":"Melinda","last":"Prichard"},"age":44}
// 2: {"name":{"first":"Janet","last":"Prichard"},"age":47}

Descending Ordered Index

Any index can be put in descending order by wrapping it's less function with buntdb.Desc.

db.CreateIndex("last_name_age", "*",
    buntdb.IndexJSON("name.last"),
    buntdb.Desc(buntdb.IndexJSON("age")),
)

This will create a multi value index where the last name is ascending and the age is descending.

Collate i18n Indexes

Using the external collate package it's possible to create indexes that are sorted by the specified language. This is similar to the SQL COLLATE keyword found in traditional databases.

To install:

go get -u github.com/tidwall/collate

For example:

import "github.com/tidwall/collate"

// To sort case-insensitive in French.
db.CreateIndex("name", "*", collate.IndexString("FRENCH_CI"))

// To specify that numbers should sort numerically ("2" < "12")
// and use a comma to represent a decimal point.
db.CreateIndex("amount", "*", collate.IndexString("FRENCH_NUM"))

There's also support for Collation on JSON indexes:

db.CreateIndex("last_name", "*", collate.IndexJSON("CHINESE_CI", "name.last"))

Check out the collate project for more information.

Data Expiration

Items can be automatically evicted by using the SetOptions object in the Set function to set a TTL.

db.Update(func(tx *buntdb.Tx) error {
	tx.Set("mykey", "myval", &buntdb.SetOptions{Expires:true, TTL:time.Second})
	return nil
})

Now mykey will automatically be deleted after one second. You can remove the TTL by setting the value again with the same key/value, but with the options parameter set to nil.

Delete while iterating

BuntDB does not currently support deleting a key while in the process of iterating. As a workaround you'll need to delete keys following the completion of the iterator.

var delkeys []string
tx.AscendKeys("object:*", func(k, v string) bool {
	if someCondition(k) == true {
		delkeys = append(delkeys, k)
	}
	return true // continue
})
for _, k := range delkeys {
	if _, err = tx.Delete(k); err != nil {
		return err
	}
}

Append-only File

BuntDB uses an AOF (append-only file) which is a log of all database changes that occur from operations like Set() and Delete().

The format of this file looks like:

set key:1 value1
set key:2 value2
set key:1 value3
del key:2
...

When the database opens again, it will read back the aof file and process each command in exact order. This read process happens one time when the database opens. From there on the file is only appended.

As you may guess this log file can grow large over time. There's a background routine that automatically shrinks the log file when it gets too large. There is also a Shrink() function which will rewrite the aof file so that it contains only the items in the database. The shrink operation does not lock up the database so read and write transactions can continue while shrinking is in process.

Durability and fsync

By default BuntDB executes an fsync once every second on the aof file. Which simply means that there's a chance that up to one second of data might be lost. If you need higher durability then there's an optional database config setting Config.SyncPolicy which can be set to Always.

The Config.SyncPolicy has the following options:

  • Never - fsync is managed by the operating system, less safe
  • EverySecond - fsync every second, fast and safer, this is the default
  • Always - fsync after every write, very durable, slower

Config

Here are some configuration options that can be use to change various behaviors of the database.

  • SyncPolicy adjusts how often the data is synced to disk. This value can be Never, EverySecond, or Always. Default is EverySecond.
  • AutoShrinkPercentage is used by the background process to trigger a shrink of the aof file when the size of the file is larger than the percentage of the result of the previous shrunk file. For example, if this value is 100, and the last shrink process resulted in a 100mb file, then the new aof file must be 200mb before a shrink is triggered. Default is 100.
  • AutoShrinkMinSize defines the minimum size of the aof file before an automatic shrink can occur. Default is 32MB.
  • AutoShrinkDisabled turns off automatic background shrinking. Default is false.

To update the configuration you should call ReadConfig followed by SetConfig. For example:


var config buntdb.Config
if err := db.ReadConfig(&config); err != nil{
	log.Fatal(err)
}
if err := db.SetConfig(config); err != nil{
	log.Fatal(err)
}

Performance

How fast is BuntDB?

Here are some example benchmarks when using BuntDB in a Raft Store implementation.

You can also run the standard Go benchmark tool from the project root directory:

go test --bench=.
BuntDB-Benchmark

There's a custom utility that was created specifically for benchmarking BuntDB.

These are the results from running the benchmarks on a MacBook Pro 15" 2.8 GHz Intel Core i7:

$ buntdb-benchmark -q
GET: 4609604.74 operations per second
SET: 248500.33 operations per second
ASCEND_100: 2268998.79 operations per second
ASCEND_200: 1178388.14 operations per second
ASCEND_400: 679134.20 operations per second
ASCEND_800: 348445.55 operations per second
DESCEND_100: 2313821.69 operations per second
DESCEND_200: 1292738.38 operations per second
DESCEND_400: 675258.76 operations per second
DESCEND_800: 337481.67 operations per second
SPATIAL_SET: 134824.60 operations per second
SPATIAL_INTERSECTS_100: 939491.47 operations per second
SPATIAL_INTERSECTS_200: 561590.40 operations per second
SPATIAL_INTERSECTS_400: 306951.15 operations per second
SPATIAL_INTERSECTS_800: 159673.91 operations per second

To install this utility:

go get github.com/tidwall/buntdb-benchmark

Contact

Josh Baker @tidwall

License

BuntDB source code is available under the MIT License.

Documentation

Overview

Package buntdb implements a low-level in-memory key/value store in pure Go. It persists to disk, is ACID compliant, and uses locking for multiple readers and a single writer. Bunt is ideal for projects that need a dependable database, and favor speed over data size.

Example (DescKeys) 
db, _ := Open(":memory:")
db.CreateIndex("name", "*", IndexString)
db.Update(func(tx *Tx) error {
	tx.Set("user:100:first", "Tom", nil)
	tx.Set("user:100:last", "Johnson", nil)
	tx.Set("user:101:first", "Janet", nil)
	tx.Set("user:101:last", "Prichard", nil)
	tx.Set("user:102:first", "Alan", nil)
	tx.Set("user:102:last", "Cooper", nil)
	return nil
})
db.View(func(tx *Tx) error {
	tx.AscendKeys("user:101:*",
		func(key, value string) bool {
			fmt.Printf("%s: %s\n", key, value)
			return true
		})
	tx.AscendKeys("user:10?:*",
		func(key, value string) bool {
			fmt.Printf("%s: %s\n", key, value)
			return true
		})
	tx.AscendKeys("*2*",
		func(key, value string) bool {
			fmt.Printf("%s: %s\n", key, value)
			return true
		})
	tx.DescendKeys("user:101:*",
		func(key, value string) bool {
			fmt.Printf("%s: %s\n", key, value)
			return true
		})
	tx.DescendKeys("*",
		func(key, value string) bool {
			fmt.Printf("%s: %s\n", key, value)
			return true
		})
	return nil
})

Output:

user:101:first: Janet
user:101:last: Prichard
user:100:first: Tom
user:100:last: Johnson
user:101:first: Janet
user:101:last: Prichard
user:102:first: Alan
user:102:last: Cooper
user:102:first: Alan
user:102:last: Cooper
user:101:last: Prichard
user:101:first: Janet
user:102:last: Cooper
user:102:first: Alan
user:101:last: Prichard
user:101:first: Janet
user:100:last: Johnson
user:100:first: Tom

Index

Examples

Constants

View Source 
const (
	// Never is used to disable syncing data to disk.
	// The faster and less safe method.
	Never SyncPolicy = 0
	// EverySecond is used to sync data to disk every second.
	// It's pretty fast and you can lose 1 second of data if there
	// is a disaster.
	// This is the recommended setting.
	EverySecond = 1
	// Always is used to sync data after every write to disk.
	// Slow. Very safe.
	Always = 2
)

Variables

View Source 
var (
	// ErrTxNotWritable is returned when performing a write operation on a
	// read-only transaction.
	ErrTxNotWritable = errors.New("tx not writable")

	// ErrTxClosed is returned when committing or rolling back a transaction
	// that has already been committed or rolled back.
	ErrTxClosed = errors.New("tx closed")

	// ErrNotFound is returned when an item or index is not in the database.
	ErrNotFound = errors.New("not found")

	// ErrInvalid is returned when the database file is an invalid format.
	ErrInvalid = errors.New("invalid database")

	// ErrDatabaseClosed is returned when the database is closed.
	ErrDatabaseClosed = errors.New("database closed")

	// ErrIndexExists is returned when an index already exists in the database.
	ErrIndexExists = errors.New("index exists")

	// ErrInvalidOperation is returned when an operation cannot be completed.
	ErrInvalidOperation = errors.New("invalid operation")

	// ErrInvalidSyncPolicy is returned for an invalid SyncPolicy value.
	ErrInvalidSyncPolicy = errors.New("invalid sync policy")

	// ErrShrinkInProcess is returned when a shrink operation is in-process.
	ErrShrinkInProcess = errors.New("shrink is in-process")

	// ErrPersistenceActive is returned when post-loading data from an database
	// not opened with Open(":memory:").
	ErrPersistenceActive = errors.New("persistence active")

	// ErrTxIterating is returned when Set or Delete are called while iterating.
	ErrTxIterating = errors.New("tx is iterating")
)

Functions

func Desc 

func Desc(less func(a, b string) bool) func(a, b string) bool

Desc is a helper function that changes the order of an index.

Example 
db, _ := Open(":memory:")
db.CreateIndex("last_name_age", "*", IndexJSON("name.last"), Desc(IndexJSON("age")))
db.Update(func(tx *Tx) error {
	tx.Set("1", `{"name":{"first":"Tom","last":"Johnson"},"age":38}`, nil)
	tx.Set("2", `{"name":{"first":"Janet","last":"Prichard"},"age":47}`, nil)
	tx.Set("3", `{"name":{"first":"Carol","last":"Anderson"},"age":52}`, nil)
	tx.Set("4", `{"name":{"first":"Alan","last":"Cooper"},"age":28}`, nil)
	tx.Set("5", `{"name":{"first":"Sam","last":"Anderson"},"age":51}`, nil)
	tx.Set("6", `{"name":{"first":"Melinda","last":"Prichard"},"age":44}`, nil)
	return nil
})
db.View(func(tx *Tx) error {
	tx.Ascend("last_name_age", func(key, value string) bool {
		fmt.Printf("%s: %s\n", key, value)
		return true
	})
	return nil
})

Output:

3: {"name":{"first":"Carol","last":"Anderson"},"age":52}
5: {"name":{"first":"Sam","last":"Anderson"},"age":51}
4: {"name":{"first":"Alan","last":"Cooper"},"age":28}
1: {"name":{"first":"Tom","last":"Johnson"},"age":38}
2: {"name":{"first":"Janet","last":"Prichard"},"age":47}
6: {"name":{"first":"Melinda","last":"Prichard"},"age":44}

func IndexBinary 

func IndexBinary(a, b string) bool

IndexBinary is a helper function that returns true if 'a' is less than 'b'. This compares the raw binary of the string.

func IndexFloat 

func IndexFloat(a, b string) bool

IndexFloat is a helper function that returns true if 'a' is less than 'b'. This compares float64s that are added to the database using the Float() conversion function.

func IndexInt 

func IndexInt(a, b string) bool

IndexInt is a helper function that returns true if 'a' is less than 'b'.

func IndexJSON 

func IndexJSON(path string) func(a, b string) bool

IndexJSON provides for the ability to create an index on any JSON field. When the field is a string, the comparison will be case-insensitive. It returns a helper function used by CreateIndex.

func IndexJSONCaseSensitive 

func IndexJSONCaseSensitive(path string) func(a, b string) bool

IndexJSONCaseSensitive provides for the ability to create an index on any JSON field. When the field is a string, the comparison will be case-sensitive. It returns a helper function used by CreateIndex.

func IndexRect 

func IndexRect(a string) (min, max []float64)

IndexRect is a helper function that converts string to a rect. Rect() is the reverse function and can be used to generate a string from a rect.

func IndexString 

func IndexString(a, b string) bool

IndexString is a helper function that return true if 'a' is less than 'b'. This is a case-insensitive comparison. Use the IndexBinary() for comparing case-sensitive strings.

func IndexUint 

func IndexUint(a, b string) bool

IndexUint is a helper function that returns true if 'a' is less than 'b'. This compares uint64s that are added to the database using the Uint() conversion function.

func Match 

func Match(key, pattern string) bool

Match returns true if the specified key matches the pattern. This is a very simple pattern matcher where '*' matches on any number characters and '?' matches on any one character.

func Point 

func Point(coords ...float64) string

Point is a helper function that converts a series of float64s to a rectangle for a spatial index.

func Rect 

func Rect(min, max []float64) string

Rect is helper function that returns a string representation of a rect. IndexRect() is the reverse function and can be used to generate a rect from a string.

Types

type Config 

type Config struct {
	// SyncPolicy adjusts how often the data is synced to disk.
	// This value can be Never, EverySecond, or Always.
	// The default is EverySecond.
	SyncPolicy SyncPolicy

	// AutoShrinkPercentage is used by the background process to trigger
	// a shrink of the aof file when the size of the file is larger than the
	// percentage of the result of the previous shrunk file.
	// For example, if this value is 100, and the last shrink process
	// resulted in a 100mb file, then the new aof file must be 200mb before
	// a shrink is triggered.
	AutoShrinkPercentage int

	// AutoShrinkMinSize defines the minimum size of the aof file before
	// an automatic shrink can occur.
	AutoShrinkMinSize int

	// AutoShrinkDisabled turns off automatic background shrinking
	AutoShrinkDisabled bool

	// OnExpired is used to custom handle the deletion option when a key
	// has been expired.
	OnExpired func(keys []string)

	// OnExpiredSync will be called inside the same transaction that is
	// performing the deletion of expired items. If OnExpired is present then
	// this callback will not be called. If this callback is present, then the
	// deletion of the timeed-out item is the explicit responsibility of this
	// callback.
	OnExpiredSync func(key, value string, tx *Tx) error
}

Config represents database configuration options. These options are used to change various behaviors of the database.

type DB 

type DB struct {
	// contains filtered or unexported fields
}

DB represents a collection of key-value pairs that persist on disk. Transactions are used for all forms of data access to the DB.

func Open 

func Open(path string) (*DB, error)

Open opens a database at the provided path. If the file does not exist then it will be created automatically.

func (*DB) Begin 

func (db *DB) Begin(writable bool) (*Tx, error)

Begin opens a new transaction. Multiple read-only transactions can be opened at the same time but there can only be one read/write transaction at a time. Attempting to open a read/write transactions while another one is in progress will result in blocking until the current read/write transaction is completed.

All transactions must be closed by calling Commit() or Rollback() when done.

func (*DB) Close 

func (db *DB) Close() error

Close releases all database resources. All transactions must be closed before closing the database.

func (*DB) CreateIndex 

func (db *DB) CreateIndex(name, pattern string,
	less ...func(a, b string) bool) error

CreateIndex builds a new index and populates it with items. The items are ordered in an b-tree and can be retrieved using the Ascend* and Descend* methods. An error will occur if an index with the same name already exists.

When a pattern is provided, the index will be populated with keys that match the specified pattern. This is a very simple pattern match where '*' matches on any number characters and '?' matches on any one character. The less function compares if string 'a' is less than string 'b'. It allows for indexes to create custom ordering. It's possible that the strings may be textual or binary. It's up to the provided less function to handle the content format and comparison. There are some default less function that can be used such as IndexString, IndexBinary, etc.

Example (Ints) 
db, _ := Open(":memory:")
db.CreateIndex("age", "*", IndexInt)
db.Update(func(tx *Tx) error {
	tx.Set("1", "30", nil)
	tx.Set("2", "51", nil)
	tx.Set("3", "16", nil)
	tx.Set("4", "76", nil)
	tx.Set("5", "23", nil)
	tx.Set("6", "43", nil)
	return nil
})
db.View(func(tx *Tx) error {
	tx.Ascend("age", func(key, value string) bool {
		fmt.Printf("%s: %s\n", key, value)
		return true
	})
	return nil
})

Output:

3: 16
5: 23
1: 30
6: 43
2: 51
4: 76
Example (JSON) 
db, _ := Open(":memory:")
db.CreateIndex("last_name", "*", IndexJSON("name.last"))
db.CreateIndex("age", "*", IndexJSON("age"))
db.Update(func(tx *Tx) error {
	tx.Set("1", `{"name":{"first":"Tom","last":"Johnson"},"age":38}`, nil)
	tx.Set("2", `{"name":{"first":"Janet","last":"Prichard"},"age":47}`, nil)
	tx.Set("3", `{"name":{"first":"Carol","last":"Anderson"},"age":52}`, nil)
	tx.Set("4", `{"name":{"first":"Alan","last":"Cooper"},"age":28}`, nil)
	return nil
})
db.View(func(tx *Tx) error {
	fmt.Println("Order by last name")
	tx.Ascend("last_name", func(key, value string) bool {
		fmt.Printf("%s: %s\n", key, value)
		return true
	})
	fmt.Println("Order by age")
	tx.Ascend("age", func(key, value string) bool {
		fmt.Printf("%s: %s\n", key, value)
		return true
	})
	fmt.Println("Order by age range 30-50")
	tx.AscendRange("age", `{"age":30}`, `{"age":50}`, func(key, value string) bool {
		fmt.Printf("%s: %s\n", key, value)
		return true
	})
	return nil
})

Output:

Order by last name
3: {"name":{"first":"Carol","last":"Anderson"},"age":52}
4: {"name":{"first":"Alan","last":"Cooper"},"age":28}
1: {"name":{"first":"Tom","last":"Johnson"},"age":38}
2: {"name":{"first":"Janet","last":"Prichard"},"age":47}
Order by age
4: {"name":{"first":"Alan","last":"Cooper"},"age":28}
1: {"name":{"first":"Tom","last":"Johnson"},"age":38}
2: {"name":{"first":"Janet","last":"Prichard"},"age":47}
3: {"name":{"first":"Carol","last":"Anderson"},"age":52}
Order by age range 30-50
1: {"name":{"first":"Tom","last":"Johnson"},"age":38}
2: {"name":{"first":"Janet","last":"Prichard"},"age":47}
Example (MultipleFields) 
db, _ := Open(":memory:")
db.CreateIndex("last_name_age", "*", IndexJSON("name.last"), IndexJSON("age"))
db.Update(func(tx *Tx) error {
	tx.Set("1", `{"name":{"first":"Tom","last":"Johnson"},"age":38}`, nil)
	tx.Set("2", `{"name":{"first":"Janet","last":"Prichard"},"age":47}`, nil)
	tx.Set("3", `{"name":{"first":"Carol","last":"Anderson"},"age":52}`, nil)
	tx.Set("4", `{"name":{"first":"Alan","last":"Cooper"},"age":28}`, nil)
	tx.Set("5", `{"name":{"first":"Sam","last":"Anderson"},"age":51}`, nil)
	tx.Set("6", `{"name":{"first":"Melinda","last":"Prichard"},"age":44}`, nil)
	return nil
})
db.View(func(tx *Tx) error {
	tx.Ascend("last_name_age", func(key, value string) bool {
		fmt.Printf("%s: %s\n", key, value)
		return true
	})
	return nil
})

Output:

5: {"name":{"first":"Sam","last":"Anderson"},"age":51}
3: {"name":{"first":"Carol","last":"Anderson"},"age":52}
4: {"name":{"first":"Alan","last":"Cooper"},"age":28}
1: {"name":{"first":"Tom","last":"Johnson"},"age":38}
6: {"name":{"first":"Melinda","last":"Prichard"},"age":44}
2: {"name":{"first":"Janet","last":"Prichard"},"age":47}
Example (Strings) 
db, _ := Open(":memory:")
db.CreateIndex("name", "*", IndexString)
db.Update(func(tx *Tx) error {
	tx.Set("1", "Tom", nil)
	tx.Set("2", "Janet", nil)
	tx.Set("3", "Carol", nil)
	tx.Set("4", "Alan", nil)
	tx.Set("5", "Sam", nil)
	tx.Set("6", "Melinda", nil)
	return nil
})
db.View(func(tx *Tx) error {
	tx.Ascend("name", func(key, value string) bool {
		fmt.Printf("%s: %s\n", key, value)
		return true
	})
	return nil
})

Output:

4: Alan
3: Carol
2: Janet
6: Melinda
5: Sam
1: Tom

func (*DB) CreateSpatialIndex 

func (db *DB) CreateSpatialIndex(name, pattern string,
	rect func(item string) (min, max []float64)) error

CreateSpatialIndex builds a new index and populates it with items. The items are organized in an r-tree and can be retrieved using the Intersects method. An error will occur if an index with the same name already exists.

The rect function converts a string to a rectangle. The rectangle is represented by two arrays, min and max. Both arrays may have a length between 1 and 20, and both arrays must match in length. A length of 1 is a one dimensional rectangle, and a length of 4 is a four dimension rectangle. There is support for up to 20 dimensions. The values of min must be less than the values of max at the same dimension. Thus min[0] must be less-than-or-equal-to max[0]. The IndexRect is a default function that can be used for the rect parameter.

func (*DB) DropIndex 

func (db *DB) DropIndex(name string) error

DropIndex removes an index.

func (*DB) Indexes 

func (db *DB) Indexes() ([]string, error)

Indexes returns a list of index names.

func (*DB) Load 

func (db *DB) Load(rd io.Reader) error

Load loads commands from reader. This operation blocks all reads and writes. Note that this can only work for fully in-memory databases opened with Open(":memory:").

func (*DB) ReadConfig 

func (db *DB) ReadConfig(config *Config) error

ReadConfig returns the database configuration.

func (*DB) ReplaceIndex 

func (db *DB) ReplaceIndex(name, pattern string,
	less ...func(a, b string) bool) error

ReplaceIndex builds a new index and populates it with items. The items are ordered in an b-tree and can be retrieved using the Ascend* and Descend* methods. If a previous index with the same name exists, that index will be deleted.

func (*DB) ReplaceSpatialIndex 

func (db *DB) ReplaceSpatialIndex(name, pattern string,
	rect func(item string) (min, max []float64)) error

ReplaceSpatialIndex builds a new index and populates it with items. The items are organized in an r-tree and can be retrieved using the Intersects method. If a previous index with the same name exists, that index will be deleted.

func (*DB) Save 

func (db *DB) Save(wr io.Writer) error

Save writes a snapshot of the database to a writer. This operation blocks all writes, but not reads. This can be used for snapshots and backups for pure in-memory databases using the ":memory:". Database that persist to disk can be snapshotted by simply copying the database file.

func (*DB) SetConfig 

func (db *DB) SetConfig(config Config) error

SetConfig updates the database configuration.

func (*DB) Shrink 

func (db *DB) Shrink() error

Shrink will make the database file smaller by removing redundant log entries. This operation does not block the database.

func (*DB) Update 

func (db *DB) Update(fn func(tx *Tx) error) error

Update executes a function within a managed read/write transaction. The transaction has been committed when no error is returned. In the event that an error is returned, the transaction will be rolled back. When a non-nil error is returned from the function, the transaction will be rolled back and the that error will be return to the caller of Update().

Executing a manual commit or rollback from inside the function will result in a panic.

func (*DB) View 

func (db *DB) View(fn func(tx *Tx) error) error

View executes a function within a managed read-only transaction. When a non-nil error is returned from the function that error will be return to the caller of View().

Executing a manual commit or rollback from inside the function will result in a panic.

type IndexOptions 

type IndexOptions struct {
	// CaseInsensitiveKeyMatching allow for case-insensitive
	// matching on keys when setting key/values.
	CaseInsensitiveKeyMatching bool
}

IndexOptions provides an index with additional features or alternate functionality.

type SetOptions 

type SetOptions struct {
	// Expires indicates that the Set() key-value will expire
	Expires bool
	// for keys that would not match any indexes for sure
	NoCustomIndexCheck bool
	// TTL is how much time the key-value will exist in the database
	// before being evicted. The Expires field must also be set to true.
	// TTL stands for Time-To-Live.
	TTL time.Duration
}

SetOptions represents options that may be included with the Set() command.

type SyncPolicy 

type SyncPolicy int

SyncPolicy represents how often data is synced to disk.

type Tx 

type Tx struct {
	// contains filtered or unexported fields
}

Tx represents a transaction on the database. This transaction can either be read-only or read/write. Read-only transactions can be used for retrieving values for keys and iterating through keys and values. Read/write transactions can set and delete keys.

All transactions must be committed or rolled-back when done.

func (*Tx) Ascend 

func (tx *Tx) Ascend(index string,
	iterator func(key, value string) bool) error

Ascend calls the iterator for every item in the database within the range [first, last], until iterator returns false. When an index is provided, the results will be ordered by the item values as specified by the less() function of the defined index. When an index is not provided, the results will be ordered by the item key. An invalid index will return an error.

func (*Tx) AscendEqual 

func (tx *Tx) AscendEqual(index, pivot string,
	iterator func(key, value string) bool) error

AscendEqual calls the iterator for every item in the database that equals pivot, until iterator returns false. When an index is provided, the results will be ordered by the item values as specified by the less() function of the defined index. When an index is not provided, the results will be ordered by the item key. An invalid index will return an error.

func (*Tx) AscendGreaterOrEqual 

func (tx *Tx) AscendGreaterOrEqual(index, pivot string,
	iterator func(key, value string) bool) error

AscendGreaterOrEqual calls the iterator for every item in the database within the range [pivot, last], until iterator returns false. When an index is provided, the results will be ordered by the item values as specified by the less() function of the defined index. When an index is not provided, the results will be ordered by the item key. An invalid index will return an error.

func (*Tx) AscendKeys 

func (tx *Tx) AscendKeys(pattern string,
	iterator func(key, value string) bool) error

AscendKeys allows for iterating through keys based on the specified pattern.

func (*Tx) AscendLessThan 

func (tx *Tx) AscendLessThan(index, pivot string,
	iterator func(key, value string) bool) error

AscendLessThan calls the iterator for every item in the database within the range [first, pivot), until iterator returns false. When an index is provided, the results will be ordered by the item values as specified by the less() function of the defined index. When an index is not provided, the results will be ordered by the item key. An invalid index will return an error.

func (*Tx) AscendRange 

func (tx *Tx) AscendRange(index, greaterOrEqual, lessThan string,
	iterator func(key, value string) bool) error

AscendRange calls the iterator for every item in the database within the range [greaterOrEqual, lessThan), until iterator returns false. When an index is provided, the results will be ordered by the item values as specified by the less() function of the defined index. When an index is not provided, the results will be ordered by the item key. An invalid index will return an error.

func (*Tx) Commit 

func (tx *Tx) Commit() error

Commit writes all changes to disk. An error is returned when a write error occurs, or when a Commit() is called from a read-only transaction.

func (*Tx) CreateIndex 

func (tx *Tx) CreateIndex(name, pattern string,
	less ...func(a, b string) bool) error

CreateIndex builds a new index and populates it with items. The items are ordered in an b-tree and can be retrieved using the Ascend* and Descend* methods. An error will occur if an index with the same name already exists.

When a pattern is provided, the index will be populated with keys that match the specified pattern. This is a very simple pattern match where '*' matches on any number characters and '?' matches on any one character. The less function compares if string 'a' is less than string 'b'. It allows for indexes to create custom ordering. It's possible that the strings may be textual or binary. It's up to the provided less function to handle the content format and comparison. There are some default less function that can be used such as IndexString, IndexBinary, etc.

func (*Tx) CreateIndexOptions 

func (tx *Tx) CreateIndexOptions(name, pattern string,
	opts *IndexOptions,
	less ...func(a, b string) bool) error

CreateIndexOptions is the same as CreateIndex except that it allows for additional options.

func (*Tx) CreateSpatialIndex 

func (tx *Tx) CreateSpatialIndex(name, pattern string,
	rect func(item string) (min, max []float64)) error

CreateSpatialIndex builds a new index and populates it with items. The items are organized in an r-tree and can be retrieved using the Intersects method. An error will occur if an index with the same name already exists.

The rect function converts a string to a rectangle. The rectangle is represented by two arrays, min and max. Both arrays may have a length between 1 and 20, and both arrays must match in length. A length of 1 is a one dimensional rectangle, and a length of 4 is a four dimension rectangle. There is support for up to 20 dimensions. The values of min must be less than the values of max at the same dimension. Thus min[0] must be less-than-or-equal-to max[0]. The IndexRect is a default function that can be used for the rect parameter.

func (*Tx) CreateSpatialIndexOptions 

func (tx *Tx) CreateSpatialIndexOptions(name, pattern string,
	opts *IndexOptions,
	rect func(item string) (min, max []float64)) error

CreateSpatialIndexOptions is the same as CreateSpatialIndex except that it allows for additional options.

func (*Tx) Delete 

func (tx *Tx) Delete(key string) (val string, err error)

Delete removes an item from the database based on the item's key. If the item does not exist or if the item has expired then ErrNotFound is returned.

Only a writable transaction can be used for this operation. This operation is not allowed during iterations such as Ascend* & Descend*.

func (*Tx) DeleteAll 

func (tx *Tx) DeleteAll() error

DeleteAll deletes all items from the database.

func (*Tx) Descend 

func (tx *Tx) Descend(index string,
	iterator func(key, value string) bool) error

Descend calls the iterator for every item in the database within the range [last, first], until iterator returns false. When an index is provided, the results will be ordered by the item values as specified by the less() function of the defined index. When an index is not provided, the results will be ordered by the item key. An invalid index will return an error.

func (*Tx) DescendEqual 

func (tx *Tx) DescendEqual(index, pivot string,
	iterator func(key, value string) bool) error

DescendEqual calls the iterator for every item in the database that equals pivot, until iterator returns false. When an index is provided, the results will be ordered by the item values as specified by the less() function of the defined index. When an index is not provided, the results will be ordered by the item key. An invalid index will return an error.

func (*Tx) DescendGreaterThan 

func (tx *Tx) DescendGreaterThan(index, pivot string,
	iterator func(key, value string) bool) error

DescendGreaterThan calls the iterator for every item in the database within the range [last, pivot), until iterator returns false. When an index is provided, the results will be ordered by the item values as specified by the less() function of the defined index. When an index is not provided, the results will be ordered by the item key. An invalid index will return an error.

func (*Tx) DescendKeys 

func (tx *Tx) DescendKeys(pattern string,
	iterator func(key, value string) bool) error

DescendKeys allows for iterating through keys based on the specified pattern.

func (*Tx) DescendLessOrEqual 

func (tx *Tx) DescendLessOrEqual(index, pivot string,
	iterator func(key, value string) bool) error

DescendLessOrEqual calls the iterator for every item in the database within the range [pivot, first], until iterator returns false. When an index is provided, the results will be ordered by the item values as specified by the less() function of the defined index. When an index is not provided, the results will be ordered by the item key. An invalid index will return an error.

func (*Tx) DescendRange 

func (tx *Tx) DescendRange(index, lessOrEqual, greaterThan string,
	iterator func(key, value string) bool) error

DescendRange calls the iterator for every item in the database within the range [lessOrEqual, greaterThan), until iterator returns false. When an index is provided, the results will be ordered by the item values as specified by the less() function of the defined index. When an index is not provided, the results will be ordered by the item key. An invalid index will return an error.

func (*Tx) DropIndex 

func (tx *Tx) DropIndex(name string) error

DropIndex removes an index.

func (*Tx) Get 

func (tx *Tx) Get(key string, ignoreExpired ...bool) (val string, err error)

Get returns a value for a key. If the item does not exist or if the item has expired then ErrNotFound is returned. If ignoreExpired is true, then the found value will be returned even if it is expired.

func (*Tx) GetLess 

func (tx *Tx) GetLess(index string) (func(a, b string) bool, error)

GetLess returns the less function for an index. This is handy for doing ad-hoc compares inside a transaction. Returns ErrNotFound if the index is not found or there is no less function bound to the index

func (*Tx) GetRect 

func (tx *Tx) GetRect(index string) (func(s string) (min, max []float64),
	error)

GetRect returns the rect function for an index. This is handy for doing ad-hoc searches inside a transaction. Returns ErrNotFound if the index is not found or there is no rect function bound to the index

func (*Tx) Indexes 

func (tx *Tx) Indexes() ([]string, error)

Indexes returns a list of index names.

func (*Tx) Intersects 

func (tx *Tx) Intersects(index, bounds string,
	iterator func(key, value string) bool) error

Intersects searches for rectangle items that intersect a target rect. The specified index must have been created by AddIndex() and the target is represented by the rect string. This string will be processed by the same bounds function that was passed to the CreateSpatialIndex() function. An invalid index will return an error.

func (*Tx) Len 

func (tx *Tx) Len() (int, error)

Len returns the number of items in the database

func (*Tx) Nearby 

func (tx *Tx) Nearby(index, bounds string,
	iterator func(key, value string, dist float64) bool) error

Nearby searches for rectangle items that are nearby a target rect. All items belonging to the specified index will be returned in order of nearest to farthest. The specified index must have been created by AddIndex() and the target is represented by the rect string. This string will be processed by the same bounds function that was passed to the CreateSpatialIndex() function. An invalid index will return an error. The dist param is the distance of the bounding boxes. In the case of simple 2D points, it's the distance of the two 2D points squared.

func (*Tx) Rollback 

func (tx *Tx) Rollback() error

Rollback closes the transaction and reverts all mutable operations that were performed on the transaction such as Set() and Delete().

Read-only transactions can only be rolled back, not committed.

func (*Tx) Set 

func (tx *Tx) Set(key, value string, opts *SetOptions) (previousValue string,
	replaced bool, err error)

Set inserts or replaces an item in the database based on the key. The opt params may be used for additional functionality such as forcing the item to be evicted at a specified time. When the return value for err is nil the operation succeeded. When the return value of replaced is true, then the operaton replaced an existing item whose value will be returned through the previousValue variable. The results of this operation will not be available to other transactions until the current transaction has successfully committed.

Only a writable transaction can be used with this operation. This operation is not allowed during iterations such as Ascend* & Descend*.

func (*Tx) TTL 

func (tx *Tx) TTL(key string) (time.Duration, error)

TTL returns the remaining time-to-live for an item. A negative duration will be returned for items that do not have an expiration.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.