Documentation ¶
Overview ¶
Package buckets provides a simplified interface to a Bolt database.
A buckets DB is a Bolt database, but it allows you to easily create new bucket instances. The database is represented by a single file on disk. A bucket is a collection of unique keys that are associated with values.
The Bucket type has nifty convenience methods for operating on key/value pairs within it. It streamlines simple transactions (a single put, get, or delete) and working with subsets of items within a bucket (via prefix and range scans). It's not designed to handle more complex or batch transactions. For such cases, use the standard techniques offered by Bolt.
---
What is bolt?
"Bolt implements a low-level key/value store in pure Go. It supports fully serializable transactions, ACID semantics, and lock-free MVCC with multiple readers and a single writer. Bolt can be used for projects that want a simple data store without the need to add large dependencies such as Postgres or MySQL."
See https://github.com/boltdb/bolt for important details.
Index ¶
- type Bucket
- func (bk *Bucket) Delete(k []byte) error
- func (bk *Bucket) Get(k []byte) (value []byte, err error)
- func (bk *Bucket) Insert(items []struct{ ... }) error
- func (bk *Bucket) InsertNX(items []struct{ ... }) error
- func (bk *Bucket) Items() (items []Item, err error)
- func (bk *Bucket) Map(do func(k, v []byte) error) error
- func (bk *Bucket) MapPrefix(do func(k, v []byte) error, pre []byte) error
- func (bk *Bucket) MapRange(do func(k, v []byte) error, min, max []byte) error
- func (bk *Bucket) NewPrefixScanner(pre []byte) *PrefixScanner
- func (bk *Bucket) NewRangeScanner(min, max []byte) *RangeScanner
- func (bk *Bucket) PrefixItems(pre []byte) (items []Item, err error)
- func (bk *Bucket) Put(k, v []byte) error
- func (bk *Bucket) PutNX(k, v []byte) error
- func (bk *Bucket) RangeItems(min []byte, max []byte) (items []Item, err error)
- type DB
- type Item
- type PrefixScanner
- func (ps *PrefixScanner) Count() (count int, err error)
- func (ps *PrefixScanner) ItemMapping() (map[string][]byte, error)
- func (ps *PrefixScanner) Items() (items []Item, err error)
- func (ps *PrefixScanner) Keys() (keys [][]byte, err error)
- func (ps *PrefixScanner) Map(do func(k, v []byte) error) error
- func (ps *PrefixScanner) Values() (values [][]byte, err error)
- type RangeScanner
- func (rs *RangeScanner) Count() (count int, err error)
- func (rs *RangeScanner) ItemMapping() (map[string][]byte, error)
- func (rs *RangeScanner) Items() (items []Item, err error)
- func (rs *RangeScanner) Keys() (keys [][]byte, err error)
- func (rs *RangeScanner) Map(do func(k, v []byte) error) error
- func (rs *RangeScanner) Values() (values [][]byte, err error)
- type Scanner
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Bucket ¶
type Bucket struct { Name []byte // contains filtered or unexported fields }
Bucket represents a collection of key/value pairs inside the database.
Example ¶
package main import ( "encoding/binary" "fmt" "io/ioutil" "log" "math/rand" "net/http" "net/http/httptest" "os" "github.com/joyrexus/buckets" ) // Set this to see how the counts are actually updated. const verbose = false // Counter updates a the hits bucket for every URL path requested. type counter struct { hits *buckets.Bucket } // Our handler communicates the new count from a successful database // transaction. func (c counter) ServeHTTP(rw http.ResponseWriter, req *http.Request) { key := []byte(req.URL.String()) // Decode handles key not found for us. value, _ := c.hits.Get(key) count := decode(value) + 1 if err := c.hits.Put(key, encode(count)); err != nil { http.Error(rw, err.Error(), 500) return } if verbose { log.Printf("server: %s: %d", req.URL.String(), count) } // Reply with the new count . rw.Header().Set("Content-Type", "application/octet-stream") fmt.Fprintf(rw, "%d\n", count) } func client(id int, base string, paths []string) error { // Process paths in random order. rng := rand.New(rand.NewSource(int64(id))) permutation := rng.Perm(len(paths)) for i := range paths { path := paths[permutation[i]] resp, err := http.Get(base + path) if err != nil { return err } defer resp.Body.Close() buf, err := ioutil.ReadAll(resp.Body) if err != nil { return err } if verbose { log.Printf("client: %s: %s", path, buf) } } return nil } func main() { // Open the database. bx, _ := buckets.Open(tempfile()) defer os.Remove(bx.Path()) defer bx.Close() // Create a hits bucket hits, _ := bx.New([]byte("hits")) // Start our web server count := counter{hits} srv := httptest.NewServer(count) defer srv.Close() // Get every path multiple times. paths := []string{ "/foo", "/bar", "/baz", "/quux", "/thud", "/xyzzy", } for id := 0; id < 10; id++ { if err := client(id, srv.URL, paths); err != nil { fmt.Printf("client error: %v", err) } } // Check the final result do := func(k, v []byte) error { fmt.Printf("hits to %s: %d\n", k, decode(v)) return nil } hits.Map(do) // outputs ... // hits to /bar: 10 // hits to /baz: 10 // hits to /foo: 10 // hits to /quux: 10 // hits to /thud: 10 // hits to /xyzzy: 10 } // encode marshals a counter. func encode(n uint64) []byte { buf := make([]byte, 8) binary.BigEndian.PutUint64(buf, n) return buf } // decode unmarshals a counter. Nil buffers are decoded as 0. func decode(buf []byte) uint64 { if buf == nil { return 0 } return binary.BigEndian.Uint64(buf) }
Output: hits to /bar: 10 hits to /baz: 10 hits to /foo: 10 hits to /quux: 10 hits to /thud: 10 hits to /xyzzy: 10
func (*Bucket) Insert ¶
Insert iterates over a slice of k/v pairs, putting each item in the bucket as part of a single transaction. For large insertions, be sure to pre-sort your items (by Key in byte-sorted order), which will result in much more efficient insertion times and storage costs.
Example ¶
Show we can insert items into a bucket and get them back out.
bx, _ := buckets.Open(tempfile()) defer os.Remove(bx.Path()) defer bx.Close() letters, _ := bx.New([]byte("letters")) // Setup items to insert in `letters` bucket. items := []struct { Key, Value []byte }{ {[]byte("A"), []byte("alpha")}, {[]byte("B"), []byte("beta")}, {[]byte("C"), []byte("gamma")}, } // Insert items into `letters` bucket. if err := letters.Insert(items); err != nil { fmt.Println("could not insert items!") } // Get items back out in separate read-only transaction. results, _ := letters.Items() for _, item := range results { fmt.Printf("%s -> %s\n", item.Key, item.Value) }
Output: A -> alpha B -> beta C -> gamma
func (*Bucket) InsertNX ¶
InsertNX (insert-if-not-exists) iterates over a slice of k/v pairs, putting each item in the bucket as part of a single transaction. Unlike Insert, however, InsertNX will not update the value for an existing key.
func (*Bucket) Items ¶
Items returns a slice of key/value pairs. Each k/v pair in the slice is of type Item (`struct{ Key, Value []byte }`).
func (*Bucket) MapPrefix ¶
MapPrefix applies `do` on each k/v pair of keys with prefix.
Example ¶
Show that we can apply a function to the k/v pairs of keys with a given prefix.
bx, _ := buckets.Open(tempfile()) defer os.Remove(bx.Path()) defer bx.Close() // Create a new things bucket. things, _ := bx.New([]byte("things")) // Setup items to insert. items := []struct { Key, Value []byte }{ {[]byte("A"), []byte("1")}, // `A` prefix match {[]byte("AA"), []byte("2")}, // match {[]byte("AAA"), []byte("3")}, // match {[]byte("AAB"), []byte("2")}, // match {[]byte("B"), []byte("O")}, {[]byte("BA"), []byte("0")}, {[]byte("BAA"), []byte("0")}, } // Insert 'em. if err := things.Insert(items); err != nil { fmt.Printf("could not insert items in `things` bucket: %v\n", err) } // Now collect each item whose key starts with "A". prefix := []byte("A") // Setup slice of items. type item struct { Key, Value []byte } results := []item{} // Anon func to map over matched keys. do := func(k, v []byte) error { results = append(results, item{k, v}) return nil } if err := things.MapPrefix(do, prefix); err != nil { fmt.Printf("could not map items with prefix %s: %v\n", prefix, err) } for _, item := range results { fmt.Printf("%s -> %s\n", item.Key, item.Value) }
Output: A -> 1 AA -> 2 AAA -> 3 AAB -> 2
func (*Bucket) MapRange ¶
MapRange applies `do` on each k/v pair of keys within range.
Example ¶
Show that we can apply a function to the k/v pairs of keys within a given range.
bx, _ := buckets.Open(tempfile()) defer os.Remove(bx.Path()) defer bx.Close() // Delete any existing bucket named "years". bx.Delete([]byte("years")) // Create a new bucket named "years". years, _ := bx.New([]byte("years")) // Setup items to insert in `years` bucket items := []struct { Key, Value []byte }{ {[]byte("1970"), []byte("70")}, {[]byte("1975"), []byte("75")}, {[]byte("1980"), []byte("80")}, {[]byte("1985"), []byte("85")}, {[]byte("1990"), []byte("90")}, // min = 1990 {[]byte("1995"), []byte("95")}, // min < 1995 < max {[]byte("2000"), []byte("00")}, // max = 2000 {[]byte("2005"), []byte("05")}, {[]byte("2010"), []byte("10")}, } // Insert 'em. if err := years.Insert(items); err != nil { fmt.Printf("could not insert items in `years` bucket: %v\n", err) } // Time range to map over: 1990 <= key <= 2000. min := []byte("1990") max := []byte("2000") // Setup slice of items to collect results. type item struct { Key, Value []byte } results := []item{} // Anon func to map over matched keys. do := func(k, v []byte) error { results = append(results, item{k, v}) return nil } if err := years.MapRange(do, min, max); err != nil { fmt.Printf("could not map items within range: %v\n", err) } for _, item := range results { fmt.Printf("%s -> %s\n", item.Key, item.Value) }
Output: 1990 -> 90 1995 -> 95 2000 -> 00
func (*Bucket) NewPrefixScanner ¶
func (bk *Bucket) NewPrefixScanner(pre []byte) *PrefixScanner
NewPrefixScanner initializes a new prefix scanner.
func (*Bucket) NewRangeScanner ¶
func (bk *Bucket) NewRangeScanner(min, max []byte) *RangeScanner
NewRangeScanner initializes a new range scanner. It takes a `min` and a `max` key for specifying the range paramaters.
func (*Bucket) PrefixItems ¶
PrefixItems returns a slice of key/value pairs for all keys with a given prefix. Each k/v pair in the slice is of type Item (`struct{ Key, Value []byte }`).
Example ¶
Show that we can get items for all keys with a given prefix.
bx, _ := buckets.Open(tempfile()) defer os.Remove(bx.Path()) defer bx.Close() // Create a new things bucket. things, _ := bx.New([]byte("things")) // Setup items to insert. items := []struct { Key, Value []byte }{ {[]byte("A"), []byte("1")}, // `A` prefix match {[]byte("AA"), []byte("2")}, // match {[]byte("AAA"), []byte("3")}, // match {[]byte("AAB"), []byte("2")}, // match {[]byte("B"), []byte("O")}, {[]byte("BA"), []byte("0")}, {[]byte("BAA"), []byte("0")}, } // Insert 'em. if err := things.Insert(items); err != nil { fmt.Printf("could not insert items in `things` bucket: %v\n", err) } // Now get items whose key starts with "A". prefix := []byte("A") results, err := things.PrefixItems(prefix) if err != nil { fmt.Printf("could not get items with prefix %q: %v\n", prefix, err) } for _, item := range results { fmt.Printf("%s -> %s\n", item.Key, item.Value) }
Output: A -> 1 AA -> 2 AAA -> 3 AAB -> 2
func (*Bucket) Put ¶
Put inserts value `v` with key `k`.
Example ¶
Show we can put an item in a bucket and get it back out.
bx, _ := buckets.Open(tempfile()) defer os.Remove(bx.Path()) defer bx.Close() // Create a new `things` bucket. bucket := []byte("things") things, _ := bx.New(bucket) // Put key/value into the `things` bucket. key, value := []byte("A"), []byte("alpha") if err := things.Put(key, value); err != nil { fmt.Printf("could not insert item: %v", err) } // Read value back in a different read-only transaction. got, _ := things.Get(key) fmt.Printf("The value of %q in `%s` is %q\n", key, bucket, got)
Output: The value of "A" in `things` is "alpha"
func (*Bucket) PutNX ¶
PutNX (put-if-not-exists) inserts value `v` with key `k` if key doesn't exist.
Example ¶
Show we don't overwrite existing values when using PutNX.
bx, _ := buckets.Open(tempfile()) defer os.Remove(bx.Path()) defer bx.Close() // Create a new `things` bucket. bucket := []byte("things") things, _ := bx.New(bucket) // Put key/value into the `things` bucket. key, value := []byte("A"), []byte("alpha") if err := things.Put(key, value); err != nil { fmt.Printf("could not insert item: %v", err) } // Read value back in a different read-only transaction. got, _ := things.Get(key) fmt.Printf("The value of %q in `%s` is %q\n", key, bucket, got) // Try putting another value with same key. things.PutNX(key, []byte("beta")) // Read value back in a different read-only transaction. got, _ = things.Get(key) fmt.Printf("The value of %q in `%s` is still %q\n", key, bucket, got)
Output: The value of "A" in `things` is "alpha" The value of "A" in `things` is still "alpha"
func (*Bucket) RangeItems ¶
RangeItems returns a slice of key/value pairs for all keys within a given range. Each k/v pair in the slice is of type Item (`struct{ Key, Value []byte }`).
Example ¶
Show that we get items for keys within a given range.
bx, _ := buckets.Open(tempfile()) defer os.Remove(bx.Path()) defer bx.Close() // Create a new bucket named "years". years, _ := bx.New([]byte("years")) // Setup items to insert in `years` bucket items := []struct { Key, Value []byte }{ {[]byte("1970"), []byte("70")}, {[]byte("1975"), []byte("75")}, {[]byte("1980"), []byte("80")}, {[]byte("1985"), []byte("85")}, {[]byte("1990"), []byte("90")}, // min = 1990 {[]byte("1995"), []byte("95")}, // min < 1995 < max {[]byte("2000"), []byte("00")}, // max = 2000 {[]byte("2005"), []byte("05")}, {[]byte("2010"), []byte("10")}, } // Insert 'em. if err := years.Insert(items); err != nil { fmt.Printf("could not insert items in `years` bucket: %v\n", err) } // Time range: 1990 <= key <= 2000. min := []byte("1990") max := []byte("2000") results, err := years.RangeItems(min, max) if err != nil { fmt.Printf("could not get items within range: %v\n", err) } for _, item := range results { fmt.Printf("%s -> %s\n", item.Key, item.Value) }
Output: 1990 -> 90 1995 -> 95 2000 -> 00
type DB ¶
A DB is a bolt database with convenience methods for working with buckets.
A DB embeds the exposed bolt.DB methods.
type PrefixScanner ¶
type PrefixScanner struct { BucketName []byte Prefix []byte // contains filtered or unexported fields }
A PrefixScanner scans a bucket for keys with a given prefix.
func (*PrefixScanner) Count ¶
func (ps *PrefixScanner) Count() (count int, err error)
Count returns a count of the keys with prefix.
func (*PrefixScanner) ItemMapping ¶
func (ps *PrefixScanner) ItemMapping() (map[string][]byte, error)
ItemMapping returns a map of key/value pairs for keys with prefix. This only works with buckets whose keys are byte-sliced strings.
func (*PrefixScanner) Items ¶
func (ps *PrefixScanner) Items() (items []Item, err error)
Items returns a slice of key/value pairs for keys with prefix.
func (*PrefixScanner) Keys ¶
func (ps *PrefixScanner) Keys() (keys [][]byte, err error)
Keys returns a slice of keys with prefix.
func (*PrefixScanner) Map ¶
func (ps *PrefixScanner) Map(do func(k, v []byte) error) error
Map applies `do` on each key/value pair for keys with prefix.
func (*PrefixScanner) Values ¶
func (ps *PrefixScanner) Values() (values [][]byte, err error)
Values returns a slice of values for keys with prefix.
type RangeScanner ¶
type RangeScanner struct { BucketName []byte Min []byte Max []byte // contains filtered or unexported fields }
A RangeScanner scans a bucket for keys within a given range.
func (*RangeScanner) Count ¶
func (rs *RangeScanner) Count() (count int, err error)
Count returns a count of the keys within the range.
func (*RangeScanner) ItemMapping ¶
func (rs *RangeScanner) ItemMapping() (map[string][]byte, error)
ItemMapping returns a map of key/value pairs for keys within the range. This only works with buckets whose keys are byte-sliced strings.
func (*RangeScanner) Items ¶
func (rs *RangeScanner) Items() (items []Item, err error)
Items returns a slice of key/value pairs for keys within the range. Note that the returned slice contains elements of type Item.
func (*RangeScanner) Keys ¶
func (rs *RangeScanner) Keys() (keys [][]byte, err error)
Keys returns a slice of keys within the range.
func (*RangeScanner) Map ¶
func (rs *RangeScanner) Map(do func(k, v []byte) error) error
Map applies `do` on each key/value pair for keys within range.
func (*RangeScanner) Values ¶
func (rs *RangeScanner) Values() (values [][]byte, err error)
Values returns a slice of values for keys within the range.
type Scanner ¶
type Scanner interface { // Map applies a func on each key/value pair scanned. Map(func(k, v []byte) error) error // Count returns a count of the scanned keys. Count() (int, error) // Keys returns a slice of the scanned keys. Keys() ([][]byte, error) // Values returns a slice of values from scanned keys. Values() ([][]byte, error) // Items returns a slice of k/v pairs from scanned keys. Items() ([]Item, error) // ItemMapping returns a mapping of k/v pairs from scanned keys. ItemMapping() (map[string][]byte, error) }
A Scanner implements methods for scanning a subset of keys in a bucket and retrieving data from or about those keys.