badger

package module
v1.5.1-0...-38533b6 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 15, 2024 License: Apache-2.0 Imports: 40 Imported by: 73

README

BadgerDB GoDoc Go Report Card Build Status Appveyor Coverage Status

Badger mascot

BadgerDB is an embeddable, persistent, simple and fast key-value (KV) database written in pure Go. It's meant to be a performant alternative to non-Go-based key-value stores like RocksDB.

Project Status

Badger v1.0 was released in Nov 2017. Check the Changelog for the full details.

We introduced transactions in v0.9.0 which involved a major API change. If you have a Badger datastore prior to that, please use v0.8.1, but we strongly urge you to upgrade. Upgrading from both v0.8 and v0.9 will require you to take backups and restore using the new version.

Table of Contents

Getting Started

Installing

To start using Badger, install Go 1.8 or above and run go get:

$ go get github.com/dgraph-io/badger/...

This will retrieve the library and install the badger_info command line utility into your $GOBIN path.

Opening a database

The top-level object in Badger is a DB. It represents multiple files on disk in specific directories, which contain the data for a single database.

To open your database, use the badger.Open() function, with the appropriate options. The Dir and ValueDir options are mandatory and must be specified by the client. They can be set to the same value to simplify things.

package main

import (
	"log"

	"github.com/dgraph-io/badger"
)

func main() {
  // Open the Badger database located in the /tmp/badger directory.
  // It will be created if it doesn't exist.
  opts := badger.DefaultOptions
  opts.Dir = "/tmp/badger"
  opts.ValueDir = "/tmp/badger"
  db, err := badger.Open(opts)
  if err != nil {
	  log.Fatal(err)
  }
  defer db.Close()
  // Your code here…
}

Please note that Badger obtains a lock on the directories so multiple processes cannot open the same database at the same time.

Transactions
Read-only transactions

To start a read-only transaction, you can use the DB.View() method:

err := db.View(func(txn *badger.Txn) error {
  // Your code here…
  return nil
})

You cannot perform any writes or deletes within this transaction. Badger ensures that you get a consistent view of the database within this closure. Any writes that happen elsewhere after the transaction has started, will not be seen by calls made within the closure.

Read-write transactions

To start a read-write transaction, you can use the DB.Update() method:

err := db.Update(func(txn *badger.Txn) error {
  // Your code here…
  return nil
})

All database operations are allowed inside a read-write transaction.

Always check the returned error value. If you return an error within your closure it will be passed through.

An ErrConflict error will be reported in case of a conflict. Depending on the state of your application, you have the option to retry the operation if you receive this error.

An ErrTxnTooBig will be reported in case the number of pending writes/deletes in the transaction exceed a certain limit. In that case, it is best to commit the transaction and start a new transaction immediately. Here is an example (we are not checking for errors in some places for simplicity):

updates := make(map[string]string)
txn := db.NewTransaction(true)
for k,v := range updates {
  if err := txn.Set([]byte(k),[]byte(v)); err == ErrTxnTooBig {
    _ = txn.Commit()
    txn = db.NewTransaction(..)
    _ = txn.Set([]byte(k),[]byte(v))
  }
}
_ = txn.Commit()
Managing transactions manually

The DB.View() and DB.Update() methods are wrappers around the DB.NewTransaction() and Txn.Commit() methods (or Txn.Discard() in case of read-only transactions). These helper methods will start the transaction, execute a function, and then safely discard your transaction if an error is returned. This is the recommended way to use Badger transactions.

However, sometimes you may want to manually create and commit your transactions. You can use the DB.NewTransaction() function directly, which takes in a boolean argument to specify whether a read-write transaction is required. For read-write transactions, it is necessary to call Txn.Commit() to ensure the transaction is committed. For read-only transactions, calling Txn.Discard() is sufficient. Txn.Commit() also calls Txn.Discard() internally to cleanup the transaction, so just calling Txn.Commit() is sufficient for read-write transaction. However, if your code doesn’t call Txn.Commit() for some reason (for e.g it returns prematurely with an error), then please make sure you call Txn.Discard() in a defer block. Refer to the code below.

// Start a writable transaction.
txn, err := db.NewTransaction(true)
if err != nil {
    return err
}
defer txn.Discard()

// Use the transaction...
err := txn.Set([]byte("answer"), []byte("42"))
if err != nil {
    return err
}

// Commit the transaction and check for error.
if err := txn.Commit(); err != nil {
    return err
}

The first argument to DB.NewTransaction() is a boolean stating if the transaction should be writable.

Badger allows an optional callback to the Txn.Commit() method. Normally, the callback can be set to nil, and the method will return after all the writes have succeeded. However, if this callback is provided, the Txn.Commit() method returns as soon as it has checked for any conflicts. The actual writing to the disk happens asynchronously, and the callback is invoked once the writing has finished, or an error has occurred. This can improve the throughput of the application in some cases. But it also means that a transaction is not durable until the callback has been invoked with a nil error value.

Using key/value pairs

To save a key/value pair, use the Txn.Set() method:

err := db.Update(func(txn *badger.Txn) error {
  err := txn.Set([]byte("answer"), []byte("42"))
  return err
})

This will set the value of the "answer" key to "42". To retrieve this value, we can use the Txn.Get() method:

err := db.View(func(txn *badger.Txn) error {
  item, err := txn.Get([]byte("answer"))
  if err != nil {
    return err
  }
  val, err := item.Value()
  if err != nil {
    return err
  }
  fmt.Printf("The answer is: %s\n", val)
  return nil
})

Txn.Get() returns ErrKeyNotFound if the value is not found.

Please note that values returned from Get() are only valid while the transaction is open. If you need to use a value outside of the transaction then you must use copy() to copy it to another byte slice.

Use the Txn.Delete() method to delete a key.

Iterating over keys

To iterate over keys, we can use an Iterator, which can be obtained using the Txn.NewIterator() method. Iteration happens in byte-wise lexicographical sorting order.

err := db.View(func(txn *badger.Txn) error {
  opts := badger.DefaultIteratorOptions
  opts.PrefetchSize = 10
  it := txn.NewIterator(opts)
  defer it.Close()
  for it.Rewind(); it.Valid(); it.Next() {
    item := it.Item()
    k := item.Key()
    v, err := item.Value()
    if err != nil {
      return err
    }
    fmt.Printf("key=%s, value=%s\n", k, v)
  }
  return nil
})

The iterator allows you to move to a specific point in the list of keys and move forward or backward through the keys one at a time.

By default, Badger prefetches the values of the next 100 items. You can adjust that with the IteratorOptions.PrefetchSize field. However, setting it to a value higher than GOMAXPROCS (which we recommend to be 128 or higher) shouldn’t give any additional benefits. You can also turn off the fetching of values altogether. See section below on key-only iteration.

Prefix scans

To iterate over a key prefix, you can combine Seek() and ValidForPrefix():

db.View(func(txn *badger.Txn) error {
  it := txn.NewIterator(badger.DefaultIteratorOptions)
  defer it.Close()
  prefix := []byte("1234")
  for it.Seek(prefix); it.ValidForPrefix(prefix); it.Next() {
    item := it.Item()
    k := item.Key()
    v, err := item.Value()
    if err != nil {
      return err
    }
    fmt.Printf("key=%s, value=%s\n", k, v)
  }
  return nil
})
Key-only iteration

Badger supports a unique mode of iteration called key-only iteration. It is several order of magnitudes faster than regular iteration, because it involves access to the LSM-tree only, which is usually resident entirely in RAM. To enable key-only iteration, you need to set the IteratorOptions.PrefetchValues field to false. This can also be used to do sparse reads for selected keys during an iteration, by calling item.Value() only when required.

err := db.View(func(txn *badger.Txn) error {
  opts := badger.DefaultIteratorOptions
  opts.PrefetchValues = false
  it := txn.NewIterator(opts)
  defer it.Close()
  for it.Rewind(); it.Valid(); it.Next() {
    item := it.Item()
    k := item.Key()
    fmt.Printf("key=%s\n", k)
  }
  return nil
})
Garbage Collection

Badger values need to be garbage collected, because of two reasons:

  • Badger keeps values separately from the LSM tree. This means that the compaction operations that clean up the LSM tree do not touch the values at all. Values need to be cleaned up separately.

  • Concurrent read/write transactions could leave behind multiple values for a single key, because they are stored with different versions. These could accumulate, and take up unneeded space beyond the time these older versions are needed.

Badger relies on the client to perform garbage collection at a time of their choosing. It provides the following methods, which can be invoked at an appropriate time:

  • DB.PurgeOlderVersions(): This method iterates over the database, and cleans up all but the latest versions of the key-value pairs. It marks the older versions as deleted, which makes them eligible for garbage collection.

  • DB.PurgeVersionsBelow(key, ts): This method is useful to do a more targeted clean up of older versions of key-value pairs. You can specify a key, and a timestamp. All versions of the key older than the timestamp are marked as deleted, making them eligible for garbage collection.

  • DB.RunValueLogGC(): This method is designed to do garbage collection while Badger is online. Please ensure that you call the DB.Purge…() methods first before invoking this method. It uses any statistics generated by the DB.Purge(…) methods to pick files that are likely to lead to maximum space reclamation. It loops until it encounters a file which does not lead to any garbage collection.

    It could lead to increased I/O if DB.RunValueLogGC() hasn’t been called for a long time, and many deletes have happened in the meanwhile. So it is recommended that this method be called regularly.

Database backup

There are two public API methods DB.Backup() and DB.Load() which can be used to do online backups and restores. Badger v0.9 provides a CLI tool badger, which can do offline backup/restore. Make sure you have $GOPATH/bin in your PATH to use this tool.

The command below will create a version-agnostic backup of the database, to a file badger.bak in the current working directory

badger backup --dir <path/to/badgerdb>

To restore badger.bak in the current working directory to a new database:

badger restore --dir <path/to/badgerdb>

See badger --help for more details.

If you have a Badger database that was created using v0.8 (or below), you can use the badger_backup tool provided in v0.8.1, and then restore it using the command above to upgrade your database to work with the latest version.

badger_backup --dir <path/to/badgerdb> --backup-file badger.bak
Memory usage

Badger's memory usage can be managed by tweaking several options available in the Options struct that is passed in when opening the database using DB.Open.

  • Options.ValueLogLoadingMode can be set to options.FileIO (instead of the default options.MemoryMap) to avoid memory-mapping log files. This can be useful in environments with low RAM.
  • Number of memtables (Options.NumMemtables)
    • If you modify Options.NumMemtables, also adjust Options.NumLevelZeroTables and Options.NumLevelZeroTablesStall accordingly.
  • Number of concurrent compactions (Options.NumCompactors)
  • Mode in which LSM tree is loaded (Options.TableLoadingMode)
  • Size of table (Options.MaxTableSize)
  • Size of value log file (Options.ValueLogFileSize)

If you want to decrease the memory usage of Badger instance, tweak these options (ideally one at a time) until you achieve the desired memory usage.

Statistics

Badger records metrics using the expvar package, which is included in the Go standard library. All the metrics are documented in y/metrics.go file.

expvar package adds a handler in to the default HTTP server (which has to be started explicitly), and serves up the metrics at the /debug/vars endpoint. These metrics can then be collected by a system like Prometheus, to get better visibility into what Badger is doing.

Resources

Blog Posts
  1. Introducing Badger: A fast key-value store written natively in Go
  2. Make Badger crash resilient with ALICE
  3. Badger vs LMDB vs BoltDB: Benchmarking key-value databases in Go
  4. Concurrent ACID Transactions in Badger

Design

Badger was written with these design goals in mind:

  • Write a key-value database in pure Go.
  • Use latest research to build the fastest KV database for data sets spanning terabytes.
  • Optimize for SSDs.

Badger’s design is based on a paper titled WiscKey: Separating Keys from Values in SSD-conscious Storage.

Comparisons
Feature Badger RocksDB BoltDB
Design LSM tree with value log LSM tree only B+ tree
High Read throughput Yes No Yes
High Write throughput Yes Yes No
Designed for SSDs Yes (with latest research 1) Not specifically 2 No
Embeddable Yes Yes Yes
Sorted KV access Yes Yes Yes
Pure Go (no Cgo) Yes No Yes
Transactions Yes, ACID, concurrent with SSI3 Yes (but non-ACID) Yes, ACID
Snapshots Yes Yes Yes
TTL support No Yes No

1 The WISCKEY paper (on which Badger is based) saw big wins with separating values from keys, significantly reducing the write amplification compared to a typical LSM tree.

2 RocksDB is an SSD optimized version of LevelDB, which was designed specifically for rotating disks. As such RocksDB's design isn't aimed at SSDs.

3 SSI: Serializable Snapshot Isolation. For more details, see the blog post Concurrent ACID Transactions in Badger

Benchmarks

We have run comprehensive benchmarks against RocksDB, Bolt and LMDB. The benchmarking code, and the detailed logs for the benchmarks can be found in the badger-bench repo. More explanation, including graphs can be found the blog posts (linked above).

Other Projects Using Badger

Below is a list of known projects that use Badger:

  • 0-stor - Single device object store.
  • Dgraph - Distributed graph database.
  • Sandglass - distributed, horizontally scalable, persistent, time sorted message queue.
  • Usenet Express - Serving over 300TB of data with Badger.
  • go-ipfs - Go client for the InterPlanetary File System (IPFS), a new hypermedia distribution protocol.
  • gorush - A push notification server written in Go.

If you are using Badger in a project please send a pull request to add it to the list.

Frequently Asked Questions

  • My writes are getting stuck. Why?

This can happen if a long running iteration with Prefetch is set to false, but a Item::Value call is made internally in the loop. That causes Badger to acquire read locks over the value log files to avoid value log GC removing the file from underneath. As a side effect, this also blocks a new value log GC file from being created, when the value log file boundary is hit.

Please see Github issues #293 and #315.

There are multiple workarounds during iteration:

  1. Use Item::ValueCopy instead of Item::Value when retrieving value.
  2. Set Prefetch to true. Badger would then copy over the value and release the file lock immediately.
  3. When Prefetch is false, don't call Item::Value and do a pure key-only iteration. This might be useful if you just want to delete a lot of keys.
  4. Do the writes in a separate transaction after the reads.
  • My writes are really slow. Why?

Are you creating a new transaction for every single key update? This will lead to very low throughput. To get best write performance, batch up multiple writes inside a transaction using single DB.Update() call. You could also have multiple such DB.Update() calls being made concurrently from multiple goroutines.

  • I don't see any disk write. Why?

If you're using Badger with SyncWrites=false, then your writes might not be written to value log and won't get synced to disk immediately. Writes to LSM tree are done inmemory first, before they get compacted to disk. The compaction would only happen once MaxTableSize has been reached. So, if you're doing a few writes and then checking, you might not see anything on disk. Once you Close the database, you'll see these writes on disk.

  • Reverse iteration doesn't give me the right results.

Just like forward iteration goes to the first key which is equal or greater than the SEEK key, reverse iteration goes to the first key which is equal or lesser than the SEEK key. Therefore, SEEK key would not be part of the results. You can typically add a tilde (~) as a suffix to the SEEK key to include it in the results. See the following issues: #436 and #347.

  • Which instances should I use for Badger?

We recommend using instances which provide local SSD storage, without any limit on the maximum IOPS. In AWS, these are storage optimized instances like i3. They provide local SSDs which clock 100K IOPS over 4KB blocks easily.

  • I'm getting a closed channel error. Why?
panic: close of closed channel
panic: send on closed channel

If you're seeing panics like above, this would be because you're operating on a closed DB. This can happen, if you call Close() before sending a write, or multiple times. You should ensure that you only call Close() once, and all your read/write operations finish before closing.

  • Are there any Go specific settings that I should use?

We highly recommend setting a high number for GOMAXPROCS, which allows Go to observe the full IOPS throughput provided by modern SSDs. In Dgraph, we have set it to 128. For more details, see this thread.

  • Are there any linux specific settings that I should use?

We recommend setting max file descriptors to a high number depending upon the expected size of you data.

Contact

Documentation

Overview

Package badger implements an embeddable, simple and fast key-value database, written in pure Go. It is designed to be highly performant for both reads and writes simultaneously. Badger uses Multi-Version Concurrency Control (MVCC), and supports transactions. It runs transactions concurrently, with serializable snapshot isolation guarantees.

Badger uses an LSM tree along with a value log to separate keys from values, hence reducing both write amplification and the size of the LSM tree. This allows LSM tree to be served entirely from RAM, while the values are served from SSD.

Usage

Badger has the following main types: DB, Txn, Item and Iterator. DB contains keys that are associated with values. It must be opened with the appropriate options before it can be accessed.

All operations happen inside a Txn. Txn represents a transaction, which can be read-only or read-write. Read-only transactions can read values for a given key (which are returned inside an Item), or iterate over a set of key-value pairs using an Iterator (which are returned as Item type values as well). Read-write transactions can also update and delete keys from the DB.

See the examples for more usage details.

Index

Examples

Constants

View Source 
const (
	// ManifestFilename is the filename for the manifest file.
	ManifestFilename = "MANIFEST"
)

Variables

View Source 
var (
	// ErrValueLogSize is returned when opt.ValueLogFileSize option is not within the valid
	// range.
	ErrValueLogSize = errors.New("Invalid ValueLogFileSize, must be between 1MB and 2GB")

	// ErrValueThreshold is returned when ValueThreshold is set to a value close to or greater than
	// uint16.
	ErrValueThreshold = errors.New("Invalid ValueThreshold, must be lower than uint16.")

	// ErrKeyNotFound is returned when key isn't found on a txn.Get.
	ErrKeyNotFound = errors.New("Key not found")

	// ErrTxnTooBig is returned if too many writes are fit into a single transaction.
	ErrTxnTooBig = errors.New("Txn is too big to fit into one request")

	// ErrConflict is returned when a transaction conflicts with another transaction. This can happen if
	// the read rows had been updated concurrently by another transaction.
	ErrConflict = errors.New("Transaction Conflict. Please retry")

	// ErrReadOnlyTxn is returned if an update function is called on a read-only transaction.
	ErrReadOnlyTxn = errors.New("No sets or deletes are allowed in a read-only transaction")

	// ErrDiscardedTxn is returned if a previously discarded transaction is re-used.
	ErrDiscardedTxn = errors.New("This transaction has been discarded. Create a new one")

	// ErrEmptyKey is returned if an empty key is passed on an update function.
	ErrEmptyKey = errors.New("Key cannot be empty")

	// ErrRetry is returned when a log file containing the value is not found.
	// This usually indicates that it may have been garbage collected, and the
	// operation needs to be retried.
	ErrRetry = errors.New("Unable to find log file. Please retry")

	// ErrThresholdZero is returned if threshold is set to zero, and value log GC is called.
	// In such a case, GC can't be run.
	ErrThresholdZero = errors.New(
		"Value log GC can't run because threshold is set to zero")

	// ErrNoRewrite is returned if a call for value log GC doesn't result in a log file rewrite.
	ErrNoRewrite = errors.New(
		"Value log GC attempt didn't result in any cleanup")

	// ErrRejected is returned if a value log GC is called either while another GC is running, or
	// after DB::Close has been called.
	ErrRejected = errors.New("Value log GC request rejected")

	// ErrInvalidRequest is returned if the user request is invalid.
	ErrInvalidRequest = errors.New("Invalid request")

	// ErrManagedTxn is returned if the user tries to use an API which isn't
	// allowed due to external management of transactions, when using ManagedDB.
	ErrManagedTxn = errors.New(
		"Invalid API request. Not allowed to perform this action using ManagedDB")

	// ErrInvalidDump if a data dump made previously cannot be loaded into the database.
	ErrInvalidDump = errors.New("Data dump cannot be read")

	// ErrZeroBandwidth is returned if the user passes in zero bandwidth for sequence.
	ErrZeroBandwidth = errors.New("Bandwidth must be greater than zero")

	// ErrReplayNeeded is returned when opt.ReadOnly is set but the
	// database requires a value log replay.
	ErrReplayNeeded = errors.New("Database was not properly closed, cannot open read-only")

	// ErrWindowsNotSupported is returned when opt.ReadOnly is used on Windows
	ErrWindowsNotSupported = errors.New("Read-only mode is not supported on Windows")

	// ErrTruncateNeeded is returned when the value log gets corrupt, and requires truncation of
	// corrupt data to allow Badger to run properly.
	ErrTruncateNeeded = errors.New("Value log truncate required to run DB. This might result in data loss.")

	// ErrTruncateNeeded is returned when UserMate size exceed 255.
	ErrUserMetaTooLarge = errors.New("UserMate size exceed 255.")
)
View Source 
var DefaultIteratorOptions = IteratorOptions{
	Reverse:     false,
	AllVersions: false,
}

DefaultIteratorOptions contains default options when iterating over Badger key-value stores.

View Source 
var DefaultOptions = Options{
	DoNotCompact:            false,
	LevelOneSize:            256 << 20,
	MaxMemTableSize:         64 << 20,
	NumCompactors:           3,
	NumLevelZeroTables:      5,
	NumLevelZeroTablesStall: 10,
	NumMemtables:            5,
	SyncWrites:              true,
	ValueLogFileSize:        256 << 20,
	ValueLogMaxEntries:      1000000,
	ValueLogMaxNumFiles:     1,
	ValueThreshold:          32,
	Truncate:                false,
	MaxBlockCacheSize:       1 << 30,
	MaxIndexCacheSize:       1 << 30,
	TableBuilderOptions: options.TableBuilderOptions{
		MaxTableSize:        8 << 20,
		SuRFStartLevel:      8,
		HashUtilRatio:       0.75,
		WriteBufferSize:     2 * 1024 * 1024,
		BytesPerSecond:      -1,
		MaxLevels:           7,
		LevelSizeMultiplier: 10,
		BlockSize:           64 * 1024,

		CompressionPerLevel: []options.CompressionType{options.None, options.None, options.Snappy, options.Snappy, options.Snappy, options.ZSTD, options.ZSTD},
		LogicalBloomFPR:     0.01,
		SuRFOptions: options.SuRFOptions{
			HashSuffixLen:  8,
			RealSuffixLen:  8,
			BitsPerKeyHint: 40,
		},
	},
	ValueLogWriteOptions: options.ValueLogWriterOptions{
		WriteBufferSize: 2 * 1024 * 1024,
	},
	CompactL0WhenClose: true,
}

DefaultOptions sets a list of recommended options for good performance. Feel free to modify these to suit your needs.

View Source 
var ErrExternalTableOverlap = errors.New("keys of external tables has overlap")

ErrExternalTableOverlap returned by IngestExternalFiles when files overlaps.

View Source 
var LSMOnlyOptions = Options{}

LSMOnlyOptions follows from DefaultOptions, but sets a higher ValueThreshold so values would be colocated with the LSM tree, with value log largely acting as a write-ahead log only. These options would reduce the disk usage of value log, and make Badger act like a typical LSM tree.

Functions

func CompactTables 

func CompactTables(cd *CompactDef, stats *y.CompactionStats, discardStats *DiscardStats) ([]*sstable.BuildResult, error)

CompactTables compacts tables in CompactDef and returns the file names.

Types

type CompactDef 

type CompactDef struct {
	Level int

	Top []table.Table
	Bot []table.Table

	SkippedTbls []table.Table
	SafeTS      uint64
	Guards      []Guard
	Filter      CompactionFilter
	HasOverlap  bool
	Opt         options.TableBuilderOptions
	Dir         string
	AllocIDFunc func() uint64
	Limiter     *rate.Limiter
	InMemory    bool
	// contains filtered or unexported fields
}

func (*CompactDef) String 

func (cd *CompactDef) String() string

type CompactionFilter 

type CompactionFilter interface {
	// Filter is the method the compaction process invokes for kv that is being compacted. The returned decision
	// indicates that the kv should be preserved, deleted or dropped in the output of this compaction run.
	Filter(key, val, userMeta []byte) Decision

	// Guards returns specifications that may splits the SST files
	// A key is associated to a guard that has the longest matched Prefix.
	Guards() []Guard
}

CompactionFilter is an interface that user can implement to remove certain keys.

type CompactionReq 

type CompactionReq struct {
	Level        int     `json:"level"`
	Overlap      bool    `json:"overlap"`
	NumTop       int     `json:"num_top"`
	FileSizes    []int64 `json:"file_sizes"`
	SafeTS       uint64  `json:"safe_ts"`
	MaxTableSize int64   `json:"max_table_size"`
}

type CompactionResp 

type CompactionResp struct {
	Error     string             `json:"error"`
	FileSizes []int64            `json:"file_sizes"`
	Stats     *y.CompactionStats `json:"stats"`
	NumSkip   int64              `json:"num_skip"`
	SkipBytes int64              `json:"skip_bytes"`
}

type CompactionServer 

type CompactionServer struct {
	// contains filtered or unexported fields
}

func NewCompactionServer 

func NewCompactionServer(addr string) (*CompactionServer, error)

func (*CompactionServer) Close 

func (s *CompactionServer) Close()

func (*CompactionServer) Run 

func (s *CompactionServer) Run()

type DB added in v0.9.0 

type DB struct {
	// contains filtered or unexported fields
}

DB provides the various functions required to interact with Badger. DB is thread-safe.

func Open added in v0.9.0 

func Open(opt Options) (db *DB, err error)

Open returns a new DB object.

Example 
dir, err := ioutil.TempDir("", "badger")
if err != nil {
	log.Fatal(err)
}
defer os.RemoveAll(dir)
opts := DefaultOptions
opts.Dir = dir
opts.ValueDir = dir
db, err := Open(opts)
if err != nil {
	log.Fatal(err)
}
defer db.Close()

err = db.View(func(txn *Txn) error {
	_, err := txn.Get([]byte("key"))
	// We expect ErrKeyNotFound
	fmt.Println(err)
	return nil
})

if err != nil {
	log.Fatal(err)
}

txn := db.NewTransaction(true) // Read-write txn
err = txn.Set([]byte("key"), []byte("value"))
if err != nil {
	log.Fatal(err)
}
err = txn.Commit()
if err != nil {
	log.Fatal(err)
}

err = db.View(func(txn *Txn) error {
	item, err := txn.Get([]byte("key"))
	if err != nil {
		return err
	}
	val, err := item.Value()
	if err != nil {
		return err
	}
	fmt.Printf("%s\n", string(val))
	return nil
})

if err != nil {
	log.Fatal(err)
}

Output:

Key not found
value

func (*DB) Backup added in v0.9.0 

func (db *DB) Backup(w io.Writer, since uint64) (uint64, error)

Backup dumps a protobuf-encoded list of all entries in the database into the given writer, that are newer than the specified version. It returns a timestamp indicating when the entries were dumped which can be passed into a later invocation to generate an incremental dump, of entries that have been added/modified since the last invocation of DB.Backup()

This can be used to backup the data in a database at a given point in time.

func (*DB) CacheMetrics 

func (db *DB) CacheMetrics() *cache.Metrics

CacheMetrics returns the metrics for the underlying cache.

func (*DB) Close added in v0.9.0 

func (db *DB) Close() (err error)

Close closes a DB. It's crucial to call it to ensure all the pending updates make their way to disk. Calling DB.Close() multiple times is not safe and would cause panic.

func (*DB) DeleteFilesInRange 

func (db *DB) DeleteFilesInRange(start, end []byte)

DeleteFilesInRange delete files in [start, end). If some file contains keys outside the range, they will not be deleted. This function is designed to reclaim space quickly. If you want to ensure no future transaction can read keys in range, considering iterate and delete the remained keys, or using compaction filter to cleanup them asynchronously.

func (*DB) GetVLogOffset 

func (db *DB) GetVLogOffset() uint64

func (*DB) IngestExternalFiles 

func (db *DB) IngestExternalFiles(files []ExternalTableSpec) (int, error)

IngestExternalFiles ingest external constructed tables into DB. Note: insure there is no concurrent write overlap with tables to be ingested.

func (*DB) IsManaged 

func (db *DB) IsManaged() bool

func (*DB) IterateVLog 

func (db *DB) IterateVLog(offset uint64, fn func(e Entry)) error

IterateVLog iterates VLog for external replay, this function should be called only when there is no concurrent write operation on the DB.

func (*DB) Load added in v0.9.0 

func (db *DB) Load(r io.Reader) error

Load reads a protobuf-encoded list of all entries from a reader and writes them to the database. This can be used to restore the database from a backup made by calling DB.Backup().

DB.Load() should be called on a database that is not running any other concurrent transactions while it is running.

func (*DB) NewExternalTableBuilder 

func (db *DB) NewExternalTableBuilder(f *os.File, compression options.CompressionType, limiter *rate.Limiter) *sstable.Builder

NewExternalTableBuilder returns a new sst builder.

func (*DB) NewTransaction added in v0.9.0 

func (db *DB) NewTransaction(update bool) *Txn

NewTransaction creates a new transaction. Badger supports concurrent execution of transactions, providing serializable snapshot isolation, avoiding write skews. Badger achieves this by tracking the keys read and at Commit time, ensuring that these read keys weren't concurrently modified by another transaction.

For read-only transactions, set update to false. In this mode, we don't track the rows read for any changes. Thus, any long running iterations done in this mode wouldn't pay this overhead.

Running transactions concurrently is OK. However, a transaction itself isn't thread safe, and should only be run serially. It doesn't matter if a transaction is created by one goroutine and passed down to other, as long as the Txn APIs are called serially.

When you create a new transaction, it is absolutely essential to call Discard(). This should be done irrespective of what the update param is set to. Commit API internally runs Discard, but running it twice wouldn't cause any issues. 

txn := db.NewTransaction(false)
defer txn.Discard()
// Call various APIs.

func (*DB) Size added in v1.3.0 

func (db *DB) Size() (lsm int64, vlog int64)

Size returns the size of lsm and value log files in bytes. It can be used to decide how often to call RunValueLogGC.

func (*DB) Tables added in v1.4.0 

func (db *DB) Tables() []TableInfo

func (*DB) Update added in v0.9.0 

func (db *DB) Update(fn func(txn *Txn) error) error

Update executes a function, creating and managing a read-write transaction for the user. Error returned by the function is relayed by the Update method.

func (*DB) UpdateSafeTs 

func (db *DB) UpdateSafeTs(ts uint64)

UpdateSafeTs is used for Managed DB, during compaction old version smaller than the safe ts will be discarded. If this is not called, all old versions are kept.

func (*DB) View added in v0.9.0 

func (db *DB) View(fn func(txn *Txn) error) error

View executes a function creating and managing a read-only transaction for the user. Error returned by the function is relayed by the View method.

type Decision 

type Decision int

Decision is the type for compaction filter decision. 

const (
	// DecisionKeep indicates the entry should be reserved.
	DecisionKeep Decision = 0
	// DecisionMarkTombstone converts the entry to a delete tombstone.
	DecisionMarkTombstone Decision = 1
	// DecisionDrop simply drops the entry, doesn't leave a delete tombstone.
	DecisionDrop Decision = 2
)

type DiscardStats 

type DiscardStats struct {
	// contains filtered or unexported fields
}

func (*DiscardStats) String 

func (ds *DiscardStats) String() string

type Entry 

type Entry struct {
	Key      y.Key
	Value    []byte
	UserMeta []byte
	// contains filtered or unexported fields
}

Entry provides Key, Value, UserMeta. This struct can be used by the user to set data.

func (*Entry) SetDelete 

func (e *Entry) SetDelete()

type ExternalTableSpec 

type ExternalTableSpec struct {
	Filename string
}

type Guard 

type Guard struct {
	Prefix   []byte
	MatchLen int
	MinSize  int64
}

Guard specifies when to finish a SST file during compaction. The rule is the following: 1. The key must match the Prefix of the Guard, otherwise the SST should finish. 2. If the key up to MatchLen is the different than the previous key and MinSize is reached, the SST should finish.

type Item added in v0.9.0 

type Item struct {
	// contains filtered or unexported fields
}

Item is returned during iteration. Both the Key() and Value() output is only valid until iterator.Next() is called.

func (*Item) EstimatedSize added in v0.9.0 

func (item *Item) EstimatedSize() int64

EstimatedSize returns approximate size of the key-value pair.

This can be called while iterating through a store to quickly estimate the size of a range of key-value pairs (without fetching the corresponding values).

func (*Item) IsDeleted 

func (item *Item) IsDeleted() bool

IsDeleted returns true if item contains deleted or expired value.

func (*Item) IsEmpty 

func (item *Item) IsEmpty() bool

IsEmpty checks if the value is empty.

func (*Item) Key added in v0.9.0 

func (item *Item) Key() []byte

Key returns the key.

Key is only valid as long as item is valid, or transaction is valid. If you need to use it outside its validity, please use KeyCopy

func (*Item) KeyCopy added in v1.4.0 

func (item *Item) KeyCopy(dst []byte) []byte

KeyCopy returns a copy of the key of the item, writing it to dst slice. If nil is passed, or capacity of dst isn't sufficient, a new slice would be allocated and returned.

func (*Item) String added in v1.4.0 

func (item *Item) String() string

String returns a string representation of Item

func (*Item) UserMeta added in v0.9.0 

func (item *Item) UserMeta() []byte

UserMeta returns the userMeta set by the user. Typically, this byte, optionally set by the user is used to interpret the value.

func (*Item) Value added in v0.9.0 

func (item *Item) Value() ([]byte, error)

Value retrieves the value of the item from the value log.

This method must be called within a transaction. Calling it outside a transaction is considered undefined behavior. If an iterator is being used, then Item.Value() is defined in the current iteration only, because items are reused.

If you need to use a value outside a transaction, please use Item.ValueCopy instead, or copy it yourself. Value might change once discard or commit is called. Use ValueCopy if you want to do a Set after Get.

func (*Item) ValueCopy added in v1.1.0 

func (item *Item) ValueCopy(dst []byte) ([]byte, error)

ValueCopy returns a copy of the value of the item from the value log, writing it to dst slice. If nil is passed, or capacity of dst isn't sufficient, a new slice would be allocated and returned. Tip: It might make sense to reuse the returned slice as dst argument for the next call.

This function is useful in long running iterate/update transactions to avoid a write deadlock. See Github issue: https://github.com/pingcap/badger/issues/315

func (*Item) ValueSize 

func (item *Item) ValueSize() int

ValueSize returns the size of the value without the cost of retrieving the value.

func (*Item) Version added in v0.9.0 

func (item *Item) Version() uint64

Version returns the commit timestamp of the item.

type Iterator 

type Iterator struct {
	// contains filtered or unexported fields
}

Iterator helps iterating over the KV pairs in a lexicographically sorted order.

func (*Iterator) Close 

func (it *Iterator) Close()

Close would close the iterator. It is important to call this when you're done with iteration.

func (*Iterator) Item 

func (it *Iterator) Item() *Item

Item returns pointer to the current key-value pair. This item is only valid until it.Next() gets called.

func (*Iterator) Next 

func (it *Iterator) Next()

Next would advance the iterator by one. Always check it.Valid() after a Next() to ensure you have access to a valid it.Item().

func (*Iterator) Rewind 

func (it *Iterator) Rewind()

Rewind would rewind the iterator cursor all the way to zero-th position, which would be the smallest key if iterating forward, and largest if iterating backward. It does not keep track of whether the cursor started with a Seek().

func (*Iterator) Seek 

func (it *Iterator) Seek(key []byte)

Seek would seek to the provided key if present. If absent, it would seek to the next smallest key greater than provided if iterating in the forward direction. Behavior would be reversed is iterating backwards.

func (*Iterator) SetAllVersions 

func (it *Iterator) SetAllVersions(allVersions bool)

func (*Iterator) Valid 

func (it *Iterator) Valid() bool

Valid returns false when iteration is done.

func (*Iterator) ValidForPrefix 

func (it *Iterator) ValidForPrefix(prefix []byte) bool

ValidForPrefix returns false when iteration is done or when the current key is not prefixed by the specified prefix.

type IteratorOptions 

type IteratorOptions struct {
	Reverse     bool // Direction of iteration. False is forward, true is backward.
	AllVersions bool // Fetch all valid versions of the same key.

	// StartKey and EndKey are used to prune non-overlapping table iterators.
	// They are not boundary limits, the EndKey is exclusive.
	StartKey y.Key
	EndKey   y.Key
	// contains filtered or unexported fields
}

IteratorOptions is used to set options when iterating over Badger key-value stores.

This package provides DefaultIteratorOptions which contains options that should work for most applications. Consider using that as a starting point before customizing it for your own needs.

func (*IteratorOptions) OverlapMemTable 

func (opts *IteratorOptions) OverlapMemTable(t *memtable.Table) bool

func (*IteratorOptions) OverlapPending 

func (opts *IteratorOptions) OverlapPending(it *pendingWritesIterator) bool

func (*IteratorOptions) OverlapTable 

func (opts *IteratorOptions) OverlapTable(t table.Table) bool

func (*IteratorOptions) OverlapTables 

func (opts *IteratorOptions) OverlapTables(tables []table.Table) []table.Table

type ManagedDB added in v0.9.0 

type ManagedDB struct {
	*DB
}

ManagedDB allows end users to manage the transactions themselves. Transaction start and commit timestamps are set by end-user.

This is only useful for databases built on Top of Badger (like Dgraph), and can be ignored by most users.

WARNING: This is an experimental feature and may be changed significantly in a future release. So please proceed with caution.

func OpenManaged added in v0.9.0 

func OpenManaged(opts Options) (*ManagedDB, error)

OpenManaged returns a new ManagedDB, which allows more control over setting transaction timestamps.

This is only useful for databases built on Top of Badger (like Dgraph), and can be ignored by most users.

func (*ManagedDB) NewTransaction added in v0.9.0 

func (db *ManagedDB) NewTransaction(update bool)

NewTransaction overrides DB.NewTransaction() and panics when invoked. Use NewTransactionAt() instead.

func (*ManagedDB) NewTransactionAt added in v0.9.0 

func (db *ManagedDB) NewTransactionAt(readTs uint64, update bool) *Txn

NewTransactionAt follows the same logic as DB.NewTransaction(), but uses the provided read timestamp.

This is only useful for databases built on Top of Badger (like Dgraph), and can be ignored by most users.

type Manifest 

type Manifest struct {
	Levels []levelManifest
	Tables map[uint64]tableManifest

	// Contains total number of creation and deletion changes in the manifest -- used to compute
	// whether it'd be useful to rewrite the manifest.
	Creations int
	Deletions int

	Head *protos.HeadInfo
}

Manifest represents the contents of the MANIFEST file in a Badger store.

The MANIFEST file describes the startup state of the db -- all LSM files and what level they're at.

It consists of a sequence of ManifestChangeSet objects. Each of these is treated atomically, and contains a sequence of ManifestChange's (file creations/deletions) which we use to reconstruct the manifest at startup.

func ReplayManifestFile 

func ReplayManifestFile(fp *os.File) (ret Manifest, truncOffset int64, err error)

ReplayManifestFile reads the manifest file and constructs two manifest objects. (We need one immutable copy and one mutable copy of the manifest. Easiest way is to construct two of them.) Also, returns the last offset after a completely read manifest entry -- the file must be truncated at that point before further appends are made (if there is a partial entry after that). In normal conditions, truncOffset is the file size.

type Options 

type Options struct {
	// 1. Mandatory flags
	// -------------------
	// Directory to store the data in. Should exist and be writable.
	Dir string
	// Directory to store the value log in. Can be the same as Dir. Should
	// exist and be writable.
	ValueDir string

	// 2. Frequently modified flags
	// -----------------------------
	// Sync all writes to disk. Setting this to true would slow down data
	// loading significantly.
	SyncWrites bool

	// 3. Flags that user might want to review
	// ----------------------------------------
	// The following affect all levels of LSM tree.
	MaxMemTableSize int64 // Each mem table is at most this size.
	// If value size >= this threshold, only store value offsets in tree.
	// If set to 0, all values are stored in SST.
	ValueThreshold int
	// Maximum number of tables to keep in memory, before stalling.
	NumMemtables int
	// The following affect how we handle LSM tree L0.
	// Maximum number of Level 0 tables before we start compacting.
	NumLevelZeroTables int

	// If we hit this number of Level 0 tables, we will stall until L0 is
	// compacted away.
	NumLevelZeroTablesStall int

	MaxBlockCacheSize int64
	MaxIndexCacheSize int64

	// Maximum total size for L1.
	LevelOneSize int64

	// Size of single value log file.
	ValueLogFileSize int64

	// Max number of entries a value log file can hold (approximately). A value log file would be
	// determined by the smaller of its file size and max entries.
	ValueLogMaxEntries uint32

	// Max number of value log files to keep before safely remove.
	ValueLogMaxNumFiles int

	// Number of compaction workers to run concurrently.
	NumCompactors int

	// Transaction start and commit timestamps are managed by end-user.
	// A managed transaction can only set values by SetEntry with a non-zero version key.
	ManagedTxns bool

	// 4. Flags for testing purposes
	// ------------------------------
	VolatileMode bool
	DoNotCompact bool // Stops LSM tree from compactions.

	// Open the DB as read-only. With this set, multiple processes can
	// open the same Badger DB. Note: if the DB being opened had crashed
	// before and has vlog data to be replayed, ReadOnly will cause Open
	// to fail with an appropriate message.
	ReadOnly bool

	// Truncate value log to delete corrupt data, if any. Would not truncate if ReadOnly is set.
	Truncate bool

	TableBuilderOptions options.TableBuilderOptions

	ValueLogWriteOptions options.ValueLogWriterOptions

	CompactionFilterFactory func(targetLevel int, smallest, biggest []byte) CompactionFilter

	CompactL0WhenClose bool

	RemoteCompactionAddr string
	// contains filtered or unexported fields
}

Options are params for creating DB object.

This package provides DefaultOptions which contains options that should work for most applications. Consider using that as a starting point before customizing it for your own needs.

type TableInfo added in v1.4.0 

type TableInfo struct {
	ID    uint64
	Level int
	Left  []byte
	Right []byte
}

type Txn added in v0.9.0 

type Txn struct {
	// contains filtered or unexported fields
}

Txn represents a Badger transaction.

func (*Txn) Commit added in v0.9.0 

func (txn *Txn) Commit() error

Commit commits the transaction, following these steps:

1. If there are no writes, return immediately.

2. Check if read rows were updated since txn started. If so, return ErrConflict.

3. If no conflict, generate a commit timestamp and update written rows' commit ts.

4. Batch up all writes, write them to value log and LSM tree.

If error is nil, the transaction is successfully committed. In case of a non-nil error, the LSM tree won't be updated, so there's no need for any rollback.

func (*Txn) CommitAt added in v0.9.0 

func (txn *Txn) CommitAt(commitTs uint64) error

CommitAt commits the transaction, following the same logic as Commit(), but at the given commit timestamp. This will panic if not used with ManagedDB.

This is only useful for databases built on Top of Badger (like Dgraph), and can be ignored by most users.

func (*Txn) Delete added in v0.9.0 

func (txn *Txn) Delete(key []byte) error

Delete deletes a key. This is done by adding a delete marker for the key at commit timestamp. Any reads happening before this timestamp would be unaffected. Any reads after this commit would see the deletion.

func (*Txn) Discard added in v0.9.0 

func (txn *Txn) Discard()

Discard discards a created transaction. This method is very important and must be called. Commit method calls this internally, however, calling this multiple times doesn't cause any issues. So, this can safely be called via a defer right when transaction is created.

NOTE: If any operations are run on a discarded transaction, ErrDiscardedTxn is returned.

func (*Txn) Get added in v0.9.0 

func (txn *Txn) Get(key []byte) (item *Item, rerr error)

Get looks for key and returns corresponding Item. If key is not found, ErrKeyNotFound is returned.

func (*Txn) MultiGet 

func (txn *Txn) MultiGet(keys [][]byte) (items []*Item, err error)

MultiGet gets items for keys, if not found, the corresponding item will be nil. It only supports read-only transaction for simplicity.

func (*Txn) NewIterator added in v0.9.0 

func (txn *Txn) NewIterator(opt IteratorOptions) *Iterator

NewIterator returns a new iterator. Depending upon the options, either only keys, or both key-value pairs would be fetched. The keys are returned in lexicographically sorted order. Avoid long running iterations in update transactions.

Example 
dir, err := ioutil.TempDir("", "badger")
if err != nil {
	log.Fatal(err)
}
defer os.RemoveAll(dir)

opts := DefaultOptions
opts.Dir = dir
opts.ValueDir = dir

db, err := Open(opts)
if err != nil {
	log.Fatal(err)
}
defer db.Close()

bkey := func(i int) []byte {
	return []byte(fmt.Sprintf("%09d", i))
}
bval := func(i int) []byte {
	return []byte(fmt.Sprintf("%025d", i))
}

txn := db.NewTransaction(true)

// Fill in 1000 items
n := 1000
for i := 0; i < n; i++ {
	err := txn.Set(bkey(i), bval(i))
	if err != nil {
		log.Fatal(err)
	}
}

err = txn.Commit()
if err != nil {
	log.Fatal(err)
}

// Iterate over 1000 items
var count int
err = db.View(func(txn *Txn) error {
	it := txn.NewIterator(DefaultIteratorOptions)
	defer it.Close()
	for it.Rewind(); it.Valid(); it.Next() {
		count++
	}
	return nil
})
if err != nil {
	log.Fatal(err)
}
fmt.Printf("Counted %d elements", count)

Output:

Counted 1000 elements

func (*Txn) Set added in v0.9.0 

func (txn *Txn) Set(key, val []byte) error

Set adds a key-value pair to the database.

It will return ErrReadOnlyTxn if update flag was set to false when creating the transaction.

func (*Txn) SetEntry added in v1.2.0 

func (txn *Txn) SetEntry(e *Entry) error

SetEntry takes an Entry struct and adds the key-value pair in the struct, along with other metadata to the database.

func (*Txn) SetReadTS 

func (txn *Txn) SetReadTS(readTS uint64)

SetReadTS reads the DB with a given TS, it can only be used in a managed DB.

func (*Txn) SetWithMeta added in v1.0.0 

func (txn *Txn) SetWithMeta(key, val []byte, meta byte) error

SetWithMeta adds a key-value pair to the database, along with a metadata byte. This byte is stored alongside the key, and can be used as an aid to interpret the value or store other contextual bits corresponding to the key-value pair.

func (*Txn) SetWithMetaSlice 

func (txn *Txn) SetWithMetaSlice(key, val, meta []byte) error

Source Files

View all Source files

Directories

Path Synopsis
Package buffer implements a variable-sized bytes pool.
 Package buffer implements a variable-sized bytes pool.
Ristretto is a fast, fixed size, in-memory cache with a dual focus on throughput and hit ratio performance.
 Ristretto is a fast, fixed size, in-memory cache with a dual focus on throughput and hit ratio performance.
sim
z
cmd
badger
badger/cmd
directio is a proxy package for github.com/pingcap/badger/directio
 directio is a proxy package for github.com/pingcap/badger/directio
memtable
sstable
y

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.