badger

package
v1.2.5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 8, 2024 License: GPL-2.0, MIT Imports: 40 Imported by: 0

Documentation

Index

Constants

View Source
const DefaultMaxLimit = 1024
View Source
const IndexGCSkip = 5

Variables

This section is empty.

Functions

func GetCounterKey

func GetCounterKey(ser *serial.T) (key []byte)

GetCounterKey returns the proper counter key for a given event ID.

func GetIndexKeysForEvent

func GetIndexKeysForEvent(ev *event.T, ser *serial.T) (keyz [][]byte)

GetIndexKeysForEvent generates all the index keys required to filter for events. evtSerial should be the output of Serial() which gets a unique, monotonic counter value for each new event.

func GetTagKeyElements

func GetTagKeyElements(tagValue string, CA *createdat.T, ser *serial.T) (prf index.P,
	elems []keys.Element, err error)

func GetTagKeyPrefix

func GetTagKeyPrefix(tagValue string) (key []byte, err error)

GetTagKeyPrefix returns tag index prefixes based on the initial field of a tag.

There is 3 types of index tag keys:

- TagAddr: [ 8 ][ 2b Kind ][ 8b Pubkey ][ address/URL ][ 8b Serial ] - Tag32: [ 7 ][ 8b Pubkey ][ 8b Serial ] - Tag: [ 6 ][ address/URL ][ 8b Serial ]

This function produces the initial bytes without the index.

func PrepareQueries

func PrepareQueries(f *filter.T) (
	qs []query,
	ext *filter.T,
	since uint64,
	err error,
)

PrepareQueries analyses a filter and generates a set of query specs that produce key prefixes to search for in the badger key indexes.

Types

type AccessEvent

type AccessEvent struct {
	EvID eventid.T
	Ts   timestamp.T
	Ser  *serial.T
}

func MakeAccessEvent

func MakeAccessEvent(EvID eventid.T, Ser *serial.T) (ae *AccessEvent)

MakeAccessEvent generates an *AccessEvent from an event ID and serial.

func (AccessEvent) String

func (a AccessEvent) String() (s string)

type Backend

type Backend struct {
	Ctx  context.T
	WG   *sync.WaitGroup
	Path string
	// MaxLimit is the largest a single event JSON can be, in bytes.
	MaxLimit int
	// DBSizeLimit is the number of bytes we want to keep the data store from
	// exceeding.
	DBSizeLimit int
	// DBLowWater is the percentage of DBSizeLimit a GC run will reduce the used
	// storage down to.
	DBLowWater int
	// DBHighWater is the trigger point at which a GC run should start if
	// exceeded.
	DBHighWater int
	// GCFrequency is the frequency of checks of the current utilisation.
	GCFrequency time.Duration
	HasL2       bool
	// Encoder maintains a pool of events that are recently decoded to avoid memory
	// allocation and unnecessary work.
	Encoder *cache.Encoder
	// DB is the badger db interface
	*badger.DB
	// contains filtered or unexported fields
}

func GetBackend

func GetBackend(
	Ctx context.T,
	WG *sync.WaitGroup,
	path string,
	hasL2 bool,
	maxCacheSize int,
	params ...int,
) (b *Backend)

GetBackend returns a reasonably configured badger.Backend.

The variadic params correspond to DBSizeLimit, DBLowWater, DBHighWater and GCFrequency as an integer multiplier of number of seconds.

Note that the cancel function for the context needs to be managed by the caller.

func (*Backend) AccessLoop

func (b *Backend) AccessLoop(c context.T, txMx *sync.Mutex, accCh chan *AccessEvent)

AccessLoop is meant to be run as a goroutine to gather access events in a query and when it finishes, bump all the access records

func (*Backend) Close

func (b *Backend) Close()

func (*Backend) CountEvents

func (b *Backend) CountEvents(c context.T, f *filter.T) (count int, err error)

func (*Backend) DeleteEvent

func (b *Backend) DeleteEvent(c context.T, ev *event.T) (err error)

func (*Backend) EventGCCount added in v1.2.0

func (b *Backend) EventGCCount() (countItems count.Items, total int, err error)

func (*Backend) EventGCMark added in v1.2.0

func (b *Backend) EventGCMark() (deleteItems del.Items, err error)

EventGCMark scans for counter entries and based on GC parameters returns a list of the serials of the events that need to be pruned.

func (*Backend) EventGCRun added in v1.2.0

func (b *Backend) EventGCRun() (err error)

func (*Backend) EventGCSweep added in v1.2.0

func (b *Backend) EventGCSweep(serials del.Items) (err error)

EventGCSweep implements the EventGCSweep function. If hasL2 is true, a separate prune function is called.

func (*Backend) GarbageCollector

func (b *Backend) GarbageCollector()

GarbageCollector starts up a ticker that runs a check on space utilisation and when it exceeds the high-water mark, prunes back to the low-water mark.

This function should be invoked as a goroutine, and will terminate when the backend context is canceled.

func (*Backend) IncrementAccesses

func (b *Backend) IncrementAccesses(txMx *sync.Mutex, acc []*AccessEvent) (err error)

IncrementAccesses takes a list of event IDs of events that were accessed in a query and updates their access counter records.

func (*Backend) IndexGCCount added in v1.2.0

func (b *Backend) IndexGCCount() (serials IndexMap, err error)

func (*Backend) IndexGCMark added in v1.2.0

func (b *Backend) IndexGCMark() (toDelete []uint64, err error)

IndexGCMark scans all indexes, collects the data related to the set of pruned indexes, sorts them by access time and selects the set of index serials to remove

func (*Backend) IndexGCRun added in v1.2.0

func (b *Backend) IndexGCRun() (err error)

func (*Backend) IndexGCSweep added in v1.2.0

func (b *Backend) IndexGCSweep(toDelete []uint64) (err error)

func (*Backend) Init

func (b *Backend) Init() (err error)

func (*Backend) QueryEvents

func (b *Backend) QueryEvents(c context.T, f *filter.T) (ch event.C, err error)

func (*Backend) SaveEvent

func (b *Backend) SaveEvent(c context.T, ev *event.T) (err error)

func (*Backend) Serial

func (b *Backend) Serial() (ser uint64, err error)

func (*Backend) SerialBytes

func (b *Backend) SerialBytes() (ser []byte, err error)

SerialBytes returns a new serial value, used to store an event record with a conflict-free unique code (it is a monotonic, atomic, ascending counter).

func (*Backend) SerialKey

func (b *Backend) SerialKey() (idx []byte, ser *serial.T)

SerialKey returns a key used for storing events, and the raw serial counter bytes to copy into index keys.

func (*Backend) SweepHasL2 added in v1.2.0

func (b *Backend) SweepHasL2(serials del.Items) (err error)

SweepHasL2 implements a prune where only events and access records are deleted. This way searches can find them and fetch them from an L2 event store.

func (*Backend) SweepL1Only added in v1.2.0

func (b *Backend) SweepL1Only(serials del.Items) (err error)

SweepL1Only implements a simple prune that deletes an event and all related indexes.

func (*Backend) Update

func (b *Backend) Update(fn func(txn *badger.Txn) (err error)) (err error)

func (*Backend) View

func (b *Backend) View(fn func(txn *badger.Txn) (err error)) (err error)

func (*Backend) Wipe

func (b *Backend) Wipe() (err error)

type GCCountFunc

type GCCountFunc func(ifc any) (deleteItems del.Items, err error)

type IndexCount added in v1.2.0

type IndexCount struct {
	// contains filtered or unexported fields
}

type IndexMap added in v1.2.0

type IndexMap map[uint64]*IndexCount

type PruneFunc

type PruneFunc func(ifc any, deleteItems del.Items) (err error)

type Results

type Results struct {
	Ev  *event.T
	TS  timestamp.T
	Ser *serial.T
}

type SortPruned added in v1.2.0

type SortPruned []prunedItem

func (SortPruned) Len added in v1.2.0

func (s SortPruned) Len() int

func (SortPruned) Less added in v1.2.0

func (s SortPruned) Less(i, j int) bool

func (SortPruned) Swap added in v1.2.0

func (s SortPruned) Swap(i, j int)

func (SortPruned) Total added in v1.2.0

func (s SortPruned) Total() (total int)

Directories

Path Synopsis
Package keys is a composable framework for constructing badger keys from fields of events.
Package keys is a composable framework for constructing badger keys from fields of events.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL