badger

package
v1.2.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 5, 2024 License: GPL-2.0, MIT Imports: 40 Imported by: 0

Documentation

Index

Constants

View Source 
const DefaultMaxLimit = 1024
View Source 
const IndexGCSkip = 5

Variables

This section is empty.

Functions

func GetCounterKey 

func GetCounterKey(ser *serial.T) (key []byte)

GetCounterKey returns the proper counter key for a given event ID.

func GetIndexKeysForEvent 

func GetIndexKeysForEvent(ev *event.T, ser *serial.T) (keyz [][]byte)

GetIndexKeysForEvent generates all the index keys required to filter for events. evtSerial should be the output of Serial() which gets a unique, monotonic counter value for each new event.

func GetTagKeyElements 

func GetTagKeyElements(tagValue string, CA *createdat.T, ser *serial.T) (prf index.P,
	elems []keys.Element, err error)

func GetTagKeyPrefix 

func GetTagKeyPrefix(tagValue string) (key []byte, err error)

GetTagKeyPrefix returns tag index prefixes based on the initial field of a tag.

There is 3 types of index tag keys:

- TagAddr: [ 8 ][ 2b Kind ][ 8b Pubkey ][ address/URL ][ 8b Serial ] - Tag32: [ 7 ][ 8b Pubkey ][ 8b Serial ] - Tag: [ 6 ][ address/URL ][ 8b Serial ]

This function produces the initial bytes without the index.

func PrepareQueries 

func PrepareQueries(f *filter.T) (
	qs []query,
	ext *filter.T,
	since uint64,
	err error,
)

PrepareQueries analyses a filter and generates a set of query specs that produce key prefixes to search for in the badger key indexes.

Types

type AccessEvent 

type AccessEvent struct {
	EvID eventid.T
	Ts   timestamp.T
	Ser  *serial.T
}

func MakeAccessEvent 

func MakeAccessEvent(EvID eventid.T, Ser *serial.T) (ae *AccessEvent)

MakeAccessEvent generates an *AccessEvent from an event ID and serial.

func (AccessEvent) String 

func (a AccessEvent) String() (s string)

type Backend 

type Backend struct {
	Ctx  context.T
	WG   *sync.WaitGroup
	Path string
	// MaxLimit is the largest a single event JSON can be, in bytes.
	MaxLimit int
	// DBSizeLimit is the number of bytes we want to keep the data store from
	// exceeding.
	DBSizeLimit int
	// DBLowWater is the percentage of DBSizeLimit a GC run will reduce the used
	// storage down to.
	DBLowWater int
	// DBHighWater is the trigger point at which a GC run should start if
	// exceeded.
	DBHighWater int
	// GCFrequency is the frequency of checks of the current utilisation.
	GCFrequency time.Duration
	HasL2       bool
	// Encoder maintains a pool of events that are recently decoded to avoid memory
	// allocation and unnecessary work.
	Encoder *cache.Encoder
	// DB is the badger db interface
	*badger.DB
	// contains filtered or unexported fields
}

func GetBackend 

func GetBackend(
	Ctx context.T,
	WG *sync.WaitGroup,
	path string,
	hasL2 bool,
	maxCacheSize int,
	params ...int,
) (b *Backend)

GetBackend returns a reasonably configured badger.Backend.

The variadic params correspond to DBSizeLimit, DBLowWater, DBHighWater and GCFrequency as an integer multiplier of number of seconds.

Note that the cancel function for the context needs to be managed by the caller.

func (*Backend) AccessLoop 

func (b *Backend) AccessLoop(c context.T, txMx *sync.Mutex, accCh chan *AccessEvent)

AccessLoop is meant to be run as a goroutine to gather access events in a query and when it finishes, bump all the access records

func (*Backend) Close 

func (b *Backend) Close()

func (*Backend) CountEvents 

func (b *Backend) CountEvents(c context.T, f *filter.T) (count int, err error)

func (*Backend) DeleteEvent 

func (b *Backend) DeleteEvent(c context.T, ev *event.T) (err error)

func (*Backend) EventGCCount added in v1.2.0 

func (b *Backend) EventGCCount() (countItems count.Items, total int, err error)

func (*Backend) EventGCMark added in v1.2.0 

func (b *Backend) EventGCMark() (deleteItems del.Items, err error)

EventGCMark scans for counter entries and based on GC parameters returns a list of the serials of the events that need to be pruned.

func (*Backend) EventGCRun added in v1.2.0 

func (b *Backend) EventGCRun() (err error)

func (*Backend) EventGCSweep added in v1.2.0 

func (b *Backend) EventGCSweep(serials del.Items) (err error)

EventGCSweep implements the EventGCSweep function. If hasL2 is true, a separate prune function is called.

func (*Backend) GarbageCollector 

func (b *Backend) GarbageCollector()

GarbageCollector starts up a ticker that runs a check on space utilisation and when it exceeds the high-water mark, prunes back to the low-water mark.

This function should be invoked as a goroutine, and will terminate when the backend context is canceled.

func (*Backend) IncrementAccesses 

func (b *Backend) IncrementAccesses(txMx *sync.Mutex, acc []*AccessEvent) (err error)

IncrementAccesses takes a list of event IDs of events that were accessed in a query and updates their access counter records.

func (*Backend) IndexGCCount added in v1.2.0 

func (b *Backend) IndexGCCount() (serials IndexMap, err error)

func (*Backend) IndexGCMark added in v1.2.0 

func (b *Backend) IndexGCMark() (toDelete []uint64, err error)

IndexGCMark scans all indexes, collects the data related to the set of pruned indexes, sorts them by access time and selects the set of index serials to remove

func (*Backend) IndexGCRun added in v1.2.0 

func (b *Backend) IndexGCRun() (err error)

func (*Backend) IndexGCSweep added in v1.2.0 

func (b *Backend) IndexGCSweep(toDelete []uint64) (err error)

func (*Backend) Init 

func (b *Backend) Init() (err error)

func (*Backend) QueryEvents 

func (b *Backend) QueryEvents(c context.T, f *filter.T) (ch event.C, err error)

func (*Backend) SaveEvent 

func (b *Backend) SaveEvent(c context.T, ev *event.T) (err error)

func (*Backend) Serial 

func (b *Backend) Serial() (ser uint64, err error)

func (*Backend) SerialBytes 

func (b *Backend) SerialBytes() (ser []byte, err error)

SerialBytes returns a new serial value, used to store an event record with a conflict-free unique code (it is a monotonic, atomic, ascending counter).

func (*Backend) SerialKey 

func (b *Backend) SerialKey() (idx []byte, ser *serial.T)

SerialKey returns a key used for storing events, and the raw serial counter bytes to copy into index keys.

func (*Backend) SweepHasL2 added in v1.2.0 

func (b *Backend) SweepHasL2(serials del.Items) (err error)

SweepHasL2 implements a prune where only events and access records are deleted. This way searches can find them and fetch them from an L2 event store.

func (*Backend) SweepL1Only added in v1.2.0 

func (b *Backend) SweepL1Only(serials del.Items) (err error)

SweepL1Only implements a simple prune that deletes an event and all related indexes.

func (*Backend) Update 

func (b *Backend) Update(fn func(txn *badger.Txn) (err error)) (err error)

func (*Backend) View 

func (b *Backend) View(fn func(txn *badger.Txn) (err error)) (err error)

func (*Backend) Wipe 

func (b *Backend) Wipe() (err error)

type GCCountFunc 

type GCCountFunc func(ifc any) (deleteItems del.Items, err error)

type IndexCount added in v1.2.0 

type IndexCount struct {
	// contains filtered or unexported fields
}

type IndexMap added in v1.2.0 

type IndexMap map[uint64]*IndexCount

type PruneFunc 

type PruneFunc func(ifc any, deleteItems del.Items) (err error)

type Results 

type Results struct {
	Ev  *event.T
	TS  timestamp.T
	Ser *serial.T
}

type SortPruned added in v1.2.0 

type SortPruned []prunedItem

func (SortPruned) Len added in v1.2.0 

func (s SortPruned) Len() int

func (SortPruned) Less added in v1.2.0 

func (s SortPruned) Less(i, j int) bool

func (SortPruned) Swap added in v1.2.0 

func (s SortPruned) Swap(i, j int)

func (SortPruned) Total added in v1.2.0 

func (s SortPruned) Total() (total int)

Source Files

View all Source files

Directories

Path Synopsis
Package keys is a composable framework for constructing badger keys from fields of events.
 Package keys is a composable framework for constructing badger keys from fields of events.
arb
count
createdat
id
index
kinder
pubkey
serial

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.