pebble

package
v0.37.17-remove-chunk-... Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 15, 2024 License: AGPL-3.0 Imports: 29 Imported by: 0

Documentation

Index

Constants

View Source 
const DefaultCacheSize = uint(10_000)
View Source 
const DefaultPebbleCacheSize = 1 << 20
View Source 
const (

	// MinLookupKeyLen defines the minimum length for a valid lookup key
	//
	// Lookup keys use the following format:
	//     [code] [owner] / [key] / [height]
	// Where:
	// - code: 1 byte indicating the type of data stored
	// - owner: optional variable length field
	// - key: optional variable length field
	// - height: 8 bytes representing the block height (uint64)
	// - separator: '/' is used to separate variable length fields (required 2)
	//
	// Therefore the minimum key would be 3 bytes + # of bytes for height
	//     [code] / / [height]
	MinLookupKeyLen = 3 + registers.HeightSuffixLen
)

Variables

View Source 
var ErrAlreadyBootstrapped = errors.New("found latest key set on badger instance, DB is already bootstrapped")

ErrAlreadyBootstrapped is the sentinel error for an already bootstrapped pebble instance

Functions

func DefaultPebbleOptions 

func DefaultPebbleOptions(cache *pebble.Cache, comparer *pebble.Comparer) *pebble.Options

DefaultPebbleOptions returns an optimized set of pebble options. This is mostly copied form pebble's nightly performance benchmark.

func IsBootstrapped 

func IsBootstrapped(db *pebble.DB) (bool, error)

IsBootstrapped returns true if the db is bootstrapped otherwise return false it returns error if the db is corrupted or other exceptions

func NewBootstrappedRegistersWithPathForTest added in v0.32.7 

func NewBootstrappedRegistersWithPathForTest(tb testing.TB, dir string, first, latest uint64) *pebble.DB

func OpenDefaultPebbleDB added in v0.33.36 

func OpenDefaultPebbleDB(dir string) (*pebble.DB, error)

OpenDefaultPebbleDB opens a pebble database using default options, such as cache size and comparer

func OpenRegisterPebbleDB 

func OpenRegisterPebbleDB(dir string) (*pebble.DB, error)

OpenRegisterPebbleDB opens the database The difference between OpenDefaultPebbleDB is that it uses a customized comparer (NewMVCCComparer) which is needed to implement finding register values at any given height using pebble's SeekPrefixGE function

func ReadHeightsFromBootstrappedDB 

func ReadHeightsFromBootstrappedDB(db *pebble.DB) (firstHeight uint64, latestHeight uint64, err error)

ReadHeightsFromBootstrappedDB reads the first and latest height from a bootstrapped register db If the register db is not bootstrapped, it returns storage.ErrNotBootstrapped If the register db is corrupted, it returns an error

func RunWithRegistersStorageAtInitialHeights 

func RunWithRegistersStorageAtInitialHeights(tb testing.TB, first uint64, latest uint64, f func(r *Registers))

Types

type Batch added in v0.33.36 

type Batch struct {
	// contains filtered or unexported fields
}

func NewBatch added in v0.33.36 

func NewBatch(db *pebble.DB) *Batch

func (*Batch) Close added in v0.33.36 

func (b *Batch) Close() error

func (*Batch) Flush added in v0.33.36 

func (b *Batch) Flush() error

Flush will call the badger Batch's Flush method, in addition, it will call the callbacks added by OnSucceed any error are exceptions

func (*Batch) GetWriter added in v0.33.36 

func (b *Batch) GetWriter() *pebble.Batch

func (*Batch) OnSucceed added in v0.33.36 

func (b *Batch) OnSucceed(callback func())

OnSucceed adds a callback to execute after the batch has been successfully flushed. useful for implementing the cache where we will only cache after the batch has been successfully flushed

type Cache added in v0.33.36 

type Cache[K comparable, V any] struct {
	// contains filtered or unexported fields
}

Cache is a read-through cache for underlying storage layer. Note: when a resource is not found in the cache nor the underlying storage, then it will not be cached. In other words, finding the missing item again will query the underlying storage again.

func (*Cache[K, V]) Get added in v0.33.36 

func (c *Cache[K, V]) Get(key K) func(pebble.Reader) (V, error)

Get will try to retrieve the resource from cache first, and then from the injected. During normal operations, the following error returns are expected:

  • `storage.ErrNotFound` if key is unknown.

func (*Cache[K, V]) Insert added in v0.33.36 

func (c *Cache[K, V]) Insert(key K, resource V)

Insert will add a resource directly to the cache with the given ID assuming the resource has been added to storage already.

func (*Cache[K, V]) IsCached added in v0.33.36 

func (c *Cache[K, V]) IsCached(key K) bool

IsCached returns true if the key exists in the cache. It DOES NOT check whether the key exists in the underlying data store.

func (*Cache[K, V]) PutTx added in v0.33.36 

func (c *Cache[K, V]) PutTx(key K, resource V) func(pebble.Writer) error

PutTx will return tx which adds a resource to the cache with the given ID.

func (*Cache[K, V]) Remove added in v0.33.36 

func (c *Cache[K, V]) Remove(key K)

type CacheBackend added in v0.33.12 

type CacheBackend interface {
	Get(key string) (value flow.RegisterValue, ok bool)
	Add(key string, value flow.RegisterValue)
	Contains(key string) bool
	Len() int
	Remove(key string)
}

type CacheType added in v0.33.12 

type CacheType int
const (
	CacheTypeLRU CacheType = iota + 1
	CacheTypeTwoQueue
)

func ParseCacheType added in v0.33.12 

func ParseCacheType(s string) (CacheType, error)

func (CacheType) String added in v0.33.12 

func (m CacheType) String() string

type ChunkDataPacks added in v0.33.36 

type ChunkDataPacks struct {
	// contains filtered or unexported fields
}

func NewChunkDataPacks added in v0.33.36 

func NewChunkDataPacks(collector module.CacheMetrics, db *pebble.DB, collections storage.Collections, byChunkIDCacheSize uint) *ChunkDataPacks

func (*ChunkDataPacks) BatchRemove added in v0.33.36 

func (ch *ChunkDataPacks) BatchRemove(chunkID flow.Identifier, batch storage.BatchStorage) error

BatchRemove is not used in pebble implementation

func (*ChunkDataPacks) ByChunkID added in v0.33.36 

func (ch *ChunkDataPacks) ByChunkID(chunkID flow.Identifier) (*flow.ChunkDataPack, error)

ByChunkID finds the chunk data pack by chunk ID. it returns storage.ErrNotFound if not found other errors are exceptions

func (*ChunkDataPacks) PebbleBatchRemove 

func (ch *ChunkDataPacks) PebbleBatchRemove(chunkID flow.Identifier, batch pebble.Writer) error

func (*ChunkDataPacks) Remove added in v0.33.36 

func (ch *ChunkDataPacks) Remove(cs []flow.Identifier) error

Remove removes chunk data packs by IDs, it removes them atomically. Any errors are exceptions

func (*ChunkDataPacks) Store added in v0.33.36 

func (ch *ChunkDataPacks) Store(cs []*flow.ChunkDataPack) error

Store stores the given chunk data pack lists, it stores them atomically. Any error are exceptions

type ConsumerProgress added in v0.37.1 

type ConsumerProgress struct {
	// contains filtered or unexported fields
}

func NewConsumerProgress added in v0.37.1 

func NewConsumerProgress(db *pebble.DB, consumer string) *ConsumerProgress

func (*ConsumerProgress) InitProcessedIndex added in v0.37.1 

func (cp *ConsumerProgress) InitProcessedIndex(defaultIndex uint64) error

InitProcessedIndex insert the default processed index to the storage layer, can only be done once. initialize for the second time will return storage.ErrAlreadyExists

func (*ConsumerProgress) ProcessedIndex added in v0.37.1 

func (cp *ConsumerProgress) ProcessedIndex() (uint64, error)

func (*ConsumerProgress) SetProcessedIndex added in v0.37.1 

func (cp *ConsumerProgress) SetProcessedIndex(processed uint64) error

type ReadCache added in v0.33.12 

type ReadCache struct {
	// contains filtered or unexported fields
}

func (*ReadCache) Get added in v0.33.12 

func (c *ReadCache) Get(key string) (flow.RegisterValue, error)

Get will try to retrieve the resource from cache first, and then from the injected. During normal operations, the following error returns are expected:

  • `storage.ErrNotFound` if key is unknown.

func (*ReadCache) Insert added in v0.33.12 

func (c *ReadCache) Insert(key string, resource flow.RegisterValue)

Insert will add a resource directly to the cache with the given ID

func (*ReadCache) IsCached added in v0.33.12 

func (c *ReadCache) IsCached(key string) bool

IsCached returns true if the key exists in the cache. It DOES NOT check whether the key exists in the underlying data store.

func (*ReadCache) Remove added in v0.33.12 

func (c *ReadCache) Remove(key string)

type RegisterBootstrap 

type RegisterBootstrap struct {
	// contains filtered or unexported fields
}

func NewRegisterBootstrap 

func NewRegisterBootstrap(
	db *pebble.DB,
	checkpointFile string,
	rootHeight uint64,
	rootHash ledger.RootHash,
	log zerolog.Logger,
) (*RegisterBootstrap, error)

NewRegisterBootstrap creates the bootstrap object for reading checkpoint data and the height tracker in pebble This object must be initialized and RegisterBootstrap.IndexCheckpointFile must be run to have the pebble db instance in the correct state to initialize a Registers store.

func (*RegisterBootstrap) IndexCheckpointFile 

func (b *RegisterBootstrap) IndexCheckpointFile(ctx context.Context, workerCount int) error

IndexCheckpointFile indexes the checkpoint file in the Dir provided

type Registers 

type Registers struct {
	// contains filtered or unexported fields
}

Registers library that implements pebble storage for registers given a pebble instance with root block and root height populated

func NewBootstrappedRegistersWithPath 

func NewBootstrappedRegistersWithPath(dir string) (*Registers, *pebble.DB, error)

NewBootstrappedRegistersWithPath initializes a new Registers instance with a pebble db if the database is not initialized, it close the database and return storage.ErrNotBootstrapped

func NewRegisters 

func NewRegisters(db *pebble.DB) (*Registers, error)

NewRegisters takes a populated pebble instance with LatestHeight and FirstHeight set. return storage.ErrNotBootstrapped if they those two keys are unavailable as it implies a uninitialized state return other error if database is in a corrupted state

func (*Registers) FirstHeight 

func (s *Registers) FirstHeight() uint64

FirstHeight first indexed height found in the store, typically root block for the spork

func (*Registers) Get 

func (s *Registers) Get(
	reg flow.RegisterID,
	height uint64,
) (flow.RegisterValue, error)

Get returns the most recent updated payload for the given RegisterID. "most recent" means the updates happens most recent up the given height.

For example, if there are 2 values stored for register A at height 6 and 11, then GetPayload(13, A) would return the value at height 11.

- storage.ErrNotFound if no register values are found - storage.ErrHeightNotIndexed if the requested height is out of the range of stored heights

func (*Registers) LatestHeight 

func (s *Registers) LatestHeight() uint64

LatestHeight Gets the latest height of complete registers available

func (*Registers) Store 

func (s *Registers) Store(
	entries flow.RegisterEntries,
	height uint64,
) error

Store sets the given entries in a batch. This function is expected to be called at one batch per height, sequentially. Under normal conditions, it should be called wth the value of height set to LatestHeight + 1 CAUTION: This function is not safe for concurrent use.

type RegistersCache added in v0.33.12 

type RegistersCache struct {
	*Registers
	// contains filtered or unexported fields
}

func NewRegistersCache added in v0.33.12 

func NewRegistersCache(registers *Registers, cacheType CacheType, size uint, metrics module.CacheMetrics) (*RegistersCache, error)

NewRegistersCache wraps a read cache around Get requests to a underlying Registers object.

func (*RegistersCache) Get added in v0.33.12 

func (c *RegistersCache) Get(
	reg flow.RegisterID,
	height uint64,
) (flow.RegisterValue, error)

Get returns the most recent updated payload for the given RegisterID. "most recent" means the updates happens most recent up the given height.

For example, if there are 2 values stored for register A at height 6 and 11, then GetPayload(13, A) would return the value at height 11.

- storage.ErrNotFound if no register values are found - storage.ErrHeightNotIndexed if the requested height is out of the range of stored heights

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.