bloom

Documentation

Overview

Package bloom package

Index

• Constants
• Variables
• type BlockBloomFilter
  • func NewBlockBloomFilter(n uint, fp float64, opts ...Option) (*BlockBloomFilter, error)
  • func NewBlockBloomFilterWithOption(n uint, fp float64, opt Options) (*BlockBloomFilter, error)
  • func (f *BlockBloomFilter) Add(k []byte)
  • func (f *BlockBloomFilter) AddBlock(block *storePb.BlockWithRWSet)
  • func (f *BlockBloomFilter) AddBlockRWSets(height uint64, rwSets []*commonPb.TxRWSet)
  • func (f *BlockBloomFilter) Close() error
  • func (f *BlockBloomFilter) Dump() error
  • func (f *BlockBloomFilter) Has(contractName string, key []byte) bool
  • func (f *BlockBloomFilter) LastDumpHeight() uint64
• type BlockLoader
• type BloomFilter
  • func (f *BloomFilter) Close() error
  • func (f *BloomFilter) Copy() *BloomFilter
  • func (f *BloomFilter) Dump(label uint64) error
  • func (f *BloomFilter) FP() float64
  • func (f *BloomFilter) ItemCapacity() uint
  • func (f *BloomFilter) LastDumpLabel() uint64
  • func (f *BloomFilter) ReadFrom(r io.Reader) (int64, error)
  • func (f *BloomFilter) WriteTo(w io.Writer) (int64, error)
• type BloomLoader
  • func (l *BloomLoader) Entries() uint
  • func (l *BloomLoader) FP() float64
  • func (l *BloomLoader) ID() uint64
  • func (l *BloomLoader) Label() uint64
  • func (l *BloomLoader) Size() int64
• type ContractKeyGenerator
• type Option
  • func WithBlockLoader(loader BlockLoader) Option
  • func WithContractKeyGenerator(generator ContractKeyGenerator) Option
  • func WithDumpAsync() Option
  • func WithDumpCycle(cycle int) Option
  • func WithDumpFile(file string) Option
  • func WithLogger(logger protocol.Logger) Option
  • func WithRestoreAsync() Option
• type Options
  • func (o *Options) String() string
• type WriteSyncer

Constants

const (
	// KiB 表示一个千字节（KiB），即1024字节
	KiB = 1024
	// MiB 表示兆字节，即千兆比特 通过将 KiB 乘以 1024，我们可以得到 MiB
	MiB = KiB * 1024
)

Variables

var (
	//ErrNoDumpFile no specify dump file error
	ErrNoDumpFile = errors.New("No dump file specified")
	//ErrNoBlockLoader no specify block loader error
	ErrNoBlockLoader = errors.New("No block loader specified")
	//ErrInvalidFile bloom file is invalid
	ErrInvalidFile = errors.New("invalid bloom file")
	//ErrVersionMismatch version mismatch
	ErrVersionMismatch = errors.New("version mismatch")
	//ErrChecksum data is bad
	ErrChecksum = errors.New("checksum error")
	//ErrDumpBeforeSynced cant dump manually before sync complete
	ErrDumpBeforeSynced = errors.New("cant dump manually before sync complete")
	//ErrDumpBusy dump is doing by others
	ErrDumpBusy = errors.New("dump busily")
	//ErrMetaAllInvalid meta both are bad
	ErrMetaAllInvalid = errors.New("meta can't both invalid")
)

Types

type BlockBloomFilter

type BlockBloomFilter struct {
	// contains filtered or unexported fields
}

BlockBloomFilter bloom filter for block

func NewBlockBloomFilter

func NewBlockBloomFilter(n uint, fp float64, opts ...Option) (*BlockBloomFilter, error)

NewBlockBloomFilter creates a new bloom filter. The param n is the number of items(keys) that the bloom filter will need to store. The param fp is the maximum false positive probability Uses the default options if no additional options are provided. Return an error if the bloom filter cannot be created.

func NewBlockBloomFilterWithOption

func NewBlockBloomFilterWithOption(n uint, fp float64, opt Options) (*BlockBloomFilter, error)

NewBlockBloomFilterWithOption creates a new BlockBloomFilter object with the given Options. If the dumpFile in options exists, it loads the bloom filter from the file. Create a thread pool for concurrent addition of write sets in blocks. If a block loader is provided, it asynchronously or synchronously syncs the blocks. Note: the bloom filter will not provides the query function before syncs the blocks completely.

func (*BlockBloomFilter) Add

func (f *BlockBloomFilter) Add(k []byte)

Add adds a key to the bloom filter.

func (*BlockBloomFilter) AddBlock

func (f *BlockBloomFilter) AddBlock(block *storePb.BlockWithRWSet)

AddBlock adds keys of all write sets in the block to the bloom filter.

func (*BlockBloomFilter) AddBlockRWSets

func (f *BlockBloomFilter) AddBlockRWSets(height uint64, rwSets []*commonPb.TxRWSet)

AddBlockRWSets adds keys of all write sets to the bloom filter. The param height used as a mark when dumping data to file.

func (*BlockBloomFilter) Close

func (f *BlockBloomFilter) Close() error

Close release the goroutines pool and close the bloom filter

func (*BlockBloomFilter) Dump

func (f *BlockBloomFilter) Dump() error

Dump dumping manually is not allowed before the filter has finished syncing, because this behavior can lead to errors in the restore data process. If allowed, it will dump bloom filter data to file with current height.

func (*BlockBloomFilter) Has

func (f *BlockBloomFilter) Has(contractName string, key []byte) bool

Has test a key exists in the bloom filter or not. It doesn't work and return true always if the bloom filter is restoring data.

func (*BlockBloomFilter) LastDumpHeight

func (f *BlockBloomFilter) LastDumpHeight() uint64

LastDumpHeight returns the last dump height

type BlockLoader

type BlockLoader interface {
	//Height get the last block height
	Height() uint64
	//GetBlockWithRWSets get block and all the rwsets corresponding to the block
	GetBlockWithRWSets(height uint64) (*storePb.BlockWithRWSet, error)
}

BlockLoader used to load blocks

type BloomFilter

type BloomFilter struct {
	*bloom.BloomFilter
	// contains filtered or unexported fields
}

BloomFilter a bloom filter with dump file

func (*BloomFilter) Close

func (f *BloomFilter) Close() error

Close close the bloom filter

func (*BloomFilter) Copy

func (f *BloomFilter) Copy() *BloomFilter

Copy copy bloom filter

func (*BloomFilter) Dump

func (f *BloomFilter) Dump(label uint64) error

Dump dump bloom filter data to file with label as mark.

func (*BloomFilter) FP

func (f *BloomFilter) FP() float64

FP get the false positive probability

func (*BloomFilter) ItemCapacity

func (f *BloomFilter) ItemCapacity() uint

ItemCapacity get the Pre allocated item capacity

func (*BloomFilter) LastDumpLabel

func (f *BloomFilter) LastDumpLabel() uint64

LastDumpLabel get the last dump label

func (*BloomFilter) ReadFrom

func (f *BloomFilter) ReadFrom(r io.Reader) (int64, error)

ReadFrom read the basic info(m,k,size) from reader firstly. Then read size-sized bytes of data from a file. Rebuild a bloom filter with m, k and a uint64 slice converted from bytes slice.

func (*BloomFilter) WriteTo

func (f *BloomFilter) WriteTo(w io.Writer) (int64, error)

WriteTo write bloom filter data to the writer, including basic data and valid data. Return the number of bytes written. Returns an error encountered.

type BloomLoader

type BloomLoader struct {
	io.Reader
	// contains filtered or unexported fields
}

BloomLoader used to load bloom filter data from bloom file to recovery a bloom filter instance.

func (*BloomLoader) Entries

func (l *BloomLoader) Entries() uint

Entries get preset number of entries of the filter

func (*BloomLoader) FP

func (l *BloomLoader) FP() float64

FP get preset false positive rate

func (*BloomLoader) ID

func (l *BloomLoader) ID() uint64

ID returns id of data saved. Zero means no data saved before.

func (*BloomLoader) Label

func (l *BloomLoader) Label() uint64

Label return the label of data saved .

func (*BloomLoader) Size

func (l *BloomLoader) Size() int64

Size get the size of data saved

type ContractKeyGenerator

type ContractKeyGenerator func(contractName string, key []byte) []byte

ContractKeyGenerator generate a string with contract name and key

type Option

type Option func(*Options)

Option option function used to config options.

func WithBlockLoader

func WithBlockLoader(loader BlockLoader) Option

WithBlockLoader set block loader used to restore missed blocks when block bloom filter created.

func WithContractKeyGenerator

func WithContractKeyGenerator(generator ContractKeyGenerator) Option

WithContractKeyGenerator set contract key generator, block bloom filter use it to generate key for write set.

func WithDumpAsync

func WithDumpAsync() Option

WithDumpAsync set dump async for option

func WithDumpCycle

func WithDumpCycle(cycle int) Option

WithDumpCycle set dump cycle for option

func WithDumpFile

func WithDumpFile(file string) Option

WithDumpFile set dump file for option

func WithLogger

func WithLogger(logger protocol.Logger) Option

WithLogger set logger used to print logger for block bloom filter

func WithRestoreAsync

func WithRestoreAsync() Option

WithRestoreAsync set restore async for option

type Options

type Options struct {
	// contains filtered or unexported fields
}

Options option config for bloom filter

func (*Options) String

func (o *Options) String() string

String print options

type WriteSyncer

type WriteSyncer interface {
	io.Writer
	// Sync commits the current contents of the file to stable storage.
	Sync() error
}

WriteSyncer is the interface that wraps basic Sync method, which