trie

package
v0.0.0-...-7ece11e Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 29, 2023 License: MIT Imports: 22 Imported by: 0

README

#Trie package It's patricia merkle tree based on ethereum design and implementation (geth)

The core of the trie, and its sole requirement in terms of the protocol specification is to provide a single value that identifies a given set of key-value pairs, which may be either a 32 byte sequence or the empty byte sequence

encoding

compact hex data to save storage, this kind of encoding only effect data from node to storage and vice versa

node

implemented tree nodes in patricia merkle tree, node is purely place to hold data and form a trie

node and raw node

  • node: collapsed trie node (node in trie)
  • raw node: already encoded rlp binary object (not in trie but already encoded as rlp)

hasher

hash node and return hashNode (hash of that node)

intermediate Writer

  • manage node and raw full node
  • get raw node from db
  • commit all nodes in trie from one root node (traverse all nodes from root node then put to batch then write batch to db)

trie and secure trie

  • get, update, insert, hash by trie rule from intermediate writer

iterator

  • iterate trie using depth first search (stack)

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrAlreadyProcessed = errors.New("already processed")

ErrAlreadyProcessed is returned by the trie sync when it's requested to process a node it already processed previously.

View Source 
var ErrNotRequested = errors.New("not requested")

ErrNotRequested is returned by the trie sync when it's requested to process a node it did not request.

View Source 
var Logger = TLogger{}

Global instant to use

Functions

func VerifyProof 

func VerifyProof(rootHash common.Hash, key []byte, proofDb incdb.Database) (value []byte, nodes int, err error)

VerifyProof checks merkle proofs. The given proof must contain the value for key in a trie with the given root hash. VerifyProof returns an error if the proof contains invalid trie nodes or the wrong value.

Types

type IntermediateWriter 

type IntermediateWriter struct {
	// contains filtered or unexported fields
}

IntermediateWriter is an intermediate write layer between the trie data structures and the disk database. The aim is to accumulate trie writes in-memory and only periodically flush a couple tries to disk, garbage collecting the remainder.

Note, the trie Database is **not** thread safe in its mutations, but it **is** thread safe in providing individual, independent node access. The rationale behind this split design is to provide read access to RPC handlers and sync servers even while the trie is executing expensive garbage collection.

func NewDatabaseWithCache 

func NewDatabaseWithCache(diskdb incdb.Database, cache int) *IntermediateWriter

NewDatabaseWithCache creates a new trie database to store ephemeral trie content before its written out to disk or garbage collected. It also acts as a read cache for nodes loaded from disk.

func NewIntermediateWriter 

func NewIntermediateWriter(diskdb incdb.Database) *IntermediateWriter

NewIntermediateWriter creates a new trie database to store ephemeral trie content before its written out to disk or garbage collected. No read cache is created, so all data retrievals will hit the underlying disk database.

func (*IntermediateWriter) Cap 

func (intermediateWriter *IntermediateWriter) Cap(limit common.StorageSize) error

Cap iteratively flushes old but still referenced trie nodes until the total memory usage goes below the given threshold.

Note, this method is a non-synchronized mutator. It is unsafe to call this concurrently with other mutators.

func (*IntermediateWriter) Commit 

func (intermediateWriter *IntermediateWriter) Commit(node common.Hash, report bool) error

Commit iterates over all the children of a particular node, writes them out to disk, forcefully tearing down all references in both directions. As a side effect, all pre-images accumulated up to this point are also written.

Note, this method is a non-synchronized mutator. It is unsafe to call this concurrently with other mutators.

func (*IntermediateWriter) Dereference 

func (intermediateWriter *IntermediateWriter) Dereference(root common.Hash)

Dereference removes an existing reference from a root node.

func (*IntermediateWriter) DiskDB 

func (intermediateWriter *IntermediateWriter) DiskDB() incdb.KeyValueReader

DiskDB retrieves the persistent storage backing the trie database.

func (*IntermediateWriter) InsertBlob 

func (intermediateWriter *IntermediateWriter) InsertBlob(hash common.Hash, blob []byte)

InsertBlob writes a new reference tracked blob to the memory database if it's yet unknown. This method should only be used for non-trie nodes that require reference counting, since trie nodes are garbage collected directly through their embedded children.

func (*IntermediateWriter) Node 

func (intermediateWriter *IntermediateWriter) Node(hash common.Hash) ([]byte, error)

Node retrieves an encoded cached trie node from memory. If it cannot be found cached, the method queries the persistent database for the content.

func (*IntermediateWriter) Nodes 

func (intermediateWriter *IntermediateWriter) Nodes() []common.Hash

Nodes retrieves the hashes of all the nodes cached within the memory database. This method is extremely expensive and should only be used to validate internal states in test code.

func (*IntermediateWriter) Reference 

func (intermediateWriter *IntermediateWriter) Reference(child common.Hash, parent common.Hash)

Reference adds a new reference from a parent node to a child node.

func (*IntermediateWriter) Size 

func (intermediateWriter *IntermediateWriter) Size() (common.StorageSize, common.StorageSize)

Size returns the current storage size of the memory cache in front of the persistent database layer.

type Iterator 

type Iterator struct {
	Key   []byte // Current data key on which the iterator is positioned on
	Value []byte // Current data value on which the iterator is positioned on
	Err   error
	// contains filtered or unexported fields
}

Iterator is a key-value trie iterator that traverses a Trie.

func NewIterator 

func NewIterator(it NodeIterator) *Iterator

NewIterator creates a new key-value iterator from a node iterator. Note that the value returned by the iterator is raw. If the content is encoded (e.g. storage value is RLP-encoded), it's caller's duty to decode it.

func (*Iterator) Next 

func (it *Iterator) Next(skipMiddleNode, descend, returnNotFoundChildNode bool) bool

Next moves the iterator forward one key-value entry.

func (*Iterator) Prove 

func (it *Iterator) Prove() [][]byte

Prove generates the Merkle proof for the leaf node the iterator is currently positioned on.

type LeafCallback 

type LeafCallback func(leaf []byte, parent common.Hash) error

LeafCallback is a callback type invoked when a trie operation reaches a leaf node. It's used by state sync and commit to allow handling external references between account and storage tries.

type MissingNodeError 

type MissingNodeError struct {
	NodeHash common.Hash // hash of the missing node
	Path     []byte      // hex-encoded path to the missing node
}

MissingNodeError is returned by the trie functions (TryGet, TryUpdate, TryDelete) in the case where a trie node is not present in the local database. It contains information necessary for retrieving the missing node.

func (*MissingNodeError) Error 

func (err *MissingNodeError) Error() string

type NodeIterator 

type NodeIterator interface {
	// Next moves the iterator to the next node. If the parameter is false, any child
	// nodes will be skipped.
	Next(bool, bool) bool

	// Error returns the error status of the iterator.
	Error() error

	// Hash returns the hash of the current node.
	Hash() common.Hash

	// Parent returns the hash of the parent of the current node. The hash may be the one
	// grandparent if the immediate parent is an internal node with no hash.
	Parent() common.Hash

	// Path returns the hex-encoded path to the current node.
	// Callers must not retain references to the return value after calling Next.
	// For leaf nodes, the last element of the path is the 'terminator symbol' 0x10.
	Path() []byte

	// Leaf returns true iff the current node is a leaf node.
	Leaf() bool

	// LeafKey returns the key of the leaf. The method panics if the iterator is not
	// positioned at a leaf. Callers must not retain references to the value after
	// calling Next.
	LeafKey() []byte

	// LeafBlob returns the content of the leaf. The method panics if the iterator
	// is not positioned at a leaf. Callers must not retain references to the value
	// after calling Next.
	LeafBlob() []byte

	// LeafProof returns the Merkle proof of the leaf. The method panics if the
	// iterator is not positioned at a leaf. Callers must not retain references
	// to the value after calling Next.
	LeafProof() [][]byte
}

NodeIterator is an iterator to traverse the trie pre-order.

func NewDifferenceIterator 

func NewDifferenceIterator(a, b NodeIterator) (NodeIterator, *int)

NewDifferenceIterator constructs a NodeIterator that iterates over elements in b that are not in a. Returns the iterator, and a pointer to an integer recording the number of nodes seen.

func NewUnionIterator 

func NewUnionIterator(iters []NodeIterator) (NodeIterator, *int)

NewUnionIterator constructs a NodeIterator that iterates over elements in the union of the provided NodeIterators. Returns the iterator, and a pointer to an integer recording the number of nodes visited.

type PrefixTrie 

type PrefixTrie struct {
	// contains filtered or unexported fields
}

PrefixTrie wraps a trie with key hashing. In a secure trie, all access operations hash the key using keccak256. This prevents calling code from creating long chains of nodes that increase the access time.

Contrary to a regular trie, a SecureTrie can only be created with New and must have an attached database. The database also stores the preimage of each key.

SecureTrie is not safe for concurrent use.

func NewPrefixTrie 

func NewPrefixTrie(root common.Hash, intermediateWriter *IntermediateWriter) (*PrefixTrie, error)

NewSecure creates a trie with an existing root node from a backing database and optional intermediate in-memory node pool.

If root is the zero hash or the sha3 hash of an empty string, the trie is initially empty. Otherwise, New will panic if iw is nil and returns MissingNodeError if the root node cannot be found.

Accessing the trie loads nodes from the database or node pool on demand. Loaded nodes are kept around until their 'cache generation' expires. A new cache generation is created by each call to Commit. cachelimit sets the number of past cache generations to keep.

func (*PrefixTrie) Commit 

func (t *PrefixTrie) Commit(onleaf LeafCallback) (root common.Hash, err error)

Commit writes all nodes and the secure hash pre-images to the trie's database. Nodes are stored with their sha3 hash as the key.

Committing flushes nodes from memory. Subsequent Get calls will load nodes from the database.

func (*PrefixTrie) Copy 

func (t *PrefixTrie) Copy() *PrefixTrie

Copy returns a copy of PrefixTrie.

func (*PrefixTrie) Delete 

func (t *PrefixTrie) Delete(key []byte)

Delete removes any existing value for key from the trie.

func (*PrefixTrie) Get 

func (t *PrefixTrie) Get(key []byte) []byte

Get returns the value for key stored in the trie. The value bytes must not be modified by the caller.

func (*PrefixTrie) GetKey 

func (t *PrefixTrie) GetKey(shaKey []byte) []byte

GetKey returns the sha3 preimage of a hashed key that was previously used to store a value.

func (*PrefixTrie) Hash 

func (t *PrefixTrie) Hash() common.Hash

Hash returns the root hash of PrefixTrie. It does not write to the database and can be used even if the trie doesn't have one.

func (*PrefixTrie) NodeIterator 

func (t *PrefixTrie) NodeIterator(start []byte) NodeIterator

NodeIterator returns an iterator that returns nodes of the underlying trie. Iteration starts at the key after the given start key.

func (*PrefixTrie) Prove 

func (t *PrefixTrie) Prove(key []byte, fromLevel uint, proofDb incdb.Database) error

Prove constructs a merkle proof for key. The result contains all encoded nodes on the path to the value at key. The value itself is also included in the last node and can be retrieved by verifying the proof.

If the trie does not contain a value for key, the returned proof contains all nodes of the longest existing prefix of the key (at least the root node), ending with the node that proves the absence of the key.

func (*PrefixTrie) TryDelete 

func (t *PrefixTrie) TryDelete(key []byte) error

TryDelete removes any existing value for key from the trie. If a node was not found in the database, a MissingNodeError is returned.

func (*PrefixTrie) TryGet 

func (t *PrefixTrie) TryGet(key []byte) ([]byte, error)

TryGet returns the value for key stored in the trie. The value bytes must not be modified by the caller. If a node was not found in the database, a MissingNodeError is returned.

func (*PrefixTrie) TryUpdate 

func (t *PrefixTrie) TryUpdate(key, value []byte) error

TryUpdate associates key with value in the trie. Subsequent calls to Get will return value. If value has length zero, any existing value is deleted from the trie and calls to Get will return nil.

The value bytes must not be modified by the caller while they are stored in the trie.

If a node was not found in the database, a MissingNodeError is returned.

func (*PrefixTrie) Update 

func (t *PrefixTrie) Update(key, value []byte)

Update associates key with value in the trie. Subsequent calls to Get will return value. If value has length zero, any existing value is deleted from the trie and calls to Get will return nil.

The value bytes must not be modified by the caller while they are stored in the trie.

type SecureTrie 

type SecureTrie struct {
	// contains filtered or unexported fields
}

SecureTrie wraps a trie with key hashing. In a secure trie, all access operations hash the key using keccak256. This prevents calling code from creating long chains of nodes that increase the access time.

Contrary to a regular trie, a SecureTrie can only be created with New and must have an attached database. The database also stores the preimage of each key.

SecureTrie is not safe for concurrent use.

func NewSecure 

func NewSecure(root common.Hash, intermediateWriter *IntermediateWriter) (*SecureTrie, error)

NewSecure creates a trie with an existing root node from a backing database and optional intermediate in-memory node pool.

If root is the zero hash or the sha3 hash of an empty string, the trie is initially empty. Otherwise, New will panic if iw is nil and returns MissingNodeError if the root node cannot be found.

Accessing the trie loads nodes from the database or node pool on demand. Loaded nodes are kept around until their 'cache generation' expires. A new cache generation is created by each call to Commit. cachelimit sets the number of past cache generations to keep.

func (*SecureTrie) Commit 

func (t *SecureTrie) Commit(onleaf LeafCallback) (root common.Hash, err error)

Commit writes all nodes and the secure hash pre-images to the trie's database. Nodes are stored with their sha3 hash as the key.

Committing flushes nodes from memory. Subsequent Get calls will load nodes from the database.

func (*SecureTrie) Copy 

func (t *SecureTrie) Copy() *SecureTrie

Copy returns a copy of SecureTrie.

func (*SecureTrie) Delete 

func (t *SecureTrie) Delete(key []byte)

Delete removes any existing value for key from the trie.

func (*SecureTrie) Get 

func (t *SecureTrie) Get(key []byte) []byte

Get returns the value for key stored in the trie. The value bytes must not be modified by the caller.

func (*SecureTrie) GetKey 

func (t *SecureTrie) GetKey(shaKey []byte) []byte

GetKey returns the sha3 preimage of a hashed key that was previously used to store a value.

func (*SecureTrie) Hash 

func (t *SecureTrie) Hash() common.Hash

Hash returns the root hash of SecureTrie. It does not write to the database and can be used even if the trie doesn't have one.

func (*SecureTrie) NodeIterator 

func (t *SecureTrie) NodeIterator(start []byte) NodeIterator

NodeIterator returns an iterator that returns nodes of the underlying trie. Iteration starts at the key after the given start key.

func (*SecureTrie) Prove 

func (t *SecureTrie) Prove(key []byte, fromLevel uint, proofDb incdb.Database) error

Prove constructs a merkle proof for key. The result contains all encoded nodes on the path to the value at key. The value itself is also included in the last node and can be retrieved by verifying the proof.

If the trie does not contain a value for key, the returned proof contains all nodes of the longest existing prefix of the key (at least the root node), ending with the node that proves the absence of the key.

func (*SecureTrie) TryDelete 

func (t *SecureTrie) TryDelete(key []byte) error

TryDelete removes any existing value for key from the trie. If a node was not found in the database, a MissingNodeError is returned.

func (*SecureTrie) TryGet 

func (t *SecureTrie) TryGet(key []byte) ([]byte, error)

TryGet returns the value for key stored in the trie. The value bytes must not be modified by the caller. If a node was not found in the database, a MissingNodeError is returned.

func (*SecureTrie) TryUpdate 

func (t *SecureTrie) TryUpdate(key, value []byte) error

TryUpdate associates key with value in the trie. Subsequent calls to Get will return value. If value has length zero, any existing value is deleted from the trie and calls to Get will return nil.

The value bytes must not be modified by the caller while they are stored in the trie.

If a node was not found in the database, a MissingNodeError is returned.

func (*SecureTrie) Update 

func (t *SecureTrie) Update(key, value []byte)

Update associates key with value in the trie. Subsequent calls to Get will return value. If value has length zero, any existing value is deleted from the trie and calls to Get will return nil.

The value bytes must not be modified by the caller while they are stored in the trie.

type StateBloom 

type StateBloom struct {
	// contains filtered or unexported fields
}

stateBloom is a bloom filter used during the state convesion(snapshot->state). The keys of all generated entries will be recorded here so that in the pruning stage the entries belong to the specific version can be avoided for deletion.

The false-positive is allowed here. The "false-positive" entries means they actually don't belong to the specific version but they are not deleted in the pruning. The downside of the false-positive allowance is we may leave some "dangling" nodes in the disk. But in practice the it's very unlike the dangling node is state root. So in theory this pruned state shouldn't be visited anymore. Another potential issue is for fast sync. If we do another fast sync upon the pruned database, it's problematic which will stop the expansion during the syncing.

After the entire state is generated, the bloom filter should be persisted into the disk. It indicates the whole generation procedure is finished.

func NewStateBloomFromDisk 

func NewStateBloomFromDisk(filename string) (*StateBloom, error)

NewStateBloomFromDisk loads the state bloom from the given file. In this case the assumption is held the bloom filter is complete.

func NewStateBloomWithSize 

func NewStateBloomWithSize(size uint64) (*StateBloom, error)

newStateBloomWithSize creates a brand new state bloom for state generation. The bloom filter will be created by the passing bloom filter size. According to the https://hur.st/bloomfilter/?n=600000000&p=&m=2048MB&k=4, the parameters are picked so that the false-positive rate for mainnet is low enough.

func (*StateBloom) Commit 

func (bloom *StateBloom) Commit(filename, tempname string) error

Commit flushes the bloom filter content into the disk and marks the bloom as complete.

func (*StateBloom) Contain 

func (bloom *StateBloom) Contain(key []byte) (bool, error)

Contain is the wrapper of the underlying contains function which reports whether the key is contained. - If it says yes, the key may be contained - If it says no, the key is definitely not contained.

func (*StateBloom) Delete 

func (bloom *StateBloom) Delete(key []byte) error

Delete removes the key from the key-value data store.

func (*StateBloom) Put 

func (bloom *StateBloom) Put(key []byte, value []byte) error

Put implements the KeyValueWriter interface. But here only the key is needed.

type Sync 

type Sync struct {
	// contains filtered or unexported fields
}

Sync is the main state trie synchronisation scheduler, which provides yet unknown trie hashes to retrieve, accepts node data associated with said hashes and reconstructs the trie step by step until all is done.

func NewSync 

func NewSync(root common.Hash, database incdb.Database, callback LeafCallback, bloom *SyncBloom) *Sync

NewSync creates a new trie data download scheduler.

func (*Sync) AddRawEntry 

func (s *Sync) AddRawEntry(hash common.Hash, depth int, parent common.Hash)

AddRawEntry schedules the direct retrieval of a state entry that should not be interpreted as a trie node, but rather accepted and stored into the database as is. This method's goal is to support misc state metadata retrievals (e.g. contract code).

func (*Sync) AddSubTrie 

func (s *Sync) AddSubTrie(root common.Hash, depth int, parent common.Hash, callback LeafCallback)

AddSubTrie registers a new trie to the sync code, rooted at the designated parent.

func (*Sync) Commit 

func (s *Sync) Commit(dbw incdb.Batch) error

Commit flushes the data stored in the internal membatch out to persistent storage, returning any occurred error.

func (*Sync) Missing 

func (s *Sync) Missing(max int) []common.Hash

Missing retrieves the known missing nodes from the trie for retrieval.

func (*Sync) Pending 

func (s *Sync) Pending() int

Pending returns the number of state entries currently pending for download.

func (*Sync) Process 

func (s *Sync) Process(results []SyncResult) (bool, int, error)

Process injects a batch of retrieved trie nodes data, returning if something was committed to the database and also the index of an entry if processing of it failed.

type SyncBloom 

type SyncBloom struct {
	// contains filtered or unexported fields
}

SyncBloom is a bloom filter used during fast sync to quickly decide if a trie node already exists on disk or not. It self populates from the provided disk database on creation in a background thread and will only start returning live results once that's finished.

func NewSyncBloom 

func NewSyncBloom(memory uint64, database incdb.Iteratee) *SyncBloom

NewSyncBloom creates a new bloom filter of the given size (in megabytes) and initializes it from the database. The bloom is hard coded to use 3 filters.

func (*SyncBloom) Add 

func (b *SyncBloom) Add(hash []byte)

Add inserts a new trie node hash into the bloom filter.

func (*SyncBloom) Close 

func (b *SyncBloom) Close() error

Close terminates any background initializer still running and releases all the memory allocated for the bloom.

func (*SyncBloom) Contains 

func (b *SyncBloom) Contains(hash []byte) bool

Contains tests if the bloom filter contains the given hash:

  • false: the bloom definitely does not contain hash
  • true: the bloom maybe contains hash

While the bloom is being initialized, any query will return true.

type SyncResult 

type SyncResult struct {
	Hash common.Hash // Hash of the originally unknown trie node
	Data []byte      // Data content of the retrieved node
}

SyncResult is a simple list to return missing nodes along with their request hashes.

type TLogger 

type TLogger struct {
	// contains filtered or unexported fields
}

func (*TLogger) Init 

func (tLogger *TLogger) Init(inst common.Logger)

type Trie 

type Trie struct {
	// contains filtered or unexported fields
}

Trie is a Merkle Patricia Trie. The zero value is an empty trie with no database. Use New to create a trie that sits on top of a database.

Trie is not safe for concurrent use.

func New 

func New(root common.Hash, intermediateWriter *IntermediateWriter) (*Trie, error)

New creates a trie with an existing root node from iw.

If root is the zero hash or the sha3 hash of an empty string, the trie is initially empty and does not require a database. Otherwise, New will panic if iw is nil and returns a MissingNodeError if root does not exist in the database. Accessing the trie loads nodes from iw on demand.

func (*Trie) Commit 

func (t *Trie) Commit(onleaf LeafCallback) (root common.Hash, err error)

Commit writes all nodes to the trie's memory database, tracking the internal and external (for account tries) references.

func (*Trie) Delete 

func (t *Trie) Delete(key []byte)

Delete removes any existing value for key from the trie.

func (*Trie) Get 

func (t *Trie) Get(key []byte) []byte

Get returns the value for key stored in the trie. The value bytes must not be modified by the caller.

func (*Trie) Hash 

func (t *Trie) Hash() common.Hash

Hash returns the root hash of the trie. It does not write to the database and can be used even if the trie doesn't have one.

func (*Trie) NodeIterator 

func (t *Trie) NodeIterator(start []byte) NodeIterator

NodeIterator returns an iterator that returns nodes of the trie. Iteration starts at the key after the given start key.

func (*Trie) Prove 

func (t *Trie) Prove(key []byte, fromLevel uint, proofDb incdb.Database) error

Prove constructs a merkle proof for key. The result contains all encoded nodes on the path to the value at key. The value itself is also included in the last node and can be retrieved by verifying the proof.

If the trie does not contain a value for key, the returned proof contains all nodes of the longest existing prefix of the key (at least the root node), ending with the node that proves the absence of the key.

func (*Trie) TryDelete 

func (t *Trie) TryDelete(key []byte) error

TryDelete removes any existing value for key from the trie. If a node was not found in the database, a MissingNodeError is returned.

func (*Trie) TryGet 

func (t *Trie) TryGet(key []byte) ([]byte, error)

TryGet returns the value for key stored in the trie. The value bytes must not be modified by the caller. If a node was not found in the database, a MissingNodeError is returned.

func (*Trie) TryUpdate 

func (t *Trie) TryUpdate(key, value []byte) error

TryUpdate associates key with value in the trie. Subsequent calls to Get will return value. If value has length zero, any existing value is deleted from the trie and calls to Get will return nil.

The value bytes must not be modified by the caller while they are stored in the trie.

If a node was not found in the database, a MissingNodeError is returned.

func (*Trie) Update 

func (t *Trie) Update(key, value []byte)

Update associates key with value in the trie. Subsequent calls to Get will return value. If value has length zero, any existing value is deleted from the trie and calls to Get will return nil.

The value bytes must not be modified by the caller while they are stored in the trie.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.