ipfsethdb

package module
v5.1.0-alpha Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 23, 2024 License: AGPL-3.0 Imports: 14 Imported by: 1

README

ipfs-ethdb

Go Report Card

go-ethereum ethdb interfaces for Ethereum state data stored in IPFS

Background

Go-ethereum defines a number of interfaces in the ethdb package for interfacing with a state database. These interfaces are used to build higher-level types such as the trie.Database which are used to perform the bulk of state related needs.

Ethereum data can be stored on IPFS, standard codecs for Etheruem data are defined in the go-cid library. Using our statediffing geth client it is feasible to extract every single state and storage node and publish it to IPFS.

Geth stores state data in leveldb as key-value pairs between the keccak256 hash of the rlp-encoded object and the rlp-encoded object. Ethereum data on IPFS is also stored as key-value pairs with the value being the rlp-encoded byte value for the object, but the key is a derivative of the keccak256 hash rather than the hash itself. This library provides ethdb interfaces for Ethereum data on IPFS by handling the conversion of a keccak256 hash to its multihash-derived key.

Usage

To use this module import it and build an ethdb interface around an instance of a Go IPFS blockservice, you can then employ it as you would the usual leveldb or memorydb ethdbs with some exceptions: the AncientReader, AncientWriter, Compacter, and Iteratee/Iterator interfaces are not functionally complete.

Ancient data does not currently have a representation on IPFS, and recapitulation of the database key iterator and compacter is complicated since go-ethereum types that use this interface expect the iterator and compacter to operate over keccak256 hash key ranges, whereas the keys for Ethereum data on IPFS are derived from that hash but not the hash itself.

Iteratee interface is used in Geth for various tests, in trie/sync_bloom.go (for fast sync), rawdb.InspectDatabase, and the new (1.9.15) core/state/snapshot features; Ancient interfaces are used for Ancient/frozen data operations (e.g. rawdb/table.go); and Compacter is used in core/state/snapshot, rawdb/table.go, chaincmd.go, and the private debug api.

Outside of these primarily auxiliary capabilities, this package satisfies the interfaces required for many state operations using Ethereum data on IPFS.

e.g.

go-ethereum trie.NodeIterator and state.NodeIterator can be constructed from the ethdb.KeyValueStore and ethdb.Database interfaces, respectively:

package main

import (
    "context"

    "github.com/ethereum/go-ethereum/common"
    "github.com/ethereum/go-ethereum/core/state"
    "github.com/ethereum/go-ethereum/trie"
    "github.com/ipfs/boxo/blockservice"
    "github.com/ipfs/go-ipfs/core"
    "github.com/ipfs/go-ipfs/repo/fsrepo"
    "github.com/jmoiron/sqlx"
    "github.com/cerc-io/ipfs-ethdb/v5"
)

func main() {
    // errors are ignored for brevity
    ipfsPath := "~/.ipfs"
    r, _ := fsrepo.Open(ipfsPath)
    ctx := context.Background()
    cfg := &core.BuildCfg{
        Online: false,
        Repo:   r,
    }
    ipfsNode, _ := core.NewNode(ctx, cfg)
    kvs := ipfsethdb.NewKeyValueStore(ipfsNode.Blocks)
    trieDB := trie.NewDatabase(kvs)
    t, _ := trie.New(common.Hash{}, trieDB)
    trieNodeIterator := t.NodeIterator([]byte{})
    // do stuff with trie node iterator

    database := ipfsethdb.NewDatabase(ipfsNode.Blocks)
    stateDatabase := state.NewDatabase(database)
    stateDB, _ := state.New(common.Hash{}, stateDatabase, nil)
    stateDBNodeIterator := state.NewNodeIterator(stateDB)
    // do stuff with the statedb node iterator
}

Types are also available for interfacing directly with the data stored in an IPFS-backing Postgres database

Maintainers

@vulcanize @AFDudley @i-norden

Contributing

Contributions are welcome!

VulcanizeDB follows the Contributor Covenant Code of Conduct.

License

AGPL-3.0 © Vulcanize Inc

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	EvictionWarningErr = errors.New("warn: batch has exceeded capacity, data has been evicted")
)

Functions

func Keccak256ToCid 

func Keccak256ToCid(h []byte, codec uint64) (cid.Cid, error)

Keccak256ToCid takes a keccak256 hash and returns its cid v1 using the provided codec.

func NewBatch 

func NewBatch(bs blockservice.BlockService, capacity int) (ethdb.Batch, error)

NewBatch returns a ethdb.Batch interface for IPFS

func NewBlock 

func NewBlock(key, value []byte) (blocks.Block, error)

NewBlock takes a keccak256 hash key and the rlp []byte value it was derived from and creates an ipfs block object

func NewDatabase 

func NewDatabase(bs blockservice.BlockService) ethdb.Database

NewDatabase returns a ethdb.Database interface for IPFS

func NewIterator 

func NewIterator(start, prefix []byte, bs blockservice.BlockService) ethdb.Iterator

NewIterator returns an ethdb.Iterator interface for PG-IPFS

func NewKeyValueStore 

func NewKeyValueStore(bs blockservice.BlockService) ethdb.KeyValueStore

NewKeyValueStore returns a ethdb.KeyValueStore interface for IPFS

func NewMockBlockservice 

func NewMockBlockservice() blockservice.BlockService

Types

type Batch 

type Batch struct {
	// contains filtered or unexported fields
}

Batch is the type that satisfies the ethdb.Batch interface for IPFS Ethereum data using the ipfs blockservice interface This is ipfs-backing-datastore agnostic but must operate through a configured ipfs node (and so is subject to lockfile contention with e.g. an ipfs daemon) If blockservice block exchange is configured the blockservice can fetch data that are missing locally from IPFS peers

func (*Batch) Delete 

func (b *Batch) Delete(key []byte) (err error)

Delete satisfies the ethdb.Batch interface Delete removes the key from the key-value data store

func (*Batch) Put 

func (b *Batch) Put(key []byte, value []byte) (err error)

Put satisfies the ethdb.Batch interface Put inserts the given value into the key-value data store Key is expected to be the keccak256 hash of value Returns an error when batch capacity has been exceeded and data was evicted It is up to ensure they do not exceed capacity The alternative is to check the cache len vs its capacity before inserting but this adds additional overhead to every Put/Delete

func (*Batch) Replay 

func (b *Batch) Replay(w ethdb.KeyValueWriter) error

Replay satisfies the ethdb.Batch interface Replay replays the batch contents

func (*Batch) Reset 

func (b *Batch) Reset()

Reset satisfies the ethdb.Batch interface Reset resets the batch for reuse This should be called after every write

func (*Batch) ValueSize 

func (b *Batch) ValueSize() int

ValueSize satisfies the ethdb.Batch interface ValueSize retrieves the amount of data queued up for writing The returned value is the total byte length of all data queued to write

func (*Batch) Write 

func (b *Batch) Write() error

Write satisfies the ethdb.Batch interface Write flushes any accumulated data to disk

type Database 

type Database struct {
	// contains filtered or unexported fields
}

Database is the type that satisfies the ethdb.Database and ethdb.KeyValueStore interfaces for IPFS Ethereum data This is ipfs-backing-datastore agnostic but must operate through a configured ipfs node (and so is subject to lockfile contention with e.g. an ipfs daemon) If blockservice block exchange is configured the blockservice can fetch data that are missing locally from IPFS peers

func (*Database) Ancient 

func (d *Database) Ancient(kind string, number uint64) ([]byte, error)

Ancient satisfies the ethdb.AncientReader interface Ancient retrieves an ancient binary blob from the append-only immutable files

func (*Database) AncientDatadir 

func (d *Database) AncientDatadir() (string, error)

AncientDatadir returns an error as we don't have a backing chain freezer.

func (*Database) AncientRange 

func (d *Database) AncientRange(kind string, start, count, maxBytes uint64) ([][]byte, error)

AncientRange retrieves all the items in a range, starting from the index 'start'. It will return

  • at most 'count' items,
  • at least 1 item (even if exceeding the maxBytes), but will otherwise return as many items as fit into maxBytes.

func (*Database) AncientSize 

func (d *Database) AncientSize(kind string) (uint64, error)

AncientSize satisfies the ethdb.AncientReader interface AncientSize returns the ancient size of the specified category

func (*Database) Ancients 

func (d *Database) Ancients() (uint64, error)

Ancients satisfies the ethdb.AncientReader interface Ancients returns the ancient item numbers in the ancient store

func (*Database) Close 

func (d *Database) Close() error

Close satisfies the io.Closer interface Close closes the db connection

func (*Database) Compact 

func (d *Database) Compact(start []byte, limit []byte) error

Compact satisfies the ethdb.Compacter interface Compact flattens the underlying data store for the given key range

func (*Database) Delete 

func (d *Database) Delete(key []byte) error

Delete satisfies the ethdb.KeyValueWriter interface Delete removes the key from the key-value data store

func (*Database) Get 

func (d *Database) Get(key []byte) ([]byte, error)

Get satisfies the ethdb.KeyValueReader interface Get retrieves the given key if it's present in the key-value data store

func (*Database) Has 

func (d *Database) Has(key []byte) (bool, error)

Has satisfies the ethdb.KeyValueReader interface Has retrieves if a key is present in the key-value data store This only operates on the local blockstore not through the exchange

func (*Database) HasAncient 

func (d *Database) HasAncient(kind string, number uint64) (bool, error)

HasAncient satisfies the ethdb.AncientReader interface HasAncient returns an indicator whether the specified data exists in the ancient store

func (*Database) MigrateTable 

func (d *Database) MigrateTable(string, func([]byte) ([]byte, error)) error

MigrateTable satisfies the ethdb.AncientWriter interface. MigrateTable processes and migrates entries of a given table to a new format.

func (*Database) ModifyAncients 

func (d *Database) ModifyAncients(f func(ethdb.AncientWriteOp) error) (int64, error)

func (*Database) NewBatch 

func (d *Database) NewBatch() ethdb.Batch

NewBatch satisfies the ethdb.Batcher interface NewBatch creates a write-only database that buffers changes to its host db until a final write is called

func (*Database) NewBatchWithSize 

func (d *Database) NewBatchWithSize(size int) ethdb.Batch

NewBatchWithSize satisfies the ethdb.Batcher interface. NewBatchWithSize creates a write-only database batch with pre-allocated buffer.

func (*Database) NewIterator 

func (d *Database) NewIterator(prefix []byte, start []byte) ethdb.Iterator

NewIterator satisfies the ethdb.Iteratee interface it creates a binary-alphabetical iterator over a subset of database content with a particular key prefix, starting at a particular initial key (or after, if it does not exist).

Note: This method assumes that the prefix is NOT part of the start, so there's no need for the caller to prepend the prefix to the start

func (*Database) NewSnapshot 

func (d *Database) NewSnapshot() (ethdb.Snapshot, error)

NewSnapshot satisfies the ethdb.Snapshotter interface. NewSnapshot creates a database snapshot based on the current state.

func (*Database) Put 

func (d *Database) Put(key []byte, value []byte) error

Put satisfies the ethdb.KeyValueWriter interface Put inserts the given value into the key-value data store Key is expected to be the keccak256 hash of value

func (*Database) ReadAncients 

func (d *Database) ReadAncients(fn func(ethdb.AncientReaderOp) error) (err error)

ReadAncients applies the provided AncientReader function

func (*Database) Stat 

func (d *Database) Stat(property string) (string, error)

Stat satisfies the ethdb.Stater interface Stat returns a particular internal stat of the database

func (*Database) Sync 

func (d *Database) Sync() error

Sync satisfies the ethdb.AncientWriter interface Sync flushes all in-memory ancient store data to disk

func (*Database) Tail 

func (d *Database) Tail() (uint64, error)

Tail satisfies the ethdb.AncientReader interface. Tail returns the number of first stored item in the freezer.

func (*Database) TruncateHead 

func (d *Database) TruncateHead(n uint64) (uint64, error)

TruncateHead satisfies the ethdb.AncientWriter interface. TruncateHead discards all but the first n ancient data from the ancient store.

func (*Database) TruncateTail 

func (d *Database) TruncateTail(n uint64) (uint64, error)

TruncateTail satisfies the ethdb.AncientWriter interface. TruncateTail discards the first n ancient data from the ancient store.

type DatabaseProperty 

type DatabaseProperty int

DatabaseProperty enum type 

const (
	Unknown DatabaseProperty = iota
	ExchangeOnline
)

func DatabasePropertyFromString 

func DatabasePropertyFromString(property string) (DatabaseProperty, error)

DatabasePropertyFromString helper function

type Iterator 

type Iterator struct {
	// contains filtered or unexported fields
}

Iterator is the type that satisfies the ethdb.Iterator interface for IPFS Ethereum data Iteratee interface is used in Geth for various tests, trie/sync_bloom.go (for fast sync), rawdb.InspectDatabase, and the new core/state/snapshot features. This should not be confused with trie.NodeIterator or state.NodeIteraor (which can be constructed from the ethdb.KeyValueStoreand ethdb.Database interfaces)

func (*Iterator) Error 

func (i *Iterator) Error() error

Error satisfies the ethdb.Iterator interface Error returns any accumulated error Exhausting all the key/value pairs is not considered to be an error

func (*Iterator) Key 

func (i *Iterator) Key() []byte

Key satisfies the ethdb.Iterator interface Key returns the key of the current key/value pair, or nil if done The caller should not modify the contents of the returned slice and its contents may change on the next call to Next

func (*Iterator) Next 

func (i *Iterator) Next() bool

Next satisfies the ethdb.Iterator interface Next moves the iterator to the next key/value pair It returns whether the iterator is exhausted

func (*Iterator) Release 

func (i *Iterator) Release()

Release satisfies the ethdb.Iterator interface Release releases associated resources Release should always succeed and can be called multiple times without causing error

func (*Iterator) Value 

func (i *Iterator) Value() []byte

Value satisfies the ethdb.Iterator interface Value returns the value of the current key/value pair, or nil if done The caller should not modify the contents of the returned slice and its contents may change on the next call to Next

type MockBlockservice 

type MockBlockservice struct {
	// contains filtered or unexported fields
}

func (*MockBlockservice) AddBlock 

func (mbs *MockBlockservice) AddBlock(ctx context.Context, b blocks.Block) error

func (*MockBlockservice) AddBlocks 

func (mbs *MockBlockservice) AddBlocks(ctx context.Context, bs []blocks.Block) error

func (*MockBlockservice) Blockstore 

func (mbs *MockBlockservice) Blockstore() blockstore.Blockstore

func (*MockBlockservice) Close 

func (mbs *MockBlockservice) Close() error

func (*MockBlockservice) DeleteBlock 

func (mbs *MockBlockservice) DeleteBlock(ctx context.Context, c cid.Cid) error

func (*MockBlockservice) Exchange 

func (mbs *MockBlockservice) Exchange() exchange.Interface

func (*MockBlockservice) GetBlock 

func (mbs *MockBlockservice) GetBlock(ctx context.Context, c cid.Cid) (blocks.Block, error)

func (*MockBlockservice) GetBlocks 

func (mbs *MockBlockservice) GetBlocks(ctx context.Context, cs []cid.Cid) <-chan blocks.Block

func (*MockBlockservice) SetError 

func (mbs *MockBlockservice) SetError(err error)

type MockBlockstore 

type MockBlockstore struct {
	// contains filtered or unexported fields
}

func (*MockBlockstore) AllKeysChan 

func (mbs *MockBlockstore) AllKeysChan(ctx context.Context) (<-chan cid.Cid, error)

func (*MockBlockstore) DeleteBlock 

func (mbs *MockBlockstore) DeleteBlock(ctx context.Context, c cid.Cid) error

func (*MockBlockstore) Get 

func (mbs *MockBlockstore) Get(ctx context.Context, c cid.Cid) (blocks.Block, error)

func (*MockBlockstore) GetSize 

func (mbs *MockBlockstore) GetSize(ctx context.Context, c cid.Cid) (int, error)

func (*MockBlockstore) Has 

func (mbs *MockBlockstore) Has(ctx context.Context, c cid.Cid) (bool, error)

func (*MockBlockstore) HashOnRead 

func (mbs *MockBlockstore) HashOnRead(enabled bool)

func (*MockBlockstore) Put 

func (mbs *MockBlockstore) Put(ctx context.Context, b blocks.Block) error

func (*MockBlockstore) PutMany 

func (mbs *MockBlockstore) PutMany(ctx context.Context, bs []blocks.Block) error

func (*MockBlockstore) SetError 

func (mbs *MockBlockstore) SetError(err error)

Source Files

View all Source files

Directories

Path Synopsis
postgres
shared
v0
v1

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.