blockstoreutil

package
v1.3.0-rc2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 8, 2022 License: Apache-2.0, MIT Imports: 25 Imported by: 0

Documentation

Overview

blockstoreutil contains all the basic blockstore constructors used by lotus. Any blockstoreutil not ultimately constructed out of the building blocks in this package may not work properly.

  • This package correctly wraps blockstores with the IdBlockstore. This blockstore:
  • Filters out all puts for blocks with CIDs using the "identity" hash function.
  • Extracts inlined blocks from CIDs using the identity hash function and returns them on get/has, ignoring the contents of the blockstore.
  • In the future, this package may enforce additional restrictions on block sizes, CID validity, etc.

To make auditing for misuse of blockstores tractable, this package re-exports parts of the go-ipfs-blockstore package such that no other package needs to import it directly.

Index

Constants

View Source 
const (
	// FileIO is equivalent to badger/options.FileIO.
	FileIO = options.FileIO
	// MemoryMap is equivalent to badger/options.MemoryMap.
	MemoryMap = options.MemoryMap
	// LoadToRAM is equivalent to badger/options.LoadToRAM.
	LoadToRAM = options.LoadToRAM
)

aliases to mask badger dependencies.

Variables

View Source 
var (
	// ErrBlockstoreClosed is returned from blockstore operations after
	// the blockstore has been closed.
	ErrBlockstoreClosed = fmt.Errorf("badger blockstore closed")
)
View Source 
var ErrNotFound = blockstore.ErrNotFound

ErrNotFound is an error returned when a block is not found

View Source 
var NewGCBlockstore = blockstore.NewGCBlockstore
View Source 
var NewGCLocker = blockstore.NewGCLocker

Functions

func CopyBlockstore 

func CopyBlockstore(ctx context.Context, from, to Blockstore) error

func CopyParticial 

func CopyParticial(ctx context.Context, from, to Blockstore, root cid.Cid) error

Types

type AutobatchBlockstore added in v1.2.0 

type AutobatchBlockstore struct {
	// contains filtered or unexported fields
}

func NewAutobatch added in v1.2.0 

func NewAutobatch(ctx context.Context, backingBs Blockstore, bufferCapacity int) *AutobatchBlockstore

func (*AutobatchBlockstore) AllKeysChan added in v1.2.0 

func (bs *AutobatchBlockstore) AllKeysChan(ctx context.Context) (<-chan cid.Cid, error)

func (*AutobatchBlockstore) DeleteBlock added in v1.2.0 

func (bs *AutobatchBlockstore) DeleteBlock(context.Context, cid.Cid) error

func (*AutobatchBlockstore) DeleteMany added in v1.2.0 

func (bs *AutobatchBlockstore) DeleteMany(ctx context.Context, cids []cid.Cid) error

func (*AutobatchBlockstore) Flush added in v1.2.0 

func (bs *AutobatchBlockstore) Flush(ctx context.Context) error

caller must NOT hold stateLock

func (*AutobatchBlockstore) Get added in v1.2.0 

func (bs *AutobatchBlockstore) Get(ctx context.Context, c cid.Cid) (block.Block, error)

func (*AutobatchBlockstore) GetSize added in v1.2.0 

func (bs *AutobatchBlockstore) GetSize(ctx context.Context, c cid.Cid) (int, error)

func (*AutobatchBlockstore) Has added in v1.2.0 

func (bs *AutobatchBlockstore) Has(ctx context.Context, c cid.Cid) (bool, error)

func (*AutobatchBlockstore) HashOnRead added in v1.2.0 

func (bs *AutobatchBlockstore) HashOnRead(enabled bool)

func (*AutobatchBlockstore) Put added in v1.2.0 

func (bs *AutobatchBlockstore) Put(ctx context.Context, blk block.Block) error

func (*AutobatchBlockstore) PutMany added in v1.2.0 

func (bs *AutobatchBlockstore) PutMany(ctx context.Context, blks []block.Block) error

func (*AutobatchBlockstore) Shutdown added in v1.2.0 

func (bs *AutobatchBlockstore) Shutdown(ctx context.Context) error

func (*AutobatchBlockstore) View added in v1.2.0 

func (bs *AutobatchBlockstore) View(ctx context.Context, cid cid.Cid, callback func([]byte) error) error

type BadgerBlockstore 

type BadgerBlockstore struct {
	DB *badger.DB
	// contains filtered or unexported fields
}

blockstore is a badger-backed IPLD blockstore.

NOTE: once Close() is called, methods will try their best to return ErrBlockstoreClosed. This will guaranteed to happen for all subsequent operation calls after Close() has returned, but it may not happen for operations in progress. Those are likely to fail with a different error.

func Open 

func Open(opts Options) (*BadgerBlockstore, error)

Open creates a new badger-backed blockstore, with the supplied options.

func (*BadgerBlockstore) AllKeysChan 

func (b *BadgerBlockstore) AllKeysChan(ctx context.Context) (<-chan cid.Cid, error)

AllKeysChan implements blockstore.AllKeysChan.

func (*BadgerBlockstore) Close 

func (b *BadgerBlockstore) Close() error

Close closes the store. If the store has already been closed, this noops and returns an error, even if the first closure resulted in error.

func (*BadgerBlockstore) ConvertKey 

func (b *BadgerBlockstore) ConvertKey(cid cid.Cid) datastore.Key

func (*BadgerBlockstore) DeleteBlock 

func (b *BadgerBlockstore) DeleteBlock(ctx context.Context, cid cid.Cid) error

DeleteBlock implements blockstore.DeleteBlock.

func (*BadgerBlockstore) DeleteMany added in v1.6.0 

func (b *BadgerBlockstore) DeleteMany(ctx context.Context, cids []cid.Cid) error

func (*BadgerBlockstore) Get 

func (b *BadgerBlockstore) Get(ctx context.Context, cid cid.Cid) (blocks.Block, error)

Get implements blockstore.Get.

func (*BadgerBlockstore) GetSize 

func (b *BadgerBlockstore) GetSize(ctx context.Context, cid cid.Cid) (int, error)

GetSize implements blockstore.GetSize.

func (*BadgerBlockstore) Has 

func (b *BadgerBlockstore) Has(ctx context.Context, cid cid.Cid) (bool, error)

Has implements blockstore.Has.

func (*BadgerBlockstore) HashOnRead 

func (b *BadgerBlockstore) HashOnRead(_ bool)

HashOnRead implements blockstore.HashOnRead. It is not supported by this blockstore.

func (*BadgerBlockstore) Put 

func (b *BadgerBlockstore) Put(ctx context.Context, block blocks.Block) error

Put implements blockstore.Put.

func (*BadgerBlockstore) PutMany 

func (b *BadgerBlockstore) PutMany(ctx context.Context, blks []blocks.Block) error

PutMany implements blockstore.PutMany.

func (*BadgerBlockstore) ReadonlyDatastore 

func (b *BadgerBlockstore) ReadonlyDatastore() *TxBlockstore

func (*BadgerBlockstore) View 

func (b *BadgerBlockstore) View(ctx context.Context, cid cid.Cid, fn func([]byte) error) error

View implements blockstore.Viewer, which leverages zero-copy read-only access to values.

type BatchDeleter added in v1.6.0 

type BatchDeleter interface {
	DeleteMany(ctx context.Context, cids []cid.Cid) error
}

type Blockstore 

type Blockstore interface {
	blockstore.Blockstore
	blockstore.Viewer
	BatchDeleter
}

Blockstore is the blockstore interface used by Lotus. It is the union of the basic go-ipfs blockstore, with other capabilities required by Lotus, e.g. View or Sync.

func Adapt added in v1.6.0 

func Adapt(bs blockstore.Blockstore) Blockstore

Adapt adapts a standard blockstore to a Lotus blockstore by enriching it with the extra methods that Lotus requires (e.g. View, Sync).

View proxies over to Get and calls the callback with the value supplied by Get. Sync noops.

func CachedBlockstore 

func CachedBlockstore(ctx context.Context, bs blockstore.Blockstore, opts CacheOpts) (Blockstore, error)

func NewBlockstore 

func NewBlockstore(dstore ds.Batching) Blockstore

NewBlockstore creates a new blockstore wrapped by the given datastore.

func WrapIDStore 

func WrapIDStore(bstore blockstore.Blockstore) Blockstore

WrapIDStore wraps the underlying blockstore in an "identity" blockstore.

type BufferedBS 

type BufferedBS struct {
	// contains filtered or unexported fields
}

func NewBufferedBstore 

func NewBufferedBstore(base Blockstore) *BufferedBS

func NewTieredBstore 

func NewTieredBstore(r Blockstore, w Blockstore) *BufferedBS

func (*BufferedBS) AllKeysChan 

func (bs *BufferedBS) AllKeysChan(ctx context.Context) (<-chan cid.Cid, error)

func (*BufferedBS) DeleteBlock 

func (bs *BufferedBS) DeleteBlock(ctx context.Context, c cid.Cid) error

func (*BufferedBS) DeleteMany added in v1.6.0 

func (bs *BufferedBS) DeleteMany(ctx context.Context, cids []cid.Cid) error

func (*BufferedBS) Get 

func (bs *BufferedBS) Get(ctx context.Context, c cid.Cid) (block.Block, error)

func (*BufferedBS) GetSize 

func (bs *BufferedBS) GetSize(ctx context.Context, c cid.Cid) (int, error)

func (*BufferedBS) Has 

func (bs *BufferedBS) Has(ctx context.Context, c cid.Cid) (bool, error)

func (*BufferedBS) HashOnRead 

func (bs *BufferedBS) HashOnRead(hor bool)

func (*BufferedBS) Put 

func (bs *BufferedBS) Put(ctx context.Context, blk block.Block) error

func (*BufferedBS) PutMany 

func (bs *BufferedBS) PutMany(ctx context.Context, blks []block.Block) error

func (*BufferedBS) Read 

func (bs *BufferedBS) Read() Blockstore

func (*BufferedBS) View 

func (bs *BufferedBS) View(ctx context.Context, c cid.Cid, callback func([]byte) error) error

func (*BufferedBS) Write 

func (bs *BufferedBS) Write() Blockstore

type CacheOpts 

type CacheOpts = blockstore.CacheOpts

func DefaultCacheOpts 

func DefaultCacheOpts() CacheOpts

type GCBlockstore 

type GCBlockstore = blockstore.GCBlockstore

type GCLocker 

type GCLocker = blockstore.GCLocker

type IBlockCache 

type IBlockCache interface {
	Get(key string) (value interface{}, ok bool)
	Remove(key string)
	Add(key string, value interface{})
	AddWithExpire(key string, value interface{}, dur time.Duration)
}

type LruCache 

type LruCache struct {
	// contains filtered or unexported fields
}

func NewLruCache 

func NewLruCache(size int) *LruCache

func (LruCache) Add 

func (l LruCache) Add(key string, value interface{})

func (LruCache) AddWithExpire 

func (l LruCache) AddWithExpire(key string, value interface{}, dur time.Duration)

func (LruCache) Get 

func (l LruCache) Get(key string) (interface{}, bool)

func (LruCache) Remove 

func (l LruCache) Remove(key string)

type MemBlockstore added in v1.2.0 

type MemBlockstore map[cid.Cid]blocks.Block

MemBlockstore is a terminal blockstore that keeps blocks in memory.

func NewMemory added in v1.2.0 

func NewMemory() MemBlockstore

NewMemory returns a temporary memory-backed blockstore.

func NewTemporary 

func NewTemporary() MemBlockstore

NewTemporary returns a temporary blockstore.

func (MemBlockstore) AllKeysChan added in v1.2.0 

func (m MemBlockstore) AllKeysChan(ctx context.Context) (<-chan cid.Cid, error)

AllKeysChan returns a channel from which the CIDs in the Blockstore can be read. It should respect the given context, closing the channel if it becomes Done.

func (MemBlockstore) DeleteBlock added in v1.2.0 

func (m MemBlockstore) DeleteBlock(ctx context.Context, k cid.Cid) error

func (MemBlockstore) DeleteMany added in v1.2.0 

func (m MemBlockstore) DeleteMany(ctx context.Context, ks []cid.Cid) error

func (MemBlockstore) Get added in v1.2.0 

func (m MemBlockstore) Get(ctx context.Context, k cid.Cid) (blocks.Block, error)

func (MemBlockstore) GetSize added in v1.2.0 

func (m MemBlockstore) GetSize(ctx context.Context, k cid.Cid) (int, error)

GetSize returns the CIDs mapped BlockSize

func (MemBlockstore) Has added in v1.2.0 

func (m MemBlockstore) Has(ctx context.Context, k cid.Cid) (bool, error)

func (MemBlockstore) HashOnRead added in v1.2.0 

func (m MemBlockstore) HashOnRead(enabled bool)

HashOnRead specifies if every read block should be rehashed to make sure it matches its CID.

func (MemBlockstore) Put added in v1.2.0 

func (m MemBlockstore) Put(ctx context.Context, b blocks.Block) error

Put puts a given block to the underlying datastore

func (MemBlockstore) PutMany added in v1.2.0 

func (m MemBlockstore) PutMany(ctx context.Context, bs []blocks.Block) error

PutMany puts a slice of blocks at the same time using batching capabilities of the underlying datastore whenever possible.

func (MemBlockstore) View added in v1.2.0 

func (m MemBlockstore) View(ctx context.Context, k cid.Cid, callback func([]byte) error) error

type Options 

type Options struct {
	badger.Options

	// Prefix is an optional prefix to prepend to keys. Default: "".
	Prefix string
}

Options embeds the badger options themselves, and augments them with blockstore-specific options.

func BadgerBlockstoreOptions 

func BadgerBlockstoreOptions(path string, readonly bool) (Options, error)

BadgerBlockstoreOptions returns the badger options to apply for the provided domain.

func DefaultOptions 

func DefaultOptions(path string) Options

type Set 

type Set struct {
	// contains filtered or unexported fields
}

Set is a implementation of a set of Cids, that is, a structure to which holds a single copy of every Cids that is added to it.

func NewSet 

func NewSet() *Set

NewSet initializes and returns a new Set.

func (*Set) Add 

func (s *Set) Add(c cid.Cid)

Add puts a Cid in the Set.

func (*Set) ForEach 

func (s *Set) ForEach(f func(c cid.Cid) error) error

ForEach allows to run a custom function on each Cid in the set.

func (*Set) Has 

func (s *Set) Has(c cid.Cid) bool

Has returns if the Set contains a given Cid.

func (*Set) Keys 

func (s *Set) Keys() []cid.Cid

Keys returns the Cids in the set.

func (*Set) Len 

func (s *Set) Len() int

Len returns how many elements the Set has.

func (*Set) Remove 

func (s *Set) Remove(c cid.Cid)

Remove deletes a Cid from the Set.

func (*Set) Visit 

func (s *Set) Visit(c cid.Cid) bool

Visit adds a Cid to the set only if it is not in it already.

type SyncStore 

type SyncStore struct {
	// contains filtered or unexported fields
}

func NewTemporarySync 

func NewTemporarySync() *SyncStore

NewTemporarySync returns a thread-safe temporary blockstore.

func (*SyncStore) AllKeysChan 

func (m *SyncStore) AllKeysChan(ctx context.Context) (<-chan cid.Cid, error)

AllKeysChan returns a channel from which the CIDs in the blockstore can be read. It should respect the given context, closing the channel if it becomes Done.

func (*SyncStore) DeleteBlock 

func (m *SyncStore) DeleteBlock(ctx context.Context, k cid.Cid) error

func (*SyncStore) DeleteMany added in v1.6.0 

func (m *SyncStore) DeleteMany(ctx context.Context, cids []cid.Cid) error

func (*SyncStore) Get 

func (m *SyncStore) Get(ctx context.Context, k cid.Cid) (blocks.Block, error)

func (*SyncStore) GetSize 

func (m *SyncStore) GetSize(ctx context.Context, k cid.Cid) (int, error)

GetSize returns the CIDs mapped BlockSize

func (*SyncStore) Has 

func (m *SyncStore) Has(ctx context.Context, k cid.Cid) (bool, error)

func (*SyncStore) HashOnRead 

func (m *SyncStore) HashOnRead(enabled bool)

HashOnRead specifies if every read block should be rehashed to make sure it matches its CID.

func (*SyncStore) Put 

func (m *SyncStore) Put(ctx context.Context, b blocks.Block) error

Put puts a given block to the underlying datastore

func (*SyncStore) PutMany 

func (m *SyncStore) PutMany(ctx context.Context, bs []blocks.Block) error

PutMany puts a slice of blocks at the same time using batching capabilities of the underlying datastore whenever possible.

func (*SyncStore) View added in v1.6.0 

func (m *SyncStore) View(ctx context.Context, cid cid.Cid, callback func([]byte) error) error

type TimeCache 

type TimeCache struct {
	// contains filtered or unexported fields
}

func NewTimeCache 

func NewTimeCache(expireTime, cleanTime time.Duration) *TimeCache

func (TimeCache) Add 

func (timeCache TimeCache) Add(key string, value interface{})

func (TimeCache) AddWithExpire 

func (timeCache TimeCache) AddWithExpire(key string, value interface{}, dur time.Duration)

func (TimeCache) Get 

func (timeCache TimeCache) Get(key string) (interface{}, bool)

func (TimeCache) Remove 

func (timeCache TimeCache) Remove(key string)

type TxBlockstore 

type TxBlockstore struct {
	// contains filtered or unexported fields
}

func (*TxBlockstore) AllKeysChan 

func (txBlockstore *TxBlockstore) AllKeysChan(ctx context.Context) (<-chan cid.Cid, error)

func (*TxBlockstore) ConvertKey 

func (txBlockstore *TxBlockstore) ConvertKey(cid cid.Cid) datastore.Key

func (*TxBlockstore) DeleteBlock 

func (txBlockstore *TxBlockstore) DeleteBlock(ctx context.Context, cid cid.Cid) error

func (*TxBlockstore) DeleteMany added in v1.6.0 

func (txBlockstore *TxBlockstore) DeleteMany(ctx context.Context, cids []cid.Cid) error

func (*TxBlockstore) Get 

func (txBlockstore *TxBlockstore) Get(ctx context.Context, cid cid.Cid) (blocks.Block, error)

func (*TxBlockstore) GetSize 

func (txBlockstore *TxBlockstore) GetSize(ctx context.Context, cid cid.Cid) (int, error)

func (*TxBlockstore) Has 

func (txBlockstore *TxBlockstore) Has(ctx context.Context, cid cid.Cid) (bool, error)

func (*TxBlockstore) HashOnRead 

func (txBlockstore *TxBlockstore) HashOnRead(enabled bool)

func (*TxBlockstore) Put 

func (txBlockstore *TxBlockstore) Put(ctx context.Context, block blocks.Block) error

func (*TxBlockstore) PutMany 

func (txBlockstore *TxBlockstore) PutMany(ctx context.Context, blocks []blocks.Block) error

func (*TxBlockstore) View added in v1.6.0 

func (txBlockstore *TxBlockstore) View(ctx context.Context, cid cid.Cid, callback func([]byte) error) error

type Viewer 

type Viewer = blockstore.Viewer

Alias so other packages don't have to import go-ipfs-blockstore type Blockstore = blockstore.Blockstore

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.