splitstore

package
v1.13.2-rc1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 1, 2021 License: Apache-2.0, MIT Imports: 35 Imported by: 2

README

SplitStore: An actively scalable blockstore for the Filecoin chain

The SplitStore was first introduced in lotus v1.5.1, as an experiment in reducing the performance impact of large blockstores.

With lotus v1.11.1, we introduce the next iteration in design and implementation, which we call SplitStore v1.

The new design (see #6474 evolves the splitstore to be a freestanding compacting blockstore that allows us to keep a small (60-100GB) working set in a hot blockstore and reliably archive out of scope objects in a coldstore. The coldstore can also be a discard store, whereby out of scope objects are discarded or a regular badger blockstore (the default), which can be periodically garbage collected according to configurable user retention policies.

To enable the splitstore, edit .lotus/config.toml and add the following:

[Chainstore]
  EnableSplitstore = true

If you intend to use the discard coldstore, your also need to add the following:

  [Chainstore.Splitstore]
    ColdStoreType = "discard"

In general you should not have to use the discard store, unless you are running a network assistive node (like a bootstrapper or booster) or have very constrained hardware with not enough disk space to maintain a coldstore, even with garbage collection. It is also appropriate for small nodes that are simply watching the chain.

Warning: Using the discard store for a general purpose node is discouraged, unless you really know what you are doing. Use it at your own risk.

Configuration Options

These are options in the [Chainstore.Splitstore] section of the configuration:

  • HotStoreType -- specifies the type of hotstore to use. The only currently supported option is "badger".
  • ColdStoreType -- specifies the type of coldstore to use. The default value is "universal", which will use the initial monolith blockstore as the coldstore. The other possible value is "discard", as outlined above, which is specialized for running without a coldstore. Note that the discard store wraps the initial monolith blockstore and discards writes; this is necessary to support syncing from a snapshot.
  • MarkSetType -- specifies the type of markset to use during compaction. The markset is the data structure used by compaction/gc to track live objects. The default value is "map", which will use an in-memory map; if you are limited in memory (or indeed see compaction run out of memory), you can also specify "badger" which will use an disk backed markset, using badger. This will use much less memory, but will also make compaction slower.
  • HotStoreMessageRetention -- specifies how many finalities, beyond the 4 finalities maintained by default, to maintain messages and message receipts in the hotstore. This is useful for assistive nodes that want to support syncing for other nodes beyond 4 finalities, while running with the discard coldstore option. It is also useful for miners who accept deals and need to lookback messages beyond the 4 finalities, which would otherwise hit the coldstore.
  • HotStoreFullGCFrequency -- specifies how frequenty to garbage collect the hotstore using full (moving) GC. The default value is 20, which uses full GC every 20 compactions (about once a week); set to 0 to disable full GC altogether. Rationale: badger supports online GC, and this is used by default. However it has proven to be ineffective in practice with the hotstore size slowly creeping up. In order to address this, we have added moving GC support in our badger wrapper, which can effectively reclaim all space. The downside is that it takes a bit longer to perform a moving GC and you also need enough space to house the new hotstore while the old one is still live.

Operation

When the splitstore is first enabled, the existing blockstore becomes the coldstore and a fresh hotstore is initialized.

The hotstore is warmed up on first startup so as to load all chain headers and state roots in the current head. This allows us to immediately gain the performance benefits of a smallerblockstore which can be substantial for full archival nodes.

All new writes are directed to the hotstore, while reads first hit the hotstore, with fallback to the coldstore.

Once 5 finalities have ellapsed, and every finality henceforth, the blockstore compacts. Compaction is the process of moving all unreachable objects within the last 4 finalities from the hotstore to the coldstore. If the system is configured with a discard coldstore, these objects are discarded. Note that chain headers, all the way to genesis, are considered reachable. Stateroots and messages are considered reachable only within the last 4 finalities, unless there is a live reference to them.

Compaction

Compaction works transactionally with the following algorithm:

  • We prepare a transaction, whereby all i/o referenced objects through the API are tracked.
  • We walk the chain and mark reachable objects, keeping 4 finalities of state roots and messages and all headers all the way to genesis.
  • Once the chain walk is complete, we begin full transaction protection with concurrent marking; we walk and mark all references created during the chain walk. On the same time, all I/O through the API concurrently marks objects as live references.
  • We collect cold objects by iterating through the hotstore and checking the mark set; if an object is not marked, then it is candidate for purge.
  • When running with a coldstore, we next copy all cold objects to the coldstore.
  • At this point we are ready to begin purging:
    • We sort cold objects heaviest first, so as to never delete the consituents of a DAG before the DAG itself (which would leave dangling references)
    • We delete in small batches taking a lock; each batch is checked again for marks, from the concurrent transactional mark, so as to never delete anything live
  • We then end the transaction and compact/gc the hotstore.

Garbage Collection

TBD -- see #6577

Utilities

lotus-shed has a splitstore command which provides some utilities:

  • rollback -- rolls back a splitstore installation. This command copies the hotstore on top of the coldstore, and then deletes the splitstore directory and associated metadata keys. It can also optionally compact/gc the coldstore after the copy (with the --gc-coldstore flag) and automatically rewrite the lotus config to disable splitstore (with the --rewrite-config flag). Note: the node must be stopped before running this command.
  • clear -- clears a splitstore installation for restart from snapshot.
  • check -- asynchronously runs a basic healthcheck on the splitstore. The results are appended to <lotus-repo>/datastore/splitstore/check.txt.
  • info -- prints some basic information about the splitstore.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	// CompactionThreshold is the number of epochs that need to have elapsed
	// from the previously compacted epoch to trigger a new compaction.
	//
	//        |················· CompactionThreshold ··················|
	//        |                                             |
	// =======‖≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡‖------------------------»
	//        |                    |  chain -->             ↑__ current epoch
	//        | archived epochs ___↑
	//                             ↑________ CompactionBoundary
	//
	// === :: cold (already archived)
	// ≡≡≡ :: to be archived in this compaction
	// --- :: hot
	CompactionThreshold = 5 * build.Finality

	// CompactionBoundary is the number of epochs from the current epoch at which
	// we will walk the chain for live objects.
	CompactionBoundary = 4 * build.Finality

	// SyncGapTime is the time delay from a tipset's min timestamp before we decide
	// there is a sync gap
	SyncGapTime = time.Minute
)
View Source 
var (
	// WarmupBoundary is the number of epochs to load state during warmup.
	WarmupBoundary = build.Finality
)

Functions

This section is empty.

Types

type BadgerMarkSet added in v1.11.1 

type BadgerMarkSet struct {
	// contains filtered or unexported fields
}

func (*BadgerMarkSet) Close added in v1.11.1 

func (s *BadgerMarkSet) Close() error

func (*BadgerMarkSet) Has added in v1.11.1 

func (s *BadgerMarkSet) Has(c cid.Cid) (bool, error)

func (*BadgerMarkSet) Mark added in v1.11.1 

func (s *BadgerMarkSet) Mark(c cid.Cid) error

func (*BadgerMarkSet) SetConcurrent added in v1.11.1 

func (s *BadgerMarkSet) SetConcurrent()

func (*BadgerMarkSet) Visit added in v1.11.2 

func (s *BadgerMarkSet) Visit(c cid.Cid) (bool, error)

type BadgerMarkSetEnv added in v1.11.1 

type BadgerMarkSetEnv struct {
	// contains filtered or unexported fields
}

func (*BadgerMarkSetEnv) Close added in v1.11.1 

func (e *BadgerMarkSetEnv) Close() error

func (*BadgerMarkSetEnv) Create added in v1.11.1 

func (e *BadgerMarkSetEnv) Create(name string, sizeHint int64) (MarkSet, error)

func (*BadgerMarkSetEnv) CreateVisitor added in v1.11.2 

func (e *BadgerMarkSetEnv) CreateVisitor(name string, sizeHint int64) (MarkSetVisitor, error)

func (*BadgerMarkSetEnv) SupportsVisitor added in v1.11.2 

func (e *BadgerMarkSetEnv) SupportsVisitor() bool

type ChainAccessor 

type ChainAccessor interface {
	GetTipsetByHeight(context.Context, abi.ChainEpoch, *types.TipSet, bool) (*types.TipSet, error)
	GetHeaviestTipSet() *types.TipSet
	SubscribeHeadChanges(change func(revert []*types.TipSet, apply []*types.TipSet) error)
}

ChainAccessor allows the Splitstore to access the chain. It will most likely be a ChainStore at runtime.

type Config 

type Config struct {
	// MarkSetType is the type of mark set to use.
	//
	// The default value is "map", which uses an in-memory map-backed markset.
	// If you are constrained in memory (i.e. compaction runs out of memory), you
	// can use "badger", which will use a disk-backed markset using badger.
	// Note that compaction will take quite a bit longer when using the "badger" option,
	// but that shouldn't really matter (as long as it is under 7.5hrs).
	MarkSetType string

	// DiscardColdBlocks indicates whether to skip moving cold blocks to the coldstore.
	// If the splitstore is running with a noop coldstore then this option is set to true
	// which skips moving (as it is a noop, but still takes time to read all the cold objects)
	// and directly purges cold blocks.
	DiscardColdBlocks bool

	// HotstoreMessageRetention indicates the hotstore retention policy for messages.
	// It has the following semantics:
	// - a value of 0 will only retain messages within the compaction boundary (4 finalities)
	// - a positive integer indicates the number of finalities, outside the compaction boundary,
	//   for which messages will be retained in the hotstore.
	HotStoreMessageRetention uint64

	// HotstoreFullGCFrequency indicates how frequently (in terms of compactions) to garbage collect
	// the hotstore using full (moving) GC if supported by the hotstore.
	// A value of 0 disables full GC entirely.
	// A positive value is the number of compactions before a full GC is performed;
	// a value of 1 will perform full GC in every compaction.
	HotStoreFullGCFrequency uint64
}

type MapMarkSet added in v1.11.1 

type MapMarkSet struct {
	// contains filtered or unexported fields
}

func (*MapMarkSet) Close added in v1.11.1 

func (s *MapMarkSet) Close() error

func (*MapMarkSet) Has added in v1.11.1 

func (s *MapMarkSet) Has(cid cid.Cid) (bool, error)

func (*MapMarkSet) Mark added in v1.11.1 

func (s *MapMarkSet) Mark(cid cid.Cid) error

func (*MapMarkSet) SetConcurrent added in v1.11.1 

func (s *MapMarkSet) SetConcurrent()

func (*MapMarkSet) Visit added in v1.11.2 

func (s *MapMarkSet) Visit(c cid.Cid) (bool, error)

type MapMarkSetEnv added in v1.11.1 

type MapMarkSetEnv struct{}

func NewMapMarkSetEnv added in v1.11.1 

func NewMapMarkSetEnv() (*MapMarkSetEnv, error)

func (*MapMarkSetEnv) Close added in v1.11.1 

func (e *MapMarkSetEnv) Close() error

func (*MapMarkSetEnv) Create added in v1.11.1 

func (e *MapMarkSetEnv) Create(name string, sizeHint int64) (MarkSet, error)

func (*MapMarkSetEnv) CreateVisitor added in v1.11.2 

func (e *MapMarkSetEnv) CreateVisitor(name string, sizeHint int64) (MarkSetVisitor, error)

func (*MapMarkSetEnv) SupportsVisitor added in v1.11.2 

func (e *MapMarkSetEnv) SupportsVisitor() bool

type MarkSet 

type MarkSet interface {
	Mark(cid.Cid) error
	Has(cid.Cid) (bool, error)
	Close() error
	SetConcurrent()
}

MarkSet is a utility to keep track of seen CID, and later query for them.

* If the expected dataset is large, it can be backed by a datastore (e.g. bbolt). * If a probabilistic result is acceptable, it can be backed by a bloom filter

type MarkSetEnv 

type MarkSetEnv interface {
	// Create creates a new markset within the environment.
	// name is a unique name for this markset, mapped to the filesystem in disk-backed environments
	// sizeHint is a hint about the expected size of the markset
	Create(name string, sizeHint int64) (MarkSet, error)
	// CreateVisitor is like Create, but returns a wider interface that supports atomic visits.
	// It may not be supported by some markset types (e.g. bloom).
	CreateVisitor(name string, sizeHint int64) (MarkSetVisitor, error)
	// SupportsVisitor returns true if the marksets created by this environment support the visitor interface.
	SupportsVisitor() bool
	Close() error
}

func NewBadgerMarkSetEnv added in v1.11.1 

func NewBadgerMarkSetEnv(path string) (MarkSetEnv, error)

func OpenMarkSetEnv 

func OpenMarkSetEnv(path string, mtype string) (MarkSetEnv, error)

type MarkSetVisitor added in v1.11.2 

type MarkSetVisitor interface {
	MarkSet
	ObjectVisitor
}

type ObjectVisitor added in v1.11.2 

type ObjectVisitor interface {
	Visit(cid.Cid) (bool, error)
}

ObjectVisitor is an interface for deduplicating objects during walks

type SplitStore 

type SplitStore struct {
	// contains filtered or unexported fields
}

func Open 

func Open(path string, ds dstore.Datastore, hot, cold bstore.Blockstore, cfg *Config) (*SplitStore, error)

Open opens an existing splistore, or creates a new splitstore. The splitstore is backed by the provided hot and cold stores. The returned SplitStore MUST be attached to the ChainStore with Start in order to trigger compaction.

func (*SplitStore) AddProtector added in v1.11.1 

func (s *SplitStore) AddProtector(protector func(func(cid.Cid) error) error)

func (*SplitStore) AllKeysChan 

func (s *SplitStore) AllKeysChan(ctx context.Context) (<-chan cid.Cid, error)

func (*SplitStore) Check added in v1.11.1 

func (s *SplitStore) Check() error

performs an asynchronous health-check on the splitstore; results are appended to <splitstore-path>/check.txt

func (*SplitStore) Close 

func (s *SplitStore) Close() error

func (*SplitStore) DeleteBlock 

func (s *SplitStore) DeleteBlock(_ cid.Cid) error

Blockstore interface

func (*SplitStore) DeleteMany 

func (s *SplitStore) DeleteMany(_ []cid.Cid) error

func (*SplitStore) Expose added in v1.11.1 

func (s *SplitStore) Expose() bstore.Blockstore

func (*SplitStore) Get 

func (s *SplitStore) Get(cid cid.Cid) (blocks.Block, error)

func (*SplitStore) GetSize 

func (s *SplitStore) GetSize(cid cid.Cid) (int, error)

func (*SplitStore) Has 

func (s *SplitStore) Has(cid cid.Cid) (bool, error)

func (*SplitStore) HashOnRead 

func (s *SplitStore) HashOnRead(enabled bool)

func (*SplitStore) HeadChange 

func (s *SplitStore) HeadChange(_, apply []*types.TipSet) error

func (*SplitStore) Info added in v1.11.1 

func (s *SplitStore) Info() map[string]interface{}

provides some basic information about the splitstore

func (*SplitStore) Put 

func (s *SplitStore) Put(blk blocks.Block) error

func (*SplitStore) PutMany 

func (s *SplitStore) PutMany(blks []blocks.Block) error

func (*SplitStore) Start 

func (s *SplitStore) Start(chain ChainAccessor) error

State tracking

func (*SplitStore) View 

func (s *SplitStore) View(cid cid.Cid, cb func([]byte) error) error

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.