stores

package
v0.0.0-...-f311442 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 7, 2023 License: Apache-2.0, MIT Imports: 31 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrNotFound = xerrors.New("not found")

Functions

func AllowDuplicatePuts 

func AllowDuplicatePuts(allow bool) carv2.Option

AllowDuplicatePuts is a write option which makes a CAR blockstore not deduplicate blocks in Put and PutMany. The default is to deduplicate, which matches the current semantics of go-ipfs-blockstore v1.

Note that this option only affects the blockstore, and is ignored by the root go-car/v2 package.

func DestroyShardSync 

func DestroyShardSync(ctx context.Context, ds DAGStoreWrapper, pieceCid cid.Cid) error

DestroyShardSync calls the DAGStore DestroyShard method and waits synchronously in a dedicated channel until the shard has been destroyed completely.

func FilestoreOf 

func FilestoreOf(bs bstore.Blockstore) (bstore.Blockstore, error)

FilestoreOf returns a FileManager/Filestore backed entirely by a blockstore without requiring a datastore. It achieves this by coercing the blockstore into a datastore. The resulting blockstore is suitable for usage with DagBuilderHelper with DagBuilderParams#NoCopy=true.

func HeaderSize 

func HeaderSize(h *CarHeader) (uint64, error)

func IsNotFound 

func IsNotFound(err error) bool

func LdRead 

func LdRead(r io.Reader, zeroLenAsEOF bool) ([]byte, error)

func LdSize 

func LdSize(d ...[]byte) uint64

func LdWrite 

func LdWrite(w io.Writer, d ...[]byte) error

func ReadNode 

func ReadNode(r io.Reader, zeroLenAsEOF bool) (cid.Cid, []byte, error)

func RegisterShardSync 

func RegisterShardSync(ctx context.Context, ds DAGStoreWrapper, pieceCid cid.Cid, carPath string, eagerInit bool) error

RegisterShardSync calls the DAGStore RegisterShard method and waits synchronously in a dedicated channel until the registration has completed fully.

func ToByteReader 

func ToByteReader(r io.Reader) io.ByteReader

func ToReaderAt 

func ToReaderAt(rs io.ReadSeeker) io.ReaderAt

func UseWholeCIDs 

func UseWholeCIDs(enable bool) carv2.Option

UseWholeCIDs is a read option which makes a CAR blockstore identify blocks by whole CIDs, and not just their multihashes. The default is to use multihashes, which matches the current semantics of go-ipfs-blockstore v1.

Enabling this option affects a number of methods, including read-only ones:

• Get, Has, and HasSize will only return a block only if the entire CID is present in the CAR file.

• AllKeysChan will return the original whole CIDs, instead of with their multicodec set to "raw" to just provide multihashes.

• If AllowDuplicatePuts isn't set, Put and PutMany will deduplicate by the whole CID, allowing different CIDs with equal multihashes.

Note that this option only affects the blockstore, and is ignored by the root go-car/v2 package.

func WithAsyncErrorHandler 

func WithAsyncErrorHandler(ctx context.Context, errHandler func(error)) context.Context

WithAsyncErrorHandler returns a context with async error handling set to the given errHandler. Any errors that occur during asynchronous operations of AllKeysChan will be passed to the given handler.

func WriteCar 

func WriteCar(ctx context.Context, ds format.NodeGetter, roots []cid.Cid, w io.Writer) error

func WriteHeader 

func WriteHeader(h *CarHeader, w io.Writer) error

Types

type ByteReadSeeker 

type ByteReadSeeker interface {
	io.ReadSeeker
	io.ByteReader
}

func ToByteReadSeeker 

func ToByteReadSeeker(r io.Reader) ByteReadSeeker

type BytesReader 

type BytesReader interface {
	io.Reader
	io.ByteReader
}

type CarHeader 

type CarHeader struct {
	Roots   []cid.Cid
	Version uint64
}

func LoadCar 

func LoadCar(s Store, r io.Reader) (*CarHeader, error)

func ReadHeader 

func ReadHeader(r io.Reader) (*CarHeader, error)

func (CarHeader) Matches 

func (h CarHeader) Matches(other CarHeader) bool

Matches checks whether two headers match. Two headers are considered matching if:

  1. They have the same version number, and
  2. They contain the same root CIDs in any order.

Note, this function explicitly ignores the order of roots. If order of roots matter use reflect.DeepEqual instead.

type CarReader 

type CarReader struct {
	Header *CarHeader
	// contains filtered or unexported fields
}

func NewCarReader 

func NewCarReader(r io.Reader) (*CarReader, error)

func NewCarReaderWithZeroLengthSectionAsEOF 

func NewCarReaderWithZeroLengthSectionAsEOF(r io.Reader) (*CarReader, error)

func (*CarReader) Next 

func (cr *CarReader) Next() (blocks.Block, error)

type ClosableBlockstore 

type ClosableBlockstore interface {
	bstore.Blockstore
	io.Closer
}

func ReadOnlyFilestore 

func ReadOnlyFilestore(path string) (ClosableBlockstore, error)

ReadOnlyFilestore opens the CAR in the specified path as as a read-only blockstore, and fronts it with a Filestore whose positional mappings are stored inside the CAR itself. It must be closed after done.

func ReadWriteFilestore 

func ReadWriteFilestore(path string, roots ...cid.Cid) (ClosableBlockstore, error)

ReadWriteFilestore opens the CAR in the specified path as as a read-write blockstore, and fronts it with a Filestore whose positional mappings are stored inside the CAR itself. It must be closed after done. Closing will finalize the CAR blockstore.

type DAGStoreWrapper 

type DAGStoreWrapper interface {
	// RegisterShard loads a CAR file into the DAG store and builds an
	// index for it, sending the result on the supplied channel on completion
	RegisterShard(ctx context.Context, pieceCid cid.Cid, carPath string, eagerInit bool, resch chan dagstore.ShardResult) error

	// LoadShard fetches the data for a shard and provides a blockstore
	// interface to it.
	//
	// The blockstore must be closed to release the shard.
	LoadShard(ctx context.Context, pieceCid cid.Cid) (ClosableBlockstore, error)

	// MigrateDeals migrates the supplied storage deals into the DAG store.
	MigrateDeals(ctx context.Context, deals []storagemarket.MinerDeal) (bool, error)

	// GetPiecesContainingBlock returns the CID of all pieces that contain
	// the block with the given CID
	GetPiecesContainingBlock(blockCID cid.Cid) ([]cid.Cid, error)

	GetIterableIndexForPiece(pieceCid cid.Cid) (carindex.IterableIndex, error)

	// DestroyShard initiates the registration of a new shard.
	//
	// This method returns an error synchronously if preliminary validation fails.
	// Otherwise, it queues the shard for destruction. The caller should monitor
	// supplied channel for a result.
	DestroyShard(ctx context.Context, pieceCid cid.Cid, resch chan dagstore.ShardResult) error

	// Close closes the dag store wrapper.
	Close() error
}

DAGStoreWrapper hides the details of the DAG store implementation from the other parts of go-fil-markets.

type OffsetReadSeeker 

type OffsetReadSeeker struct {
	// contains filtered or unexported fields
}

OffsetReadSeeker implements Read, and ReadAt on a section of an underlying io.ReaderAt. The main difference between io.SectionReader and OffsetReadSeeker is that NewOffsetReadSeeker does not require the user to know the number of readable bytes.

It also partially implements Seek, where the implementation panics if io.SeekEnd is passed. This is because, OffsetReadSeeker does not know the end of the file therefore cannot seek relative to it.

func NewOffsetReadSeeker 

func NewOffsetReadSeeker(r io.ReaderAt, off int64) *OffsetReadSeeker

NewOffsetReadSeeker returns an OffsetReadSeeker that reads from r starting offset offset off and stops with io.EOF when r reaches its end. The Seek function will panic if whence io.SeekEnd is passed.

func (*OffsetReadSeeker) Offset 

func (o *OffsetReadSeeker) Offset() int64

func (*OffsetReadSeeker) Position 

func (o *OffsetReadSeeker) Position() int64

Position returns the current position of this reader relative to the initial offset.

func (*OffsetReadSeeker) Read 

func (o *OffsetReadSeeker) Read(p []byte) (n int, err error)

func (*OffsetReadSeeker) ReadAt 

func (o *OffsetReadSeeker) ReadAt(p []byte, off int64) (n int, err error)

func (*OffsetReadSeeker) ReadByte 

func (o *OffsetReadSeeker) ReadByte() (byte, error)

func (*OffsetReadSeeker) Seek 

func (o *OffsetReadSeeker) Seek(offset int64, whence int) (int64, error)

type OffsetWriteSeeker 

type OffsetWriteSeeker struct {
	// contains filtered or unexported fields
}

func NewOffsetWriter 

func NewOffsetWriter(w io.WriterAt, off int64) *OffsetWriteSeeker

func (*OffsetWriteSeeker) Position 

func (ow *OffsetWriteSeeker) Position() int64

Position returns the current position of this writer relative to the initial offset, i.e. the number of bytes written.

func (*OffsetWriteSeeker) Seek 

func (ow *OffsetWriteSeeker) Seek(offset int64, whence int) (int64, error)

func (*OffsetWriteSeeker) Write 

func (ow *OffsetWriteSeeker) Write(b []byte) (n int, err error)

type ReadOnly 

type ReadOnly struct {
	// contains filtered or unexported fields
}

ReadOnly provides a read-only CAR Block Store.

func NewReadOnly 

func NewReadOnly(backing io.ReaderAt, idx index.Index, opts ...carv2.Option) (*ReadOnly, error)

NewReadOnly creates a new ReadOnly blockstore from the backing with a optional index as idx. This function accepts both CARv1 and CARv2 backing. The blockstore is instantiated with the given index if it is not nil.

Otherwise: * For a CARv1 backing an index is generated. * For a CARv2 backing an index is only generated if Header.HasIndex returns false.

There is no need to call ReadOnly.Close on instances returned by this function.

func OpenReadOnly 

func OpenReadOnly(path string, opts ...carv2.Option) (*ReadOnly, error)

OpenReadOnly opens a read-only blockstore from a CAR file (either v1 or v2), generating an index if it does not exist. Note, the generated index if the index does not exist is ephemeral and only stored in memory. See car.GenerateIndex and Index.Attach for persisting index onto a CAR file.

func (*ReadOnly) AllKeysChan 

func (b *ReadOnly) AllKeysChan(ctx context.Context) (<-chan cid.Cid, error)

AllKeysChan returns the list of keys in the CAR data payload. If the ctx is constructed using WithAsyncErrorHandler any errors that occur during asynchronous retrieval of CIDs will be passed to the error handler function set in context. Otherwise, errors will terminate the asynchronous operation silently.

See WithAsyncErrorHandler

func (*ReadOnly) Close 

func (b *ReadOnly) Close() error

Close closes the underlying reader if it was opened by OpenReadOnly. After this call, the blockstore can no longer be used.

Note that this call may block if any blockstore operations are currently in progress, including an AllKeysChan that hasn't been fully consumed or cancelled.

func (*ReadOnly) DeleteBlock 

func (b *ReadOnly) DeleteBlock(_ context.Context, _ cid.Cid) error

DeleteBlock is unsupported and always errors.

func (*ReadOnly) Get 

func (b *ReadOnly) Get(ctx context.Context, key cid.Cid) (blocks.Block, error)

Get gets a block corresponding to the given key. This API will always return true if the given key has multihash.IDENTITY code.

func (*ReadOnly) GetSize 

func (b *ReadOnly) GetSize(ctx context.Context, key cid.Cid) (int, error)

GetSize gets the size of an item corresponding to the given key.

func (*ReadOnly) Has 

func (b *ReadOnly) Has(ctx context.Context, key cid.Cid) (bool, error)

Has indicates if the store contains a block that corresponds to the given key. This function always returns true for any given key with multihash.IDENTITY code.

func (*ReadOnly) HashOnRead 

func (b *ReadOnly) HashOnRead(bool)

HashOnRead is currently unimplemented; hashing on reads never happens.

func (*ReadOnly) Put 

func (b *ReadOnly) Put(context.Context, blocks.Block) error

Put is not supported and always returns an error.

func (*ReadOnly) PutMany 

func (b *ReadOnly) PutMany(context.Context, []blocks.Block) error

PutMany is not supported and always returns an error.

func (*ReadOnly) Roots 

func (b *ReadOnly) Roots() ([]cid.Cid, error)

Roots returns the root CIDs of the backing CAR.

type ReadOnlyBlockstores 

type ReadOnlyBlockstores struct {
	// contains filtered or unexported fields
}

ReadOnlyBlockstores tracks open read blockstores.

func NewReadOnlyBlockstores 

func NewReadOnlyBlockstores() *ReadOnlyBlockstores

func (*ReadOnlyBlockstores) Get 

func (r *ReadOnlyBlockstores) Get(key string) (bstore.Blockstore, error)

func (*ReadOnlyBlockstores) Track 

func (r *ReadOnlyBlockstores) Track(key string, bs bstore.Blockstore) (bool, error)

func (*ReadOnlyBlockstores) Untrack 

func (r *ReadOnlyBlockstores) Untrack(key string) error

type ReadStore 

type ReadStore interface {
	Get(cid.Cid) (blocks.Block, error)
}

type ReadWrite 

type ReadWrite struct {
	// contains filtered or unexported fields
}

ReadWrite implements a blockstore that stores blocks in CARv2 format. Blocks put into the blockstore can be read back once they are successfully written. This implementation is preferable for a write-heavy workload. The blocks are written immediately on Put and PutAll calls, while the index is stored in memory and updated incrementally.

The Finalize function must be called once the putting blocks are finished. Upon calling Finalize header is finalized and index is written out. Once finalized, all read and write calls to this blockstore will result in errors.

func OpenReadWrite 

func OpenReadWrite(path string, roots []cid.Cid, opts ...carv2.Option) (*ReadWrite, error)

OpenReadWrite creates a new ReadWrite at the given path with a provided set of root CIDs and options.

ReadWrite.Finalize must be called once putting and reading blocks are no longer needed. Upon calling ReadWrite.Finalize the CARv2 header and index are written out onto the file and the backing file is closed. Once finalized, all read and write calls to this blockstore will result in errors. Note, ReadWrite.Finalize must be called on an open instance regardless of whether any blocks were put or not.

If a file at given path does not exist, the instantiation will write car.Pragma and data payload header (i.e. the inner CARv1 header) onto the file before returning.

When the given path already exists, the blockstore will attempt to resume from it. On resumption the existing data sections in file are re-indexed, allowing the caller to continue putting any remaining blocks without having to re-ingest blocks for which previous ReadWrite.Put returned successfully.

Resumption only works on files that were created by a previous instance of a ReadWrite blockstore. This means a file created as a result of a successful call to OpenReadWrite can be resumed from as long as write operations such as ReadWrite.Put, ReadWrite.PutMany returned successfully. On resumption the roots argument and WithDataPadding option must match the previous instantiation of ReadWrite blockstore that created the file. More explicitly, the file resuming from must:

  1. start with a complete CARv2 car.Pragma.
  2. contain a complete CARv1 data header with root CIDs matching the CIDs passed to the constructor, starting at offset optionally padded by WithDataPadding, followed by zero or more complete data sections. If any corrupt data sections are present the resumption will fail. Note, if set previously, the blockstore must use the same WithDataPadding option as before, since this option is used to locate the CARv1 data payload.

Note, resumption should be used with WithCidDeduplication, so that blocks that are successfully written into the file are not re-written. Unless, the user explicitly wants duplicate blocks.

Resuming from finalized files is allowed. However, resumption will regenerate the index regardless by scanning every existing block in file.

func (*ReadWrite) AllKeysChan 

func (b *ReadWrite) AllKeysChan(ctx context.Context) (<-chan cid.Cid, error)

func (*ReadWrite) DeleteBlock 

func (b *ReadWrite) DeleteBlock(_ context.Context, _ cid.Cid) error

func (*ReadWrite) Discard 

func (b *ReadWrite) Discard()

Discard closes this blockstore without finalizing its header and index. After this call, the blockstore can no longer be used.

Note that this call may block if any blockstore operations are currently in progress, including an AllKeysChan that hasn't been fully consumed or cancelled.

func (*ReadWrite) Finalize 

func (b *ReadWrite) Finalize() error

Finalize finalizes this blockstore by writing the CARv2 header, along with flattened index for more efficient subsequent read. After this call, the blockstore can no longer be used.

func (*ReadWrite) Get 

func (b *ReadWrite) Get(ctx context.Context, key cid.Cid) (blocks.Block, error)

func (*ReadWrite) GetSize 

func (b *ReadWrite) GetSize(ctx context.Context, key cid.Cid) (int, error)

func (*ReadWrite) Has 

func (b *ReadWrite) Has(ctx context.Context, key cid.Cid) (bool, error)

func (*ReadWrite) HashOnRead 

func (b *ReadWrite) HashOnRead(enable bool)

func (*ReadWrite) Put 

func (b *ReadWrite) Put(ctx context.Context, blk blocks.Block) error

Put puts a given block to the underlying datastore

func (*ReadWrite) PutMany 

func (b *ReadWrite) PutMany(ctx context.Context, blks []blocks.Block) error

PutMany puts a slice of blocks at the same time using batching capabilities of the underlying datastore whenever possible.

func (*ReadWrite) Roots 

func (b *ReadWrite) Roots() ([]cid.Cid, error)

type ReadWriteBlockstores 

type ReadWriteBlockstores struct {
	// contains filtered or unexported fields
}

ReadWriteBlockstores tracks open ReadWrite CAR blockstores.

func NewReadWriteBlockstores 

func NewReadWriteBlockstores() *ReadWriteBlockstores

func (*ReadWriteBlockstores) Get 

func (r *ReadWriteBlockstores) Get(key string) (*blockstore.ReadWrite, error)

func (*ReadWriteBlockstores) GetOrOpen 

func (r *ReadWriteBlockstores) GetOrOpen(key string, path string, rootCid cid.Cid) (*blockstore.ReadWrite, error)

func (*ReadWriteBlockstores) Untrack 

func (r *ReadWriteBlockstores) Untrack(key string) error

type Store 

type Store interface {
	Put(blocks.Block) error
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.