storage

Documentation

Overview

Package storage provides a CAR abstraction for the github.com/ipld/go-ipld-prime/storage interfaces in the form of a StorageCar.

THIS PACKAGE IS EXPERIMENTAL. Breaking changes may be introduced in semver-minor releases before this package stabilizes. Use with caution and prefer the blockstore API if stability is required.

StorageCar as ReadableStorage provides basic Get and Has operations. It also implements StreamingReadableStorage for the more efficient GetStreaming operation which is easily supported by the CAR format.

StorageCar as WritableStorage provides the Put operation. It does not implement StreamingWritableStorage because the CAR format requires CIDs to be written before the blocks themselves, which is not possible with StreamingWritableStorage without buffering. Therefore, the PutStream function in github.com/ipld/go-ipld-prime/storage will provide equivalent functionality if it were to be implemented here.

StorageCar can be used with an IPLD LinkSystem, defined by github.com/ipld/go-ipld-prime/linking, with the linking.SetReadStorage and linking.SetWriteStorage functions, to provide read and/or write to and/or from a CAR format as required.

The focus of the StorageCar interfaces is to use the minimal possible IO interface for the operation(s) being performed.

• OpenReadable requires an io.ReaderAt as seeking is required for random-access reads as a ReadableStore.

• NewWritable requires an io.Writer when used to write a CARv1 as this format can be written in a continuous stream as blocks are written through a WritableStore (i.e. when the WriteAsCarV1 option is turned on). When used to write a CARv2, the default mode, a random-access io.WriterAt is required as the CARv2 header must be written after the payload is finalized and index written in order to indicate payload location in the output. The plain Writable store may be used to stream CARv1 contents without buffering; only storing CIDs in memory for de-duplication (where required) and to still allow Has operations.

• NewReadableWritable requires an io.ReaderAt and an io.Writer as it combines the functionality of a NewWritable with OpenReadable, being able to random- access read any written blocks.

• OpenReadableWritable requires an io.ReaderAt, an io.Writer and an io.WriterAt as it extends the NewReadableWritable functionality with the ability to resume an existing CAR. In addition, if the CAR being resumed is a CARv2, the IO object being provided must have a Truncate() method (e.g. an io.File) in order to properly manage CAR lifecycle and avoid writing a corrupt CAR.

The following options are available to customize the behavior of the StorageCar:

• WriteAsCarV1

• StoreIdentityCIDs

• AllowDuplicatePuts

• UseWholeCIDs

• ZeroLengthSectionAsEOF

• UseIndexCodec

• UseDataPadding

• UseIndexPadding

• MaxIndexCidSize

• MaxAllowedHeaderSize

• MaxAllowedSectionSize

Index

• Variables
• func IsNotFound(err error) bool
• type ErrNotFound
  • func (e ErrNotFound) Error() string
  • func (e ErrNotFound) Is(err error) bool
  • func (e ErrNotFound) NotFound() bool
• type ReadableCar
  • func OpenReadable(reader io.ReaderAt, opts ...carv2.Option) (ReadableCar, error)
• type ReaderAtWriterAt
• type StorageCar
  • func NewReadableWritable(rw ReaderAtWriterAt, roots []cid.Cid, opts ...carv2.Option) (*StorageCar, error)
  • func OpenReadableWritable(rw ReaderAtWriterAt, roots []cid.Cid, opts ...carv2.Option) (*StorageCar, error)
  • func (sc *StorageCar) Finalize() error
  • func (sc *StorageCar) Get(ctx context.Context, keyStr string) ([]byte, error)
  • func (sc *StorageCar) GetStream(ctx context.Context, keyStr string) (io.ReadCloser, error)
  • func (sc *StorageCar) Has(ctx context.Context, keyStr string) (bool, error)
  • func (sc *StorageCar) Index() index.Index
  • func (sc *StorageCar) Put(ctx context.Context, keyStr string, data []byte) error
  • func (sc *StorageCar) Roots() []cid.Cid
• type WritableCar
  • func NewWritable(writer io.Writer, roots []cid.Cid, opts ...carv2.Option) (WritableCar, error)

Variables

var ErrClosed = errors.New("cannot use a CAR storage after closing")

Functions

func IsNotFound

func IsNotFound(err error) bool

Types

type ErrNotFound

type ErrNotFound struct {
	Cid cid.Cid
}

func (ErrNotFound) Error

func (e ErrNotFound) Error() string

func (ErrNotFound) Is

func (e ErrNotFound) Is(err error) bool

func (ErrNotFound) NotFound

func (e ErrNotFound) NotFound() bool

type ReadableCar

type ReadableCar interface {
	ipldstorage.ReadableStorage
	ipldstorage.StreamingReadableStorage
	Roots() []cid.Cid
	Index() index.Index
}

func OpenReadable

func OpenReadable(reader io.ReaderAt, opts ...carv2.Option) (ReadableCar, error)

OpenReadable opens a CARv1 or CARv2 file for reading as a ReadableStorage and StreamingReadableStorage as defined by github.com/ipld/go-ipld-prime/storage.

The returned ReadableStorage is compatible with a linksystem SetReadStorage method as defined by github.com/ipld/go-ipld-prime/linking to provide a block source backed by a CAR.

When opening a CAR, an initial scan is performed to generate an index, or load an index from a CARv2 index where available. This index data is kept in memory while the CAR is being used in order to provide efficient random Get access to blocks and Has operations.

The Readable supports StreamingReadableStorage, which allows for efficient GetStreaming operations straight out of the underlying CAR where the linksystem can make use of it.

type ReaderAtWriterAt

type ReaderAtWriterAt interface {
	io.ReaderAt
	io.Writer
	io.WriterAt
}

type StorageCar

type StorageCar struct {
	// contains filtered or unexported fields
}

func NewReadableWritable

func NewReadableWritable(rw ReaderAtWriterAt, roots []cid.Cid, opts ...carv2.Option) (*StorageCar, error)

NewReadableWritable creates a new StorageCar that is able to provide both StorageReader and StorageWriter functionality.

The returned StorageCar is compatible with a linksystem SetReadStorage and SetWriteStorage methods as defined by github.com/ipld/go-ipld-prime/linking.

When writing a CARv2 format, it is important to call the Finalize method on the returned WritableStorage in order to write the CARv2 header and index.

func OpenReadableWritable

func OpenReadableWritable(rw ReaderAtWriterAt, roots []cid.Cid, opts ...carv2.Option) (*StorageCar, error)

OpenReadableWritable creates a new StorageCar that is able to provide both StorageReader and StorageWriter functionality.

The returned StorageCar is compatible with a linksystem SetReadStorage and SetWriteStorage methods as defined by github.com/ipld/go-ipld-prime/linking.

It attempts to resume a CARv2 file that was previously written to by NewWritable, or NewReadableWritable.

func (*StorageCar) Finalize

func (sc *StorageCar) Finalize() error

Finalize writes the CAR index to the underlying writer if the CAR being written is a CARv2. It also writes a finalized CARv2 header which details payload location. This should be called on a writable StorageCar in order to avoid data loss.

func (*StorageCar) Get

func (sc *StorageCar) Get(ctx context.Context, keyStr string) ([]byte, error)

Get returns the block bytes identified by the given CID provided in string form. The keyStr value must be a valid CID binary string (not a multibase string representation), i.e. generated with CID#KeyString().

func (*StorageCar) GetStream

func (sc *StorageCar) GetStream(ctx context.Context, keyStr string) (io.ReadCloser, error)

GetStream returns a stream of the block bytes identified by the given CID provided in string form. The keyStr value must be a valid CID binary string (not a multibase string representation), i.e. generated with CID#KeyString().

func (*StorageCar) Has

func (sc *StorageCar) Has(ctx context.Context, keyStr string) (bool, error)

Has returns true if the CAR contains a block identified by the given CID provided in string form. The keyStr value must be a valid CID binary string (not a multibase string representation), i.e. generated with CID#KeyString().

func (*StorageCar) Index

func (sc *StorageCar) Index() index.Index

Index gives direct access to the index. It should be used with care. Modifying the index may result corruption or invalid reads.

func (*StorageCar) Put

func (sc *StorageCar) Put(ctx context.Context, keyStr string, data []byte) error

Put adds a block to the CAR, where the block is identified by the given CID provided in string form. The keyStr value must be a valid CID binary string (not a multibase string representation), i.e. generated with CID#KeyString().

func (*StorageCar) Roots

func (sc *StorageCar) Roots() []cid.Cid

Roots returns the roots of the CAR.

type WritableCar

type WritableCar interface {
	ipldstorage.WritableStorage
	Roots() []cid.Cid
	Index() index.Index
	Finalize() error
}

WritableCar is compatible with storage.WritableStorage but also returns the roots of the CAR. It does not implement ipld.StreamingWritableStorage as the CAR format does not support streaming data followed by its CID, so any streaming implementation would perform buffering and copy the existing storage.PutStream() implementation.

func NewWritable

func NewWritable(writer io.Writer, roots []cid.Cid, opts ...carv2.Option) (WritableCar, error)

NewWritable creates a new WritableStorage as defined by github.com/ipld/go-ipld-prime/storage that writes a CARv1 or CARv2 format to the given io.Writer.

The returned WritableStorage is compatible with a linksystem SetWriteStorage method as defined by github.com/ipld/go-ipld-prime/linking to provide a block sink backed by a CAR.

The WritableStorage supports Put operations, which will write blocks to the CAR in the order they are received.

When writing a CARv2 format (the default), the provided writer must be compatible with io.WriterAt in order to provide random access as the CARv2 header must be written after the blocks in order to indicate the size of the CARv2 data payload.

A CARv1 (generated using the WriteAsCarV1 option) only requires an io.Writer and can therefore stream CAR contents as blocks are written while still providing Has operations and the ability to avoid writing duplicate blocks as required.

When writing a CARv2 format, it is important to call the Finalize method on the returned WritableStorage in order to write the CARv2 header and index.