Documentation
¶
Overview ¶
package index provides indexing functionality for CARv1 data payload represented as a mapping of CID to offset. This can then be used to implement random access over a CARv1.
Index can be written or read using the following static functions: index.WriteTo and index.ReadFrom.
Index ¶
- Constants
- Variables
- func GetFirst(idx Index, key cid.Cid) (uint64, error)
- func WriteTo(idx Index, w io.Writer) (uint64, error)
- type Index
- type IterableIndex
- type MultihashIndexSorted
- func (m *MultihashIndexSorted) Codec() multicodec.Code
- func (m *MultihashIndexSorted) ForEach(f func(mh multihash.Multihash, offset uint64) error) error
- func (m *MultihashIndexSorted) GetAll(cid cid.Cid, f func(uint64) bool) error
- func (m *MultihashIndexSorted) Load(records []Record) error
- func (m *MultihashIndexSorted) Marshal(w io.Writer) (uint64, error)
- func (m *MultihashIndexSorted) Unmarshal(r io.Reader) error
- type Record
Examples ¶
Constants ¶
const CarIndexNone = 0x300000
CarIndexNone is a sentinal value used as a multicodec code for the index indicating no index.
Variables ¶
var ErrNotFound = errors.New("not found")
ErrNotFound signals a record is not found in the index.
Functions ¶
func GetFirst ¶
GetFirst is a wrapper over Index.GetAll, returning the offset for the first matching indexed CID.
func WriteTo ¶
WriteTo writes the given idx into w. The written bytes include the index encoding. This can then be read back using index.ReadFrom
Example ¶
ExampleWriteTo unmarshalls an index from an indexed CARv2 file, and stores it as a separate file on disk.
package main
import (
"fmt"
"io"
"io/ioutil"
"os"
"reflect"
carv2 "github.com/ipld/go-car/v2"
"github.com/ipld/go-car/v2/index"
)
func main() {
// Open the CARv2 file
src := "../testdata/sample-wrapped-v2.car"
cr, err := carv2.OpenReader(src)
if err != nil {
panic(err)
}
defer func() {
if err := cr.Close(); err != nil {
panic(err)
}
}()
// Read and unmarshall index within CARv2 file.
idx, err := index.ReadFrom(cr.IndexReader())
if err != nil {
panic(err)
}
// Store the index alone onto destination file.
f, err := ioutil.TempFile(os.TempDir(), "example-index-*.carindex")
if err != nil {
panic(err)
}
defer func() {
if err := f.Close(); err != nil {
panic(err)
}
}()
_, err = index.WriteTo(idx, f)
if err != nil {
panic(err)
}
// Seek to the beginning of tile to read it back.
_, err = f.Seek(0, io.SeekStart)
if err != nil {
panic(err)
}
// Read and unmarshall the destination file as a separate index instance.
reReadIdx, err := index.ReadFrom(f)
if err != nil {
panic(err)
}
// Expect indices to be equal.
if reflect.DeepEqual(idx, reReadIdx) {
fmt.Printf("Saved index file matches the index embedded in CARv2 at %v.\n", src)
} else {
panic("expected to get the same index as the CARv2 file")
}
}
Output: Saved index file matches the index embedded in CARv2 at ../testdata/sample-wrapped-v2.car.
Types ¶
type Index ¶
type Index interface {
// Codec provides the multicodec code that the index implements.
//
// Note that this may return a reserved code if the index
// implementation is not defined in a spec.
Codec() multicodec.Code
// Marshal encodes the index in serial form.
Marshal(w io.Writer) (uint64, error)
// Unmarshal decodes the index from its serial form.
Unmarshal(r io.Reader) error
// Load inserts a number of records into the index.
// Note that Index will load all given records. Any filtering of the records such as
// exclusion of CIDs with multihash.IDENTITY code must occur prior to calling this function.
//
// Further, the actual information extracted and indexed from the given records entirely
// depends on the concrete index implementation.
// For example, some index implementations may only store partial multihashes.
Load([]Record) error
// GetAll looks up all blocks matching a given CID,
// calling a function for each one of their offsets.
//
// GetAll stops if the given function returns false,
// or there are no more offsets; whichever happens first.
//
// If no error occurred and the CID isn't indexed,
// meaning that no callbacks happen,
// ErrNotFound is returned.
GetAll(cid.Cid, func(uint64) bool) error
}
Index provides an interface for looking up byte offset of a given CID.
Note that each indexing mechanism is free to match CIDs however it sees fit. For example, multicodec.CarIndexSorted only indexes multihash digests, meaning that Get and GetAll will find matching blocks even if the CID's encoding multicodec differs. Other index implementations might index the entire CID, the entire multihash, or just part of a multihash's digest.
See: multicodec.CarIndexSorted, multicodec.CarMultihashIndexSorted
func ReadFrom ¶
ReadFrom reads index from r. The reader decodes the index by reading the first byte to interpret the encoding. Returns error if the encoding is not known.
Example ¶
ExampleReadFrom unmarshalls an index from an indexed CARv2 file, and for each root CID prints the offset at which its corresponding block starts relative to the wrapped CARv1 data payload.
package main
import (
"fmt"
carv2 "github.com/ipld/go-car/v2"
"github.com/ipld/go-car/v2/index"
)
func main() {
// Open the CARv2 file
cr, err := carv2.OpenReader("../testdata/sample-wrapped-v2.car")
if err != nil {
panic(err)
}
defer cr.Close()
// Get root CIDs in the CARv1 file.
roots, err := cr.Roots()
if err != nil {
panic(err)
}
// Read and unmarshall index within CARv2 file.
idx, err := index.ReadFrom(cr.IndexReader())
if err != nil {
panic(err)
}
// For each root CID print the offset relative to CARv1 data payload.
for _, r := range roots {
offset, err := index.GetFirst(idx, r)
if err != nil {
panic(err)
}
fmt.Printf("Frame with CID %v starts at offset %v relative to CARv1 data payload.\n", r, offset)
}
}
Output: Frame with CID bafy2bzaced4ueelaegfs5fqu4tzsh6ywbbpfk3cxppupmxfdhbpbhzawfw5oy starts at offset 61 relative to CARv1 data payload.
type IterableIndex ¶ added in v2.1.0
type IterableIndex interface {
Index
// ForEach takes a callback function that will be called
// on each entry in the index. The arguments to the callback are
// the multihash of the element, and the offset in the car file
// where the element appears.
//
// If the callback returns a non-nil error, the iteration is aborted,
// and the ForEach function returns the error to the user.
//
// An index may contain multiple offsets corresponding to the same multihash, e.g. via duplicate blocks.
// In such cases, the given function may be called multiple times with the same multhihash but different offset.
//
// The order of calls to the given function is deterministic, but entirely index-specific.
ForEach(func(multihash.Multihash, uint64) error) error
}
IterableIndex extends Index in cases where the Index is able to provide an iterator for getting the list of all multihashes in the index.
Note that it is possible for an index to contain multiple offsets for a given multihash.
See: IterableIndex.ForEach, Index.GetAll.
type MultihashIndexSorted ¶ added in v2.1.0
type MultihashIndexSorted map[uint64]*multiWidthCodedIndex
MultihashIndexSorted maps multihash code (i.e. hashing algorithm) to multiWidthCodedIndex.
func NewMultihashSorted ¶ added in v2.1.0
func NewMultihashSorted() *MultihashIndexSorted
func (*MultihashIndexSorted) Codec ¶ added in v2.1.0
func (m *MultihashIndexSorted) Codec() multicodec.Code
func (*MultihashIndexSorted) ForEach ¶ added in v2.1.0
func (m *MultihashIndexSorted) ForEach(f func(mh multihash.Multihash, offset uint64) error) error
ForEach calls f for every multihash and its associated offset stored by this index.
func (*MultihashIndexSorted) GetAll ¶ added in v2.1.0
func (m *MultihashIndexSorted) GetAll(cid cid.Cid, f func(uint64) bool) error
func (*MultihashIndexSorted) Load ¶ added in v2.1.0
func (m *MultihashIndexSorted) Load(records []Record) error
Source Files