manifest

Documentation

Overview

Package manifest stores information about the IPFS hashes associated with each portion of the Unchained Index.

We call those portions Chunks. The manifest provides enough information for anyone who receives a copy to reproduce the entire index.

Index

• Variables
• func ReadUnchainedIndex(chain string, publisher base.Address, database string) (string, error)
• func RemoveChunk(chain string, publisher base.Address, bloomFn, indexFn string) (err error)
• func Specification() string
• type Manifest
  • func LoadManifest(chain string, publisher base.Address, source Source) (man *Manifest, err error)
  • func (m *Manifest) SaveManifest(chain, fileName string) error
• type Source

Variables

var ErrManifestNotFound = errors.New("could not find manifest.json or it was empty")

Functions

func ReadUnchainedIndex

func ReadUnchainedIndex(chain string, publisher base.Address, database string) (string, error)

ReadUnchainedIndex calls UnchainedIndex smart contract to get the current manifest IPFS CID as published by the given publisher

func RemoveChunk

func RemoveChunk(chain string, publisher base.Address, bloomFn, indexFn string) (err error)

RemoveChunk must remove the underlying chunk (both Bloom filter and the chunk itself) and update the manifest by removing all chunks at or after the given path. Note that if this function aborts due to error and the backup files still exist, the function will attempt to restore the backup files before returning.

func Specification

func Specification() string

Types

type Manifest

type Manifest struct {
	// The specification version used to build this instance of Unchained Index
	Version string `json:"version"`

	// The chain against which this index was created
	Chain string `json:"chain"`

	// An IPFS hash pointing to documentation describing the binary format of the files in the index
	Specification base.IpfsHash `json:"specification"`

	// An IPFS hash pointing to documentation describing the binary format of the files in the index
	Config configtypes.ScrapeSettings `json:"config"`

	// A list of pinned chunks (see types.ChunkRecord) detailing the location of all chunks in the index and associated bloom filters
	Chunks []types.ChunkRecord `json:"chunks"`

	// A map to make set membership easier
	ChunkMap map[string]*types.ChunkRecord `json:"-"`
}

Manifest is a data structure consisting of a list of chunk records (i.e. block ranges, Bloom filter hashes, and chunk hashes) and the nessacary additional information needed to reproduce the index from scratch using only the node as source of truth.

func LoadManifest

func LoadManifest(chain string, publisher base.Address, source Source) (man *Manifest, err error)

LoadManifest reads the manifest from a file or from the Unchained Index smart contract.

It first checks if the manifest file exists. If it does, it reads the manifest from the file. If the caller requests the contract or the cached manifest does not exist, it reads the manifest from the contract. It then checks if the new manifest has more chunks than the existing one. If it does (or if the file didn't exist), it saves the new manifest to the file. Finally, it creates a map of chunks for easy lookup and sets the specification if it is not already set.

func (*Manifest) SaveManifest

func (m *Manifest) SaveManifest(chain, fileName string) error

SaveManifest writes the manifest to a file in JSON format. It first removes any duplicate chunks and sorts them. It then marshals the manifest into JSON and writes it to the file. The function takes the chain name and file name as arguments.

type Source

type Source int

const (
	None Source = 1 << iota
	LocalCache
	FromContract
	TempContract
)