Documentation
¶
Overview ¶
Package index handles reading, writing, and searching the Unchained Index include both the chunk data and the Bloom filters
Index ¶
- Constants
- Variables
- func ChunkCid(path string) (chunkCid string, err error)
- func DownloadChunks(chain string, chunksToDownload []types.ChunkRecord, chunkType walk.CacheType, ...)
- func DownloadOneChunk(chain string, man *manifest.Manifest, fileRange base.FileRange) error
- func IsInitialized(chain, required string) error
- func ToBloomPath(pathIn string) string
- func ToIndexPath(pathIn string) string
- func ToStagingPath(pathIn string) string
- type AppearanceResult
- type Bloom
- type Chunk
- type Index
Constants ¶
const ( // The number of bytes in a single BloomByte structure BLOOM_WIDTH_IN_BYTES = (BLOOM_WIDTH_IN_BITS / 8) // The number of bits in a single BloomByte structure BLOOM_WIDTH_IN_BITS = (1048576) // The maximum number of addresses to add to a bloomBytes before creating a new one MAX_ADDRS_IN_BLOOM = 50000 )
const (
// AddrRecordWidth - size of Address Record
AddrRecordWidth = 28
)
const (
// AppRecordWidth - size of Appearance Record
AppRecordWidth = 8
)
const (
// HeaderWidth - size of Header Record
HeaderWidth = 44
)
Variables ¶
var ErrDownloadError = errors.New("download error")
var ErrFailedLocalFileRemoval = errors.New("failed to remove local file")
var ErrIncorrectHash = errors.New("incorrect header hash")
var ErrIncorrectMagic = errors.New("incorrect magic number")
var ErrNotInitialized = errors.New("index not initialized")
var ErrUserHitControlC = errors.New("user hit control + c")
var ErrWriteToDiscError = errors.New("write to disc error")
Functions ¶
func DownloadChunks ¶
func DownloadChunks(chain string, chunksToDownload []types.ChunkRecord, chunkType walk.CacheType, poolSize int, progressChannel progressChan)
DownloadChunks downloads, unzips and saves the chunk of type indicated by chunkType for each chunk in chunks. ProgressMsg is reported to progressChannel.
func DownloadOneChunk ¶
DownloadOneChunk a filename to an index portion, finds the correspoding CID (hash) entry in the manifest, and downloads the index chunk to the local drive
func IsInitialized ¶
IsInitialized returns an error if the version in the header is not as requested
func ToBloomPath ¶
ToBloomPath returns a path pointing to the bloom filter given either a path to itself or its associated index data
func ToIndexPath ¶
ToIndexPath returns a path pointing to the index portion
func ToStagingPath ¶
ToStagingPath returns a path pointing to the staging folder given either a neighboring path
Types ¶
type AppearanceResult ¶
type AppearanceResult struct {
Address base.Address
Range base.FileRange
AppRecords *[]types.AppRecord
Err error
}
AppearanceResult carries the appearances found in a single Index for the given address.
type Bloom ¶
type Bloom struct {
File *os.File
SizeOnDisc int64
Range base.FileRange
HeaderSize int64
Header bloomHeader
Count uint32 // Do not change the size of this field, it's stored on disc
Blooms []bloomBytes
}
Bloom structures contain an array of bloomBytes each BLOOM_WIDTH_IN_BYTES wide. A new bloomBytes is added to the Bloom when around MAX_ADDRS_IN_BLOOM addresses has been added. These Adaptive Bloom Filters allow us to maintain a near-constant false-positive rate at the expense of slightly larger bloom filters than might be expected.
func OpenBloom ¶
OpenBloom returns a newly initialized bloom filter. The bloom filter's file pointer is open (if there have been no errors) and its header data has been read into memory. The array has been created with enough space for Count blooms but has not been read from disc. The file remains open for reading (if there is no error) and is positioned at the start of the file.
func (*Bloom) InsertAddress ¶
InsertAddress adds an address to the bloom filter.
type Chunk ¶
The Chunk data structure consists of three parts. A FileRange, a Index structure, and a Bloom that carries set membership information for the Index.
func OpenChunk ¶
OpenChunk returns a fully initialized index chunk. The path argument may point to either a bloom filter file or the index data file. Either will work. The bloom filter file must exist and will be opened for reading and its header will be read into memory, but the filter itself is not. The index data file need not exist (it will be downloaded later if the bloom indicates that its needed). If the index file does exist, however, it will be opened for reading and its header will be read into memory, but the index data itself will not be.
func (*Chunk) Close ¶
func (chunk *Chunk) Close()
Close closes both the bloom filter file pointer and the index data file pointer (if they are open)
type Index ¶
type Index struct {
File *os.File
Header indexHeader
Range base.FileRange
AddrTableStart int64
AppTableStart int64
}
Index is one part of the two part Chunk (the other part is the Bloom)
Each Index contains a HeaderRecord followed by two tables: the AddressTable and a related AppearanceTable.
The HeaderRecord (44 bytes long) contains a four-byte magic number (`0xdeadbeef` -- to indicate we're reading a file of the correct type), a 32-byte hash representing the file's version, and two 4-byte integers representing the number of records in the AddressTable (nAddresses) and the number of records in the AppearanceTable (nAppearances) respectively.
The AddressTable has nAddresses records, one for each address that appears in any block between Start to End inclusive. These addresses are found using the TrueBlocks Appearance Finder algorithm. Each AddressRecord consists of a 20-byte address followed by two 4-byte integers representing the Offset into the AppearanceTable where this address's list of appearances starts followed by the Count of appearance records for the given address.
The AppearanceTable contains nAppeeances pairs of <blockNumber.transactionId> pairs arranged by the Offset and Count pairs found in the corresponding AddressTable records.
func OpenIndex ¶
OpenIndex returns an Index with an opened file pointer to the given fileName. The HeaderRecord for the chunk has been populated and the file position to the two tables are ready for use.
func (*Index) ReadAppearances ¶
func (chunk *Index) ReadAppearances(address base.Address) *AppearanceResult
ReadAppearances searches an already-opened Index for the given address. Returns a AppearanceResult or nil
Source Files