index

Documentation

Overview

Package index manages content indices.

Index

• Constants
• Variables
• type Builder
  • func (b Builder) Add(i Info)
  • func (b Builder) Build(output io.Writer, version int) error
  • func (b Builder) BuildShards(indexVersion int, stable bool, shardSize int) ([]gather.Bytes, func(), error)
  • func (b Builder) BuildStable(output io.Writer, version int) error
  • func (b Builder) Clone() Builder
• type BuilderCreator
• type FormatV1
• type FormatV2
• type ID
  • func IDFromHash(prefix IDPrefix, hash []byte) (ID, error)
  • func ParseID(s string) (ID, error)
  • func (i ID) Append(out []byte) []byte
  • func (i ID) AppendToLogBuffer(sb *logging.Buffer)
  • func (i ID) HasPrefix() bool
  • func (i ID) Hash() []byte
  • func (i ID) MarshalJSON() ([]byte, error)
  • func (i ID) Prefix() IDPrefix
  • func (i ID) String() string
  • func (i *ID) UnmarshalJSON(v []byte) error
• type IDPrefix
  • func (p IDPrefix) ValidateSingle() error
• type IDRange
  • func PrefixRange(prefix IDPrefix) IDRange
  • func (r IDRange) Contains(id ID) bool
• type Index
  • func Open(data []byte, closer func() error, v1PerContentOverhead func() int) (Index, error)
• type Info
  • func (i *Info) Less(other llrb.Item) bool
  • func (i Info) Timestamp() time.Time
• type Merged
  • func (m Merged) ApproximateCount() int
  • func (m Merged) Close() error
  • func (m Merged) GetInfo(id ID, result *Info) (bool, error)
  • func (m Merged) Iterate(r IDRange, cb func(i Info) error) error
• type OneUseBuilder
  • func NewOneUseBuilder() *OneUseBuilder
  • func (b *OneUseBuilder) Add(i Info)
  • func (b *OneUseBuilder) BuildShards(indexVersion int, stable bool, shardSize int) ([]gather.Bytes, func(), error)
  • func (b *OneUseBuilder) BuildStable(output io.Writer, version int) error
  • func (b *OneUseBuilder) Length() int

Constants

const (
	// Version1 identifies version 1 of the index, without content-level compression.
	Version1 = 1
)

const (
	// Version2 identifies version 2 of the index, supporting content-level compression.
	Version2 = 2
)

Variables

var AllIDs = IDRange{"", maxIDCharacterPlus1}

AllIDs is an IDRange that contains all valid IDs.

var AllNonPrefixedIDs = IDRange{"0", "g"}

AllNonPrefixedIDs is an IDRange that contains all valid IDs non-prefixed IDs ('0' .. 'f').

var AllPrefixedIDs = IDRange{"g", maxIDCharacterPlus1}

AllPrefixedIDs is an IDRange that contains all valid IDs prefixed IDs ('g' .. 'z').

var EmptyID = ID{} //nolint:gochecknoglobals

EmptyID represents empty content ID.

Types

type Builder

type Builder map[ID]Info

Builder prepares and writes content index.

func (Builder) Add

func (b Builder) Add(i Info)

Add adds a new entry to the builder or conditionally replaces it if the timestamp is greater.

func (Builder) Build

func (b Builder) Build(output io.Writer, version int) error

Build writes the pack index to the provided output.

func (Builder) BuildShards

func (b Builder) BuildShards(indexVersion int, stable bool, shardSize int) ([]gather.Bytes, func(), error)

BuildShards builds the set of index shards ensuring no more than the provided number of contents are in each index. Returns shard bytes and function to clean up after the shards have been written.

func (Builder) BuildStable

func (b Builder) BuildStable(output io.Writer, version int) error

BuildStable writes the pack index to the provided output.

func (Builder) Clone

func (b Builder) Clone() Builder

Clone returns a deep Clone of the Builder.

type BuilderCreator

type BuilderCreator interface {
	Add(info Info)
}

BuilderCreator is an interface for caller to add indexes to builders.

type FormatV1

type FormatV1 struct {
	Version    byte   // format version number must be 0x01
	KeySize    byte   // size of each key in bytes
	EntrySize  uint16 // size of each entry in bytes, big-endian
	EntryCount uint32 // number of sorted (key,value) entries that follow

	Entries []struct {
		Key   []byte // key bytes (KeySize)
		Entry []byte // entry bytes (EntrySize)
	}

	ExtraData []byte // extra data
}

FormatV1 describes a format of a single pack index. The actual structure is not used, it's purely for documentation purposes. The struct is byte-aligned.

type FormatV2

type FormatV2 struct {
	Header struct {
		Version           byte   // format version number must be 0x02
		KeySize           byte   // size of each key in bytes
		EntrySize         uint16 // size of each entry in bytes, big-endian
		EntryCount        uint32 // number of sorted (key,value) entries that follow
		EntriesOffset     uint32 // offset where `Entries` begins
		FormatInfosOffset uint32 // offset where `Formats` begins
		NumFormatInfos    uint32
		PacksOffset       uint32 // offset where `Packs` begins
		NumPacks          uint32
		BaseTimestamp     uint32 // base timestamp in unix seconds
	}

	Entries []struct {
		Key   []byte // key bytes (KeySize)
		Entry []byte // entry bytes (EntrySize)
	}

	// each entry contains offset+length of the name of the pack blob, so that each entry can refer to the index
	// and it resolves to a name.
	Packs []struct {
		PackNameLength byte   // length of the filename
		PackNameOffset uint32 // offset to data (within extra data)
	}

	// each entry represents unique content format.
	Formats []indexV2FormatInfo

	ExtraData []byte // extra data
}

FormatV2 describes a format of a single pack index. The actual structure is not used, it's purely for documentation purposes. The struct is byte-aligned.

type ID

type ID struct {
	// contains filtered or unexported fields
}

ID is an identifier of content in content-addressable storage.

func IDFromHash

func IDFromHash(prefix IDPrefix, hash []byte) (ID, error)

IDFromHash constructs content ID from a prefix and a hash.

func ParseID

func ParseID(s string) (ID, error)

ParseID parses and validates the content ID.

func (ID) Append

func (i ID) Append(out []byte) []byte

Append appends content ID to the slice.

func (ID) AppendToLogBuffer

func (i ID) AppendToLogBuffer(sb *logging.Buffer)

AppendToLogBuffer appends content ID to log buffer.

func (ID) HasPrefix

func (i ID) HasPrefix() bool

HasPrefix determines if the given ID has a non-empty prefix.

func (ID) Hash

func (i ID) Hash() []byte

Hash returns the hash part of content ID.

func (ID) MarshalJSON

func (i ID) MarshalJSON() ([]byte, error)

MarshalJSON implements JSON serialization.

func (ID) Prefix

func (i ID) Prefix() IDPrefix

Prefix returns a one-character prefix of a content ID or an empty string.

func (ID) String

func (i ID) String() string

String returns a string representation of ID.

func (*ID) UnmarshalJSON

func (i *ID) UnmarshalJSON(v []byte) error

UnmarshalJSON implements JSON deserialization.

type IDPrefix

type IDPrefix string

IDPrefix represents a content ID prefix (empty string or single character between 'g' and 'z').

func (IDPrefix) ValidateSingle

func (p IDPrefix) ValidateSingle() error

ValidateSingle returns an error if a given prefix is invalid.

type IDRange

type IDRange struct {
	StartID IDPrefix // inclusive
	EndID   IDPrefix // exclusive
}

IDRange represents a range of IDs.

func PrefixRange

func PrefixRange(prefix IDPrefix) IDRange

PrefixRange returns ID range that contains all IDs with a given prefix.

func (IDRange) Contains

func (r IDRange) Contains(id ID) bool

Contains determines whether given ID is in the range.

type Index

type Index interface {
	io.Closer
	ApproximateCount() int
	GetInfo(contentID ID, result *Info) (bool, error)

	// invoked the provided callback for all entries such that entry.ID >= startID and entry.ID < endID
	Iterate(r IDRange, cb func(Info) error) error
}

Index is a read-only index of packed contents.

func Open

func Open(data []byte, closer func() error, v1PerContentOverhead func() int) (Index, error)

Open reads an Index from a given reader. The caller must call Close() when the index is no longer used.

type Info

type Info struct {
	PackBlobID          blob.ID              `json:"packFile,omitempty"`
	TimestampSeconds    int64                `json:"time"`
	OriginalLength      uint32               `json:"originalLength"`
	PackedLength        uint32               `json:"length"`
	PackOffset          uint32               `json:"packOffset,omitempty"`
	CompressionHeaderID compression.HeaderID `json:"compression,omitempty"`
	ContentID           ID                   `json:"contentID"`
	Deleted             bool                 `json:"deleted"`
	FormatVersion       byte                 `json:"formatVersion"`
	EncryptionKeyID     byte                 `json:"encryptionKeyID,omitempty"`
}

Info is an implementation of Info based on a structure.

func (*Info) Less

func (i *Info) Less(other llrb.Item) bool

Less compares with another *Info by their ContentID and return true if the current one is smaller.

func (Info) Timestamp

func (i Info) Timestamp() time.Time

Timestamp implements the Info interface.

type Merged

type Merged []Index

Merged is an implementation of Index that transparently merges returns from underlying Indexes.

func (Merged) ApproximateCount

func (m Merged) ApproximateCount() int

ApproximateCount implements Index interface.

func (Merged) Close

func (m Merged) Close() error

Close closes all underlying indexes.

func (Merged) GetInfo

func (m Merged) GetInfo(id ID, result *Info) (bool, error)

GetInfo returns information about a single content. If a content is not found, returns (false,nil).

func (Merged) Iterate

func (m Merged) Iterate(r IDRange, cb func(i Info) error) error

Iterate invokes the provided callback for all unique content IDs in the underlying sources until either all contents have been visited or until an error is returned by the callback.

type OneUseBuilder

type OneUseBuilder struct {
	// contains filtered or unexported fields
}

OneUseBuilder prepares and writes content index for epoch index compaction.

func NewOneUseBuilder

func NewOneUseBuilder() *OneUseBuilder

NewOneUseBuilder create a new instance of OneUseBuilder.

func (*OneUseBuilder) Add

func (b *OneUseBuilder) Add(i Info)

Add adds a new entry to the builder or conditionally replaces it if the timestamp is greater.

func (*OneUseBuilder) BuildShards

func (b *OneUseBuilder) BuildShards(indexVersion int, stable bool, shardSize int) ([]gather.Bytes, func(), error)

BuildShards builds the set of index shards ensuring no more than the provided number of contents are in each index. Returns shard bytes and function to clean up after the shards have been written.

func (*OneUseBuilder) BuildStable

func (b *OneUseBuilder) BuildStable(output io.Writer, version int) error

BuildStable writes the pack index to the provided output.

func (*OneUseBuilder) Length

func (b *OneUseBuilder) Length() int

Length returns the number of indexes in the current builder.