index

package
v0.0.0-...-0265167 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 18, 2024 License: BSD-2-Clause Imports: 20 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var IndexFull = func(idx *Index) bool {
	idx.m.RLock()
	defer idx.m.RUnlock()

	debug.Log("checking whether index %p is full", idx)

	var blobs uint
	for typ := range idx.byType {
		blobs += idx.byType[typ].len()
	}
	age := time.Since(idx.created)

	switch {
	case age >= indexMaxAge:
		debug.Log("index %p is old enough", idx, age)
		return true
	case blobs >= indexMaxBlobs:
		debug.Log("index %p has %d blobs", idx, blobs)
		return true
	}

	debug.Log("index %p only has %d blobs and is too young (%v)", idx, blobs, age)
	return false

}

IndexFull returns true iff the index is "full enough" to be saved as a preliminary index.

Functions

func ForAllIndexes 

func ForAllIndexes(ctx context.Context, lister restic.Lister, repo restic.LoaderUnpacked,
	fn func(id restic.ID, index *Index, oldFormat bool, err error) error) error

ForAllIndexes loads all index files in parallel and calls the given callback. It is guaranteed that the function is not run concurrently. If the callback returns an error, this function is cancelled and also returns that error.

Types

type AssociatedSet 

type AssociatedSet[T any] struct {
	// contains filtered or unexported fields
}

AssociatedSet is a memory efficient implementation of a BlobSet that can store a small data item for each BlobHandle. It relies on a special property of our MasterIndex implementation. A BlobHandle can be permanently identified using an offset that never changes as MasterIndex entries cannot be modified (only added).

The AssociatedSet thus can use an array with the size of the MasterIndex to store its data. Access to an individual entry is possible by looking up the BlobHandle's offset from the MasterIndex.

BlobHandles that are not part of the MasterIndex can be stored by placing them in an overflow set that is expected to be empty in the normal case.

func NewAssociatedSet 

func NewAssociatedSet[T any](mi *MasterIndex) *AssociatedSet[T]

func (*AssociatedSet[T]) Delete 

func (a *AssociatedSet[T]) Delete(bh restic.BlobHandle)

func (*AssociatedSet[T]) For 

func (a *AssociatedSet[T]) For(cb func(bh restic.BlobHandle, val T))

func (*AssociatedSet[T]) Get 

func (a *AssociatedSet[T]) Get(bh restic.BlobHandle) (T, bool)

func (*AssociatedSet[T]) Has 

func (a *AssociatedSet[T]) Has(bh restic.BlobHandle) bool

func (*AssociatedSet[T]) Insert 

func (a *AssociatedSet[T]) Insert(bh restic.BlobHandle)

func (*AssociatedSet[T]) Len 

func (a *AssociatedSet[T]) Len() int

func (*AssociatedSet[T]) List 

func (a *AssociatedSet[T]) List() restic.BlobHandles

List returns a sorted slice of all BlobHandle in the set.

func (*AssociatedSet[T]) Set 

func (a *AssociatedSet[T]) Set(bh restic.BlobHandle, val T)

func (*AssociatedSet[T]) String 

func (a *AssociatedSet[T]) String() string

type EachByPackResult 

type EachByPackResult struct {
	PackID restic.ID
	Blobs  []restic.Blob
}

type Index 

type Index struct {
	// contains filtered or unexported fields
}

Index holds lookup tables for id -> pack.

func DecodeIndex 

func DecodeIndex(buf []byte, id restic.ID) (idx *Index, oldFormat bool, err error)

DecodeIndex unserializes an index from buf.

func NewIndex 

func NewIndex() *Index

NewIndex returns a new index.

func TestMergeIndex 

func TestMergeIndex(t testing.TB, mi *MasterIndex) ([]*Index, int, restic.IDSet)

func (*Index) BlobIndex 

func (idx *Index) BlobIndex(bh restic.BlobHandle) int

func (*Index) Dump 

func (idx *Index) Dump(w io.Writer) error

Dump writes the pretty-printed JSON representation of the index to w.

func (*Index) Each 

func (idx *Index) Each(ctx context.Context, fn func(restic.PackedBlob)) error

Each passes all blobs known to the index to the callback fn. This blocks any modification of the index.

func (*Index) EachByPack 

func (idx *Index) EachByPack(ctx context.Context, packBlacklist restic.IDSet) <-chan EachByPackResult

EachByPack returns a channel that yields all blobs known to the index grouped by packID but ignoring blobs with a packID in packPlacklist for finalized indexes. This filtering is used when rebuilding the index where we need to ignore packs from the finalized index which have been re-read into a non-finalized index. When the context is cancelled, the background goroutine terminates. This blocks any modification of the index.

func (*Index) Encode 

func (idx *Index) Encode(w io.Writer) error

Encode writes the JSON serialization of the index to the writer w.

func (*Index) Final 

func (idx *Index) Final() bool

Final returns true iff the index is already written to the repository, it is finalized.

func (*Index) Finalize 

func (idx *Index) Finalize()

Finalize sets the index to final.

func (*Index) Has 

func (idx *Index) Has(bh restic.BlobHandle) bool

Has returns true iff the id is listed in the index.

func (*Index) IDs 

func (idx *Index) IDs() (restic.IDs, error)

IDs returns the IDs of the index, if available. If the index is not yet finalized, an error is returned.

func (*Index) Len 

func (idx *Index) Len(t restic.BlobType) uint

func (*Index) Lookup 

func (idx *Index) Lookup(bh restic.BlobHandle, pbs []restic.PackedBlob) []restic.PackedBlob

Lookup queries the index for the blob ID and returns all entries including duplicates. Adds found entries to blobs and returns the result.

func (*Index) LookupSize 

func (idx *Index) LookupSize(bh restic.BlobHandle) (plaintextLength uint, found bool)

LookupSize returns the length of the plaintext content of the blob with the given id.

func (*Index) Packs 

func (idx *Index) Packs() restic.IDSet

Packs returns all packs in this index

func (*Index) SaveIndex 

func (idx *Index) SaveIndex(ctx context.Context, repo restic.SaverUnpacked) (restic.ID, error)

SaveIndex saves an index in the repository.

func (*Index) SetID 

func (idx *Index) SetID(id restic.ID) error

SetID sets the ID the index has been written to. This requires that Finalize() has been called before, otherwise an error is returned.

func (*Index) StorePack 

func (idx *Index) StorePack(id restic.ID, blobs []restic.Blob)

StorePack remembers the ids of all blobs of a given pack in the index

type MasterIndex 

type MasterIndex struct {
	// contains filtered or unexported fields
}

MasterIndex is a collection of indexes and IDs of chunks that are in the process of being saved.

func NewMasterIndex 

func NewMasterIndex() *MasterIndex

NewMasterIndex creates a new master index.

func (*MasterIndex) AddPending 

func (mi *MasterIndex) AddPending(bh restic.BlobHandle) bool

AddPending adds a given blob to list of pending Blobs Before doing so it checks if this blob is already known. Returns true if adding was successful and false if the blob was already known

func (*MasterIndex) Each 

func (mi *MasterIndex) Each(ctx context.Context, fn func(restic.PackedBlob)) error

Each runs fn on all blobs known to the index. When the context is cancelled, the index iteration return immediately. This blocks any modification of the index.

func (*MasterIndex) Has 

func (mi *MasterIndex) Has(bh restic.BlobHandle) bool

Has queries all known Indexes for the ID and returns the first match. Also returns true if the ID is pending.

func (*MasterIndex) IDs 

func (mi *MasterIndex) IDs() restic.IDSet

IDs returns the IDs of all indexes contained in the index.

func (*MasterIndex) Insert 

func (mi *MasterIndex) Insert(idx *Index)

Insert adds a new index to the MasterIndex.

func (*MasterIndex) ListPacks 

func (mi *MasterIndex) ListPacks(ctx context.Context, packs restic.IDSet) <-chan restic.PackBlobs

ListPacks returns the blobs of the specified pack files grouped by pack file.

func (*MasterIndex) Load 

func (mi *MasterIndex) Load(ctx context.Context, r restic.ListerLoaderUnpacked, p *progress.Counter, cb func(id restic.ID, idx *Index, oldFormat bool, err error) error) error

func (*MasterIndex) Lookup 

func (mi *MasterIndex) Lookup(bh restic.BlobHandle) (pbs []restic.PackedBlob)

Lookup queries all known Indexes for the ID and returns all matches.

func (*MasterIndex) LookupSize 

func (mi *MasterIndex) LookupSize(bh restic.BlobHandle) (uint, bool)

LookupSize queries all known Indexes for the ID and returns the first match.

func (*MasterIndex) MergeFinalIndexes 

func (mi *MasterIndex) MergeFinalIndexes() error

MergeFinalIndexes merges all final indexes together. After calling, there will be only one big final index in MasterIndex containing all final index contents. Indexes that are not final are left untouched. This merging can only be called after all index files are loaded - as removing of superseded index contents is only possible for unmerged indexes.

func (*MasterIndex) Packs 

func (mi *MasterIndex) Packs(packBlacklist restic.IDSet) restic.IDSet

Packs returns all packs that are covered by the index. If packBlacklist is given, those packs are only contained in the resulting IDSet if they are contained in a non-final (newly written) index.

func (*MasterIndex) Rewrite 

func (mi *MasterIndex) Rewrite(ctx context.Context, repo restic.Unpacked, excludePacks restic.IDSet, oldIndexes restic.IDSet, extraObsolete restic.IDs, opts MasterIndexRewriteOpts) error

Rewrite removes packs whose ID is in excludePacks from all known indexes. It also removes the rewritten index files and those listed in extraObsolete. If oldIndexes is not nil, then only the indexes in this set are processed. This is used by repair index to only rewrite and delete the old indexes.

Must not be called concurrently to any other MasterIndex operation.

func (*MasterIndex) SaveFallback 

func (mi *MasterIndex) SaveFallback(ctx context.Context, repo restic.SaverRemoverUnpacked, excludePacks restic.IDSet, p *progress.Counter) error

SaveFallback saves all known indexes to index files, leaving out any packs whose ID is contained in packBlacklist from finalized indexes. It is only intended for use by prune with the UnsafeRecovery option.

Must not be called concurrently to any other MasterIndex operation.

func (*MasterIndex) SaveFullIndex 

func (mi *MasterIndex) SaveFullIndex(ctx context.Context, r restic.SaverUnpacked) error

SaveFullIndex saves all full indexes in the backend.

func (*MasterIndex) SaveIndex 

func (mi *MasterIndex) SaveIndex(ctx context.Context, r restic.SaverUnpacked) error

SaveIndex saves all new indexes in the backend.

func (*MasterIndex) StorePack 

func (mi *MasterIndex) StorePack(id restic.ID, blobs []restic.Blob)

StorePack remembers the id and pack in the index.

type MasterIndexRewriteOpts 

type MasterIndexRewriteOpts struct {
	SaveProgress   *progress.Counter
	DeleteProgress func() *progress.Counter
	DeleteReport   func(id restic.ID, err error)
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.