index

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 22, 2023 License: BSD-2-Clause Imports: 17 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var IndexFull = func(idx *Index, compress bool) bool {
	idx.m.Lock()
	defer idx.m.Unlock()

	debug.Log("checking whether index %p is full", idx)

	var blobs uint
	for typ := range idx.byType {
		blobs += idx.byType[typ].len()
	}
	age := time.Since(idx.created)
	var maxBlobs uint
	if compress {
		maxBlobs = indexMaxBlobsCompressed
	} else {
		maxBlobs = indexMaxBlobs
	}

	switch {
	case age >= indexMaxAge:
		debug.Log("index %p is old enough", idx, age)
		return true
	case blobs >= maxBlobs:
		debug.Log("index %p has %d blobs", idx, blobs)
		return true
	}

	debug.Log("index %p only has %d blobs and is too young (%v)", idx, blobs, age)
	return false

}

IndexFull returns true iff the index is "full enough" to be saved as a preliminary index.

Functions

func ForAllIndexes 

func ForAllIndexes(ctx context.Context, repo restic.Repository,
	fn func(id restic.ID, index *Index, oldFormat bool, err error) error) error

ForAllIndexes loads all index files in parallel and calls the given callback. It is guaranteed that the function is not run concurrently. If the callback returns an error, this function is cancelled and also returns that error.

func SaveIndex 

func SaveIndex(ctx context.Context, repo restic.SaverUnpacked, index *Index) (restic.ID, error)

SaveIndex saves an index in the repository.

Types

type EachByPackResult 

type EachByPackResult struct {
	PackID restic.ID
	Blobs  []restic.Blob
}

type Index 

type Index struct {
	// contains filtered or unexported fields
}

Index holds lookup tables for id -> pack.

func DecodeIndex 

func DecodeIndex(buf []byte, id restic.ID) (idx *Index, oldFormat bool, err error)

DecodeIndex unserializes an index from buf.

func NewIndex 

func NewIndex() *Index

NewIndex returns a new index.

func TestMergeIndex 

func TestMergeIndex(t testing.TB, mi *MasterIndex) ([]*Index, int)

func (*Index) AddToSupersedes 

func (idx *Index) AddToSupersedes(ids ...restic.ID) error

AddToSupersedes adds the ids to the list of indexes superseded by this index. If the index has already been finalized, an error is returned.

func (*Index) Dump 

func (idx *Index) Dump(w io.Writer) error

Dump writes the pretty-printed JSON representation of the index to w.

func (*Index) Each 

func (idx *Index) Each(ctx context.Context, fn func(restic.PackedBlob))

Each passes all blobs known to the index to the callback fn. This blocks any modification of the index.

func (*Index) EachByPack 

func (idx *Index) EachByPack(ctx context.Context, packBlacklist restic.IDSet) <-chan EachByPackResult

EachByPack returns a channel that yields all blobs known to the index grouped by packID but ignoring blobs with a packID in packPlacklist for finalized indexes. This filtering is used when rebuilding the index where we need to ignore packs from the finalized index which have been re-read into a non-finalized index. When the context is cancelled, the background goroutine terminates. This blocks any modification of the index.

func (*Index) Encode 

func (idx *Index) Encode(w io.Writer) error

Encode writes the JSON serialization of the index to the writer w.

func (*Index) Final 

func (idx *Index) Final() bool

Final returns true iff the index is already written to the repository, it is finalized.

func (*Index) Finalize 

func (idx *Index) Finalize()

Finalize sets the index to final.

func (*Index) Has 

func (idx *Index) Has(bh restic.BlobHandle) bool

Has returns true iff the id is listed in the index.

func (*Index) IDs 

func (idx *Index) IDs() (restic.IDs, error)

IDs returns the IDs of the index, if available. If the index is not yet finalized, an error is returned.

func (*Index) Lookup 

func (idx *Index) Lookup(bh restic.BlobHandle, pbs []restic.PackedBlob) []restic.PackedBlob

Lookup queries the index for the blob ID and returns all entries including duplicates. Adds found entries to blobs and returns the result.

func (*Index) LookupSize 

func (idx *Index) LookupSize(bh restic.BlobHandle) (plaintextLength uint, found bool)

LookupSize returns the length of the plaintext content of the blob with the given id.

func (*Index) Packs 

func (idx *Index) Packs() restic.IDSet

Packs returns all packs in this index

func (*Index) SetID 

func (idx *Index) SetID(id restic.ID) error

SetID sets the ID the index has been written to. This requires that Finalize() has been called before, otherwise an error is returned.

func (*Index) StorePack 

func (idx *Index) StorePack(id restic.ID, blobs []restic.Blob)

StorePack remembers the ids of all blobs of a given pack in the index

func (*Index) Supersedes 

func (idx *Index) Supersedes() restic.IDs

Supersedes returns the list of indexes this index supersedes, if any.

type MasterIndex 

type MasterIndex struct {
	// contains filtered or unexported fields
}

MasterIndex is a collection of indexes and IDs of chunks that are in the process of being saved.

func NewMasterIndex 

func NewMasterIndex() *MasterIndex

NewMasterIndex creates a new master index.

func (*MasterIndex) AddPending 

func (mi *MasterIndex) AddPending(bh restic.BlobHandle) bool

AddPending adds a given blob to list of pending Blobs Before doing so it checks if this blob is already known. Returns true if adding was successful and false if the blob was already known

func (*MasterIndex) Each 

func (mi *MasterIndex) Each(ctx context.Context, fn func(restic.PackedBlob))

Each runs fn on all blobs known to the index. When the context is cancelled, the index iteration return immediately. This blocks any modification of the index.

func (*MasterIndex) Has 

func (mi *MasterIndex) Has(bh restic.BlobHandle) bool

Has queries all known Indexes for the ID and returns the first match. Also returns true if the ID is pending.

func (*MasterIndex) IDs 

func (mi *MasterIndex) IDs() restic.IDSet

IDs returns the IDs of all indexes contained in the index.

func (*MasterIndex) Insert 

func (mi *MasterIndex) Insert(idx *Index)

Insert adds a new index to the MasterIndex.

func (*MasterIndex) ListPacks 

func (mi *MasterIndex) ListPacks(ctx context.Context, packs restic.IDSet) <-chan restic.PackBlobs

ListPacks returns the blobs of the specified pack files grouped by pack file.

func (*MasterIndex) Lookup 

func (mi *MasterIndex) Lookup(bh restic.BlobHandle) (pbs []restic.PackedBlob)

Lookup queries all known Indexes for the ID and returns all matches.

func (*MasterIndex) LookupSize 

func (mi *MasterIndex) LookupSize(bh restic.BlobHandle) (uint, bool)

LookupSize queries all known Indexes for the ID and returns the first match.

func (*MasterIndex) MarkCompressed 

func (mi *MasterIndex) MarkCompressed()

func (*MasterIndex) MergeFinalIndexes 

func (mi *MasterIndex) MergeFinalIndexes() error

MergeFinalIndexes merges all final indexes together. After calling, there will be only one big final index in MasterIndex containing all final index contents. Indexes that are not final are left untouched. This merging can only be called after all index files are loaded - as removing of superseded index contents is only possible for unmerged indexes.

func (*MasterIndex) Packs 

func (mi *MasterIndex) Packs(packBlacklist restic.IDSet) restic.IDSet

Packs returns all packs that are covered by the index. If packBlacklist is given, those packs are only contained in the resulting IDSet if they are contained in a non-final (newly written) index.

func (*MasterIndex) Save 

func (mi *MasterIndex) Save(ctx context.Context, repo restic.SaverUnpacked, packBlacklist restic.IDSet, extraObsolete restic.IDs, p *progress.Counter) (obsolete restic.IDSet, err error)

Save saves all known indexes to index files, leaving out any packs whose ID is contained in packBlacklist from finalized indexes. The new index contains the IDs of all known indexes in the "supersedes" field. The IDs are also returned in the IDSet obsolete. After calling this function, you should remove the obsolete index files.

func (*MasterIndex) SaveFullIndex 

func (mi *MasterIndex) SaveFullIndex(ctx context.Context, r restic.SaverUnpacked) error

SaveFullIndex saves all full indexes in the backend.

func (*MasterIndex) SaveIndex 

func (mi *MasterIndex) SaveIndex(ctx context.Context, r restic.SaverUnpacked) error

SaveIndex saves all new indexes in the backend.

func (*MasterIndex) StorePack 

func (mi *MasterIndex) StorePack(id restic.ID, blobs []restic.Blob)

StorePack remembers the id and pack in the index.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.