repository

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 22, 2023 License: BSD-2-Clause Imports: 34 Imported by: 0

Documentation

Overview

Package repository implements a restic repository on top of a backend. In the following the abstractions used for this package are listed. More information can be found in the restic design document.

File

A file is a named handle for some data saved in the backend. For the local backend, this corresponds to actual files saved to disk. Usually, the SHA256 hash of the content is used for a file's name (hexadecimal, in lower-case ASCII characters). An exception is the file `config`. Most files are encrypted before being saved in a backend. This means that the name is the hash of the ciphertext.

Blob

A blob is a number of bytes that has a type (data or tree). Blobs are identified by an ID, which is the SHA256 hash of the blobs' contents. One or more blobs are bundled together in a Pack and then saved to the backend. Blobs are always encrypted before being bundled in a Pack.

Pack

A Pack is a File in the backend that contains one or more (encrypted) blobs, followed by a header at the end of the Pack. The header is encrypted and contains the ID, type, length and offset for each blob contained in the Pack.

Index

Constants

View Source 
const DefaultPackSize = 16 * 1024 * 1024
View Source 
const MaxPackSize = 128 * 1024 * 1024
View Source 
const MaxStreamBufferSize = 4 * 1024 * 1024
View Source 
const MinPackSize = 4 * 1024 * 1024
View Source 
const TestChunkerPol = chunker.Pol(0x3DA3358B4DC173)

Variables

View Source 
var (
	// ErrNoKeyFound is returned when no key for the repository could be decrypted.
	ErrNoKeyFound = errors.New("wrong password or no key found")

	// ErrMaxKeysReached is returned when the maximum number of keys was checked and no key could be found.
	ErrMaxKeysReached = errors.New("maximum number of keys reached")
)
View Source 
var (
	// KDFTimeout specifies the maximum runtime for the KDF.
	KDFTimeout = 500 * time.Millisecond

	// KDFMemory limits the memory the KDF is allowed to use.
	KDFMemory = 60
)
View Source 
var Params *crypto.Params

Params tracks the parameters used for the KDF. If not set, it will be calibrated on the first run of AddKey().

Functions

func BenchmarkAllVersions 

func BenchmarkAllVersions(b *testing.B, bench VersionedBenchmark)

func Repack 

func Repack(ctx context.Context, repo restic.Repository, dstRepo restic.Repository, packs restic.IDSet, keepBlobs repackBlobSet, p *progress.Counter) (obsoletePacks restic.IDSet, err error)

Repack takes a list of packs together with a list of blobs contained in these packs. Each pack is loaded and the blobs listed in keepBlobs is saved into a new pack. Returned is the list of obsolete packs which can then be removed.

The map keepBlobs is modified by Repack, it is used to keep track of which blobs have been processed.

func StreamPack 

func StreamPack(ctx context.Context, beLoad BackendLoadFn, key *crypto.Key, packID restic.ID, blobs []restic.Blob, handleBlobFn func(blob restic.BlobHandle, buf []byte, err error) error) error

StreamPack loads the listed blobs from the specified pack file. The plaintext blob is passed to the handleBlobFn callback or an error if decryption failed or the blob hash does not match. In case of download errors handleBlobFn might be called multiple times for the same blob. If the callback returns an error, then StreamPack will abort and not retry it.

func TestAllVersions 

func TestAllVersions(t *testing.T, test VersionedTest)

func TestBackend 

func TestBackend(_ testing.TB) restic.Backend

TestBackend returns a fully configured in-memory backend.

func TestOpenLocal 

func TestOpenLocal(t testing.TB, dir string) (r restic.Repository)

TestOpenLocal opens a local repository.

func TestRepository 

func TestRepository(t testing.TB) restic.Repository

TestRepository returns a repository initialized with a test password on an in-memory backend. When the environment variable RESTIC_TEST_REPO is set to a non-existing directory, a local backend is created there and this is used instead. The directory is not removed, but left there for inspection.

func TestRepositoryWithBackend 

func TestRepositoryWithBackend(t testing.TB, be restic.Backend, version uint) restic.Repository

TestRepositoryWithBackend returns a repository initialized with a test password. If be is nil, an in-memory backend is used. A constant polynomial is used for the chunker and low-security test parameters.

func TestRepositoryWithVersion 

func TestRepositoryWithVersion(t testing.TB, version uint) restic.Repository

func TestUseLowSecurityKDFParameters 

func TestUseLowSecurityKDFParameters(t logger)

TestUseLowSecurityKDFParameters configures low-security KDF parameters for testing.

func ZeroChunk 

func ZeroChunk() restic.ID

ZeroChunk computes and returns (cached) the ID of an all-zero chunk with size chunker.MinSize

Types

type BackendLoadFn 

type BackendLoadFn func(ctx context.Context, h restic.Handle, length int, offset int64, fn func(rd io.Reader) error) error

type CompressionMode 

type CompressionMode uint

CompressionMode configures if data should be compressed. 

const (
	CompressionAuto    CompressionMode = 0
	CompressionOff     CompressionMode = 1
	CompressionMax     CompressionMode = 2
	CompressionInvalid CompressionMode = 3
)

Constants for the different compression levels.

func (*CompressionMode) Set 

func (c *CompressionMode) Set(s string) error

Set implements the method needed for pflag command flag parsing.

func (*CompressionMode) String 

func (c *CompressionMode) String() string

func (*CompressionMode) Type 

func (c *CompressionMode) Type() string

type Key 

type Key struct {
	Created  time.Time `json:"created"`
	Username string    `json:"username"`
	Hostname string    `json:"hostname"`

	KDF  string `json:"kdf"`
	N    int    `json:"N"`
	R    int    `json:"r"`
	P    int    `json:"p"`
	Salt []byte `json:"salt"`
	Data []byte `json:"data"`
	// contains filtered or unexported fields
}

Key represents an encrypted master key for a repository.

func AddKey 

func AddKey(ctx context.Context, s *Repository, password, username, hostname string, template *crypto.Key) (*Key, error)

AddKey adds a new key to an already existing repository.

func LoadKey 

func LoadKey(ctx context.Context, s *Repository, id restic.ID) (k *Key, err error)

LoadKey loads a key from the backend.

func OpenKey 

func OpenKey(ctx context.Context, s *Repository, id restic.ID, password string) (*Key, error)

OpenKey tries do decrypt the key specified by name with the given password.

func SearchKey 

func SearchKey(ctx context.Context, s *Repository, password string, maxKeys int, keyHint string) (k *Key, err error)

SearchKey tries to decrypt at most maxKeys keys in the backend with the given password. If none could be found, ErrNoKeyFound is returned. When maxKeys is reached, ErrMaxKeysReached is returned. When setting maxKeys to zero, all keys in the repo are checked.

func (Key) ID 

func (k Key) ID() restic.ID

ID returns an identifier for the key.

func (*Key) String 

func (k *Key) String() string

func (*Key) Valid 

func (k *Key) Valid() bool

Valid tests whether the mac and encryption keys are valid (i.e. not zero)

type Options 

type Options struct {
	Compression CompressionMode
	PackSize    uint
}

type Packer 

type Packer struct {
	*pack.Packer
	// contains filtered or unexported fields
}

Packer holds a pack.Packer together with a hash writer.

type Repository 

type Repository struct {
	Cache *cache.Cache
	// contains filtered or unexported fields
}

Repository is used to access a repository in a backend.

func New 

func New(be restic.Backend, opts Options) (*Repository, error)

New returns a new repository with backend be.

func (*Repository) Backend 

func (r *Repository) Backend() restic.Backend

Backend returns the backend for the repository.

func (*Repository) Close 

func (r *Repository) Close() error

Close closes the repository by closing the backend.

func (*Repository) Config 

func (r *Repository) Config() restic.Config

Config returns the repository configuration.

func (*Repository) Connections 

func (r *Repository) Connections() uint

func (*Repository) CreateIndexFromPacks 

func (r *Repository) CreateIndexFromPacks(ctx context.Context, packsize map[restic.ID]int64, p *progress.Counter) (invalid restic.IDs, err error)

CreateIndexFromPacks creates a new index by reading all given pack files (with sizes). The index is added to the MasterIndex but not marked as finalized. Returned is the list of pack files which could not be read.

func (*Repository) Delete 

func (r *Repository) Delete(ctx context.Context) error

Delete calls backend.Delete() if implemented, and returns an error otherwise.

func (*Repository) DisableAutoIndexUpdate 

func (r *Repository) DisableAutoIndexUpdate()

DisableAutoIndexUpdate deactives the automatic finalization and upload of new indexes once these are full

func (*Repository) Flush 

func (r *Repository) Flush(ctx context.Context) error

Flush saves all remaining packs and the index

func (*Repository) Index 

func (r *Repository) Index() restic.MasterIndex

Index returns the currently used MasterIndex.

func (*Repository) Init 

func (r *Repository) Init(ctx context.Context, version uint, password string, chunkerPolynomial *chunker.Pol) error

Init creates a new master key with the supplied password, initializes and saves the repository config.

func (*Repository) Key 

func (r *Repository) Key() *crypto.Key

Key returns the current master key.

func (*Repository) KeyID 

func (r *Repository) KeyID() restic.ID

KeyID returns the id of the current key in the backend.

func (*Repository) List 

func (r *Repository) List(ctx context.Context, t restic.FileType, fn func(restic.ID, int64) error) error

List runs fn for all files of type t in the repo.

func (*Repository) ListPack 

func (r *Repository) ListPack(ctx context.Context, id restic.ID, size int64) ([]restic.Blob, uint32, error)

ListPack returns the list of blobs saved in the pack id and the length of the the pack header.

func (*Repository) LoadBlob 

func (r *Repository) LoadBlob(ctx context.Context, t restic.BlobType, id restic.ID, buf []byte) ([]byte, error)

LoadBlob loads a blob of type t from the repository. It may use all of buf[:cap(buf)] as scratch space.

func (*Repository) LoadIndex 

func (r *Repository) LoadIndex(ctx context.Context) error

LoadIndex loads all index files from the backend in parallel and stores them in the master index. The first error that occurred is returned.

func (*Repository) LoadUnpacked 

func (r *Repository) LoadUnpacked(ctx context.Context, t restic.FileType, id restic.ID) ([]byte, error)

LoadUnpacked loads and decrypts the file with the given type and ID.

func (*Repository) LookupBlobSize 

func (r *Repository) LookupBlobSize(id restic.ID, tpe restic.BlobType) (uint, bool)

LookupBlobSize returns the size of blob id.

func (*Repository) PackSize 

func (r *Repository) PackSize() uint

PackSize return the target size of a pack file when uploading

func (*Repository) SaveBlob 

func (r *Repository) SaveBlob(ctx context.Context, t restic.BlobType, buf []byte, id restic.ID, storeDuplicate bool) (newID restic.ID, known bool, size int, err error)

SaveBlob saves a blob of type t into the repository. It takes care that no duplicates are saved; this can be overwritten by setting storeDuplicate to true. If id is the null id, it will be computed and returned. Also returns if the blob was already known before. If the blob was not known before, it returns the number of bytes the blob occupies in the repo (compressed or not, including encryption overhead).

func (*Repository) SaveUnpacked 

func (r *Repository) SaveUnpacked(ctx context.Context, t restic.FileType, p []byte) (id restic.ID, err error)

SaveUnpacked encrypts data and stores it in the backend. Returned is the storage hash.

func (*Repository) SearchKey 

func (r *Repository) SearchKey(ctx context.Context, password string, maxKeys int, keyHint string) error

SearchKey finds a key with the supplied password, afterwards the config is read and parsed. It tries at most maxKeys key files in the repo.

func (*Repository) SetDryRun 

func (r *Repository) SetDryRun()

SetDryRun sets the repo backend into dry-run mode.

func (*Repository) SetIndex 

func (r *Repository) SetIndex(i restic.MasterIndex) error

SetIndex instructs the repository to use the given index.

func (*Repository) StartPackUploader 

func (r *Repository) StartPackUploader(ctx context.Context, wg *errgroup.Group)

func (*Repository) UseCache 

func (r *Repository) UseCache(c *cache.Cache)

UseCache replaces the backend with the wrapped cache.

type SavePacker 

type SavePacker interface {
	// contains filtered or unexported methods
}

SavePacker implements saving a pack in the repository.

type VersionedBenchmark 

type VersionedBenchmark func(b *testing.B, version uint)

type VersionedTest 

type VersionedTest func(t *testing.T, version uint)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.