archiver

package
v0.9.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 6, 2018 License: BSD-2-Clause Imports: 22 Imported by: 0

Documentation

Overview

Package archiver contains the code which reads files, splits them into chunks and saves the data to the repository.

An Archiver has a number of worker goroutines handling saving the different data structures to the repository, the details are implemented by the FileSaver, BlobSaver, and TreeSaver types.

The main goroutine (the one calling Snapshot()) traverses the directory tree and delegates all work to these worker pools. They return a type (FutureFile, FutureBlob, and FutureTree) which can be resolved later, by calling Wait() on it.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func TestCreateFiles added in v0.9.0 

func TestCreateFiles(t testing.TB, target string, dir TestDir)

TestCreateFiles creates a directory structure described by dir at target, which must already exist. On Windows, symlinks aren't created.

func TestEnsureFileContent added in v0.9.0 

func TestEnsureFileContent(ctx context.Context, t testing.TB, repo restic.Repository, filename string, node *restic.Node, file TestFile)

TestEnsureFileContent checks if the file in the repo is the same as file.

func TestEnsureFiles added in v0.9.0 

func TestEnsureFiles(t testing.TB, target string, dir TestDir)

TestEnsureFiles tests if the directory structure at target is the same as described in dir.

func TestEnsureSnapshot added in v0.9.0 

func TestEnsureSnapshot(t testing.TB, repo restic.Repository, snapshotID restic.ID, dir TestDir)

TestEnsureSnapshot tests if the snapshot in the repo has exactly the same structure as dir. On Windows, Symlinks are ignored.

func TestEnsureTree added in v0.9.0 

func TestEnsureTree(ctx context.Context, t testing.TB, prefix string, repo restic.Repository, treeID restic.ID, dir TestDir)

TestEnsureTree checks that the tree ID in the repo matches dir. On Windows, Symlinks are ignored.

func TestSnapshot 

func TestSnapshot(t testing.TB, repo restic.Repository, path string, parent *restic.ID) *restic.Snapshot

TestSnapshot creates a new snapshot of path.

func TestWalkFiles added in v0.9.0 

func TestWalkFiles(t testing.TB, target string, dir TestDir, fn TestWalkFunc)

TestWalkFiles runs fn for each file/directory in dir, the filename will be constructed with target as the prefix. Symlinks on Windows are ignored.

Types

type Archiver 

type Archiver struct {
	Repo    restic.Repository
	Select  SelectFunc
	FS      fs.FS
	Options Options

	// Error is called for all errors that occur during backup.
	Error ErrorFunc

	// CompleteItem is called for all files and dirs once they have been
	// processed successfully. The parameter item contains the path as it will
	// be in the snapshot after saving. s contains some statistics about this
	// particular file/dir.
	//
	// CompleteItem may be called asynchronously from several different
	// goroutines!
	CompleteItem func(item string, previous, current *restic.Node, s ItemStats, d time.Duration)

	// StartFile is called when a file is being processed by a worker.
	StartFile func(filename string)

	// CompleteBlob is called for all saved blobs for files.
	CompleteBlob func(filename string, bytes uint64)

	// WithAtime configures if the access time for files and directories should
	// be saved. Enabling it may result in much metadata, so it's off by
	// default.
	WithAtime bool
	// contains filtered or unexported fields
}

Archiver saves a directory structure to the repo.

func New 

func New(repo restic.Repository, fs fs.FS, opts Options) *Archiver

New initializes a new archiver.

func (*Archiver) Save 

func (arch *Archiver) Save(ctx context.Context, snPath, target string, previous *restic.Node) (fn FutureNode, excluded bool, err error)

Save saves a target (file or directory) to the repo. If the item is excluded,this function returns a nil node and error, with excluded set to true.

Errors and completion is needs to be handled by the caller.

snPath is the path within the current snapshot.

func (*Archiver) SaveDir added in v0.9.0 

func (arch *Archiver) SaveDir(ctx context.Context, snPath string, fi os.FileInfo, dir string, previous *restic.Tree) (d FutureTree, err error)

SaveDir stores a directory in the repo and returns the node. snPath is the path within the current snapshot.

func (*Archiver) SaveTree added in v0.9.0 

func (arch *Archiver) SaveTree(ctx context.Context, snPath string, atree *Tree, previous *restic.Tree) (*restic.Tree, error)

SaveTree stores a Tree in the repo, returned is the tree. snPath is the path within the current snapshot.

func (*Archiver) Snapshot 

func (arch *Archiver) Snapshot(ctx context.Context, targets []string, opts SnapshotOptions) (*restic.Snapshot, restic.ID, error)

Snapshot saves several targets and returns a snapshot.

type BlobSaver added in v0.9.0 

type BlobSaver struct {
	// contains filtered or unexported fields
}

BlobSaver concurrently saves incoming blobs to the repo.

func NewBlobSaver added in v0.9.0 

func NewBlobSaver(ctx context.Context, t *tomb.Tomb, repo Saver, workers uint) *BlobSaver

NewBlobSaver returns a new blob. A worker pool is started, it is stopped when ctx is cancelled.

func (*BlobSaver) Save added in v0.9.0 

func (s *BlobSaver) Save(ctx context.Context, t restic.BlobType, buf *Buffer) FutureBlob

Save stores a blob in the repo. It checks the index and the known blobs before saving anything. The second return parameter is true if the blob was previously unknown.

type Buffer added in v0.9.0 

type Buffer struct {
	Data []byte
	Put  func(*Buffer)
}

Buffer is a reusable buffer. After the buffer has been used, Release should be called so the underlying slice is put back into the pool.

func (*Buffer) Release added in v0.9.0 

func (b *Buffer) Release()

Release puts the buffer back into the pool it came from.

type BufferPool added in v0.9.0 

type BufferPool struct {
	// contains filtered or unexported fields
}

BufferPool implements a limited set of reusable buffers.

func NewBufferPool added in v0.9.0 

func NewBufferPool(ctx context.Context, max int, defaultSize int) *BufferPool

NewBufferPool initializes a new buffer pool. When the context is cancelled, all buffers are released. The pool stores at most max items. New buffers are created with defaultSize, buffers that are larger are released and not put back.

func (*BufferPool) Get added in v0.9.0 

func (pool *BufferPool) Get() *Buffer

Get returns a new buffer, either from the pool or newly allocated.

func (*BufferPool) Put added in v0.9.0 

func (pool *BufferPool) Put(b *Buffer)

Put returns a buffer to the pool for reuse.

type CompleteFunc added in v0.9.0 

type CompleteFunc func(*restic.Node, ItemStats)

CompleteFunc is called when the file has been saved.

type ErrorFunc added in v0.9.0 

type ErrorFunc func(file string, fi os.FileInfo, err error) error

ErrorFunc is called when an error during archiving occurs. When nil is returned, the archiver continues, otherwise it aborts and passes the error up the call stack.

type FileSaver added in v0.9.0 

type FileSaver struct {
	CompleteBlob func(filename string, bytes uint64)

	NodeFromFileInfo func(filename string, fi os.FileInfo) (*restic.Node, error)
	// contains filtered or unexported fields
}

FileSaver concurrently saves incoming files to the repo.

func NewFileSaver added in v0.9.0 

func NewFileSaver(ctx context.Context, t *tomb.Tomb, fs fs.FS, save SaveBlobFn, pol chunker.Pol, fileWorkers, blobWorkers uint) *FileSaver

NewFileSaver returns a new file saver. A worker pool with fileWorkers is started, it is stopped when ctx is cancelled.

func (*FileSaver) Save added in v0.9.0 

func (s *FileSaver) Save(ctx context.Context, snPath string, file fs.File, fi os.FileInfo, start func(), complete CompleteFunc) FutureFile

Save stores the file f and returns the data once it has been completed. The file is closed by Save.

type FutureBlob added in v0.9.0 

type FutureBlob struct {
	// contains filtered or unexported fields
}

FutureBlob is returned by SaveBlob and will return the data once it has been processed.

func (*FutureBlob) ID added in v0.9.0 

func (s *FutureBlob) ID() restic.ID

ID returns the ID of the blob after it has been saved.

func (*FutureBlob) Known added in v0.9.0 

func (s *FutureBlob) Known() bool

Known returns whether or not the blob was already known.

func (*FutureBlob) Length added in v0.9.0 

func (s *FutureBlob) Length() int

Length returns the length of the blob.

func (*FutureBlob) Wait added in v0.9.0 

func (s *FutureBlob) Wait(ctx context.Context)

Wait blocks until the result is available or the context is cancelled.

type FutureFile added in v0.9.0 

type FutureFile struct {
	// contains filtered or unexported fields
}

FutureFile is returned by Save and will return the data once it has been processed.

func (*FutureFile) Err added in v0.9.0 

func (s *FutureFile) Err() error

Err returns the error in case an error occurred.

func (*FutureFile) Node added in v0.9.0 

func (s *FutureFile) Node() *restic.Node

Node returns the node once it is available.

func (*FutureFile) Stats added in v0.9.0 

func (s *FutureFile) Stats() ItemStats

Stats returns the stats for the file once they are available.

func (*FutureFile) Wait added in v0.9.0 

func (s *FutureFile) Wait(ctx context.Context)

Wait blocks until the result of the save operation is received or ctx is cancelled.

type FutureNode added in v0.9.0 

type FutureNode struct {
	// contains filtered or unexported fields
}

FutureNode holds a reference to a node, FutureFile, or FutureTree.

type FutureTree added in v0.9.0 

type FutureTree struct {
	// contains filtered or unexported fields
}

FutureTree is returned by Save and will return the data once it has been processed.

func (*FutureTree) Node added in v0.9.0 

func (s *FutureTree) Node() *restic.Node

Node returns the node.

func (*FutureTree) Stats added in v0.9.0 

func (s *FutureTree) Stats() ItemStats

Stats returns the stats for the file.

func (*FutureTree) Wait added in v0.9.0 

func (s *FutureTree) Wait(ctx context.Context)

Wait blocks until the data has been received or ctx is cancelled.

type IndexUploader added in v0.9.0 

type IndexUploader struct {
	restic.Repository

	// Start is called when an index is to be uploaded.
	Start func()

	// Complete is called when uploading an index has finished.
	Complete func(id restic.ID)
}

IndexUploader polls the repo for full indexes and uploads them.

func (IndexUploader) Upload added in v0.9.0 

func (u IndexUploader) Upload(ctx, shutdown context.Context, interval time.Duration) error

Upload periodically uploads full indexes to the repo. When shutdown is cancelled, the last index upload will finish and then Upload returns.

type ItemStats added in v0.9.0 

type ItemStats struct {
	DataBlobs int    // number of new data blobs added for this item
	DataSize  uint64 // sum of the sizes of all new data blobs
	TreeBlobs int    // number of new tree blobs added for this item
	TreeSize  uint64 // sum of the sizes of all new tree blobs
}

ItemStats collects some statistics about a particular file or directory.

func (*ItemStats) Add added in v0.9.0 

func (s *ItemStats) Add(other ItemStats)

Add adds other to the current ItemStats.

type Options added in v0.9.0 

type Options struct {
	// FileReadConcurrency sets how many files are read in concurrently. If
	// it's set to zero, at most two files are read in concurrently (which
	// turned out to be a good default for most situations).
	FileReadConcurrency uint

	// SaveBlobConcurrency sets how many blobs are hashed and saved
	// concurrently. If it's set to zero, the default is the number of CPUs
	// available in the system.
	SaveBlobConcurrency uint

	// SaveTreeConcurrency sets how many trees are marshalled and saved to the
	// repo concurrently.
	SaveTreeConcurrency uint
}

Options is used to configure the archiver.

func (Options) ApplyDefaults added in v0.9.0 

func (o Options) ApplyDefaults() Options

ApplyDefaults returns a copy of o with the default options set for all unset fields.

type SaveBlobFn added in v0.9.0 

type SaveBlobFn func(context.Context, restic.BlobType, *Buffer) FutureBlob

SaveBlobFn saves a blob to a repo.

type Saver added in v0.9.0 

type Saver interface {
	SaveBlob(ctx context.Context, t restic.BlobType, data []byte, id restic.ID) (restic.ID, error)
	Index() restic.Index
}

Saver allows saving a blob.

type ScanStats added in v0.9.0 

type ScanStats struct {
	Files, Dirs, Others uint
	Bytes               uint64
}

ScanStats collect statistics.

type Scanner added in v0.9.0 

type Scanner struct {
	FS     fs.FS
	Select SelectFunc
	Error  ErrorFunc
	Result func(item string, s ScanStats)
}

Scanner traverses the targets and calls the function Result with cumulated stats concerning the files and folders found. Select is used to decide which items should be included. Error is called when an error occurs.

func NewScanner added in v0.9.0 

func NewScanner(fs fs.FS) *Scanner

NewScanner initializes a new Scanner.

func (*Scanner) Scan added in v0.9.0 

func (s *Scanner) Scan(ctx context.Context, targets []string) error

Scan traverses the targets. The function Result is called for each new item found, the complete result is also returned by Scan.

type SelectFunc added in v0.9.0 

type SelectFunc func(item string, fi os.FileInfo) bool

SelectFunc returns true for all items that should be included (files and dirs). If false is returned, files are ignored and dirs are not even walked.

type SnapshotOptions added in v0.9.0 

type SnapshotOptions struct {
	Tags           []string
	Hostname       string
	Excludes       []string
	Time           time.Time
	ParentSnapshot restic.ID
}

SnapshotOptions collect attributes for a new snapshot.

type TestDir added in v0.9.0 

type TestDir map[string]interface{}

TestDir describes a directory structure to create for a test.

func (TestDir) String added in v0.9.0 

func (d TestDir) String() string

type TestFile added in v0.9.0 

type TestFile struct {
	Content string
}

TestFile describes a file created for a test.

func (TestFile) String added in v0.9.0 

func (f TestFile) String() string
type TestSymlink struct {
	Target string
}

TestSymlink describes a symlink created for a test.

func (TestSymlink) String added in v0.9.0 

func (s TestSymlink) String() string

type TestWalkFunc added in v0.9.0 

type TestWalkFunc func(path string, item interface{}) error

TestWalkFunc is used by TestWalkFiles to traverse the dir. When an error is returned, traversal stops and the surrounding test is marked as failed.

type Tree added in v0.9.0 

type Tree struct {
	Nodes        map[string]Tree
	Path         string // where the files/dirs to be saved are found
	FileInfoPath string // where the dir can be found that is not included itself, but its subdirs
	Root         string // parent directory of the tree
}

Tree recursively defines how a snapshot should look like when archived.

When `Path` is set, this is a leaf node and the contents of `Path` should be inserted at this point in the tree.

The attribute `Root` is used to distinguish between files/dirs which have the same name, but live in a separate directory on the local file system.

`FileInfoPath` is used to extract metadata for intermediate (=non-leaf) trees.

func NewTree added in v0.9.0 

func NewTree(fs fs.FS, targets []string) (*Tree, error)

NewTree creates a Tree from the target files/directories.

func (*Tree) Add added in v0.9.0 

func (t *Tree) Add(fs fs.FS, path string) error

Add adds a new file or directory to the tree.

func (Tree) String added in v0.9.0 

func (t Tree) String() string

type TreeSaver added in v0.9.0 

type TreeSaver struct {
	// contains filtered or unexported fields
}

TreeSaver concurrently saves incoming trees to the repo.

func NewTreeSaver added in v0.9.0 

func NewTreeSaver(ctx context.Context, t *tomb.Tomb, treeWorkers uint, saveTree func(context.Context, *restic.Tree) (restic.ID, ItemStats, error), errFn ErrorFunc) *TreeSaver

NewTreeSaver returns a new tree saver. A worker pool with treeWorkers is started, it is stopped when ctx is cancelled.

func (*TreeSaver) Save added in v0.9.0 

func (s *TreeSaver) Save(ctx context.Context, snPath string, node *restic.Node, nodes []FutureNode) FutureTree

Save stores the dir d and returns the data once it has been completed.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.