archiver

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 22, 2023 License: BSD-2-Clause Imports: 20 Imported by: 0

Documentation

Overview

Package archiver contains the code which reads files, splits them into chunks and saves the data to the repository.

An Archiver has a number of worker goroutines handling saving the different data structures to the repository, the details are implemented by the FileSaver, BlobSaver, and TreeSaver types.

The main goroutine (the one calling Snapshot()) traverses the directory tree and delegates all work to these worker pools. They return a type (FutureFile, FutureBlob, and FutureTree) which can be resolved later, by calling Wait() on it.

Index

Constants

View Source 
const (
	ChangeIgnoreCtime = 1 << iota
	ChangeIgnoreInode
)

Flags for the ChangeIgnoreFlags bitfield.

Variables

This section is empty.

Functions

func TestCreateFiles 

func TestCreateFiles(t testing.TB, target string, dir TestDir)

TestCreateFiles creates a directory structure described by dir at target, which must already exist. On Windows, symlinks aren't created.

func TestEnsureFileContent 

func TestEnsureFileContent(ctx context.Context, t testing.TB, repo restic.Repository, filename string, node *restic.Node, file TestFile)

TestEnsureFileContent checks if the file in the repo is the same as file.

func TestEnsureFiles 

func TestEnsureFiles(t testing.TB, target string, dir TestDir)

TestEnsureFiles tests if the directory structure at target is the same as described in dir.

func TestEnsureSnapshot 

func TestEnsureSnapshot(t testing.TB, repo restic.Repository, snapshotID restic.ID, dir TestDir)

TestEnsureSnapshot tests if the snapshot in the repo has exactly the same structure as dir. On Windows, Symlinks are ignored.

func TestEnsureTree 

func TestEnsureTree(ctx context.Context, t testing.TB, prefix string, repo restic.Repository, treeID restic.ID, dir TestDir)

TestEnsureTree checks that the tree ID in the repo matches dir. On Windows, Symlinks are ignored.

func TestSnapshot 

func TestSnapshot(t testing.TB, repo restic.Repository, path string, parent *restic.ID) *restic.Snapshot

TestSnapshot creates a new snapshot of path.

func TestWalkFiles 

func TestWalkFiles(t testing.TB, target string, dir TestDir, fn TestWalkFunc)

TestWalkFiles runs fn for each file/directory in dir, the filename will be constructed with target as the prefix. Symlinks on Windows are ignored.

Types

type Archiver 

type Archiver struct {
	Repo         restic.Repository
	SelectByName SelectByNameFunc
	Select       SelectFunc
	FS           fs.FS
	Options      Options

	// Error is called for all errors that occur during backup.
	Error ErrorFunc

	// CompleteItem is called for all files and dirs once they have been
	// processed successfully. The parameter item contains the path as it will
	// be in the snapshot after saving. s contains some statistics about this
	// particular file/dir.
	//
	// Once reading a file has completed successfully (but not saving it yet),
	// CompleteItem will be called with current == nil.
	//
	// CompleteItem may be called asynchronously from several different
	// goroutines!
	CompleteItem func(item string, previous, current *restic.Node, s ItemStats, d time.Duration)

	// StartFile is called when a file is being processed by a worker.
	StartFile func(filename string)

	// CompleteBlob is called for all saved blobs for files.
	CompleteBlob func(bytes uint64)

	// WithAtime configures if the access time for files and directories should
	// be saved. Enabling it may result in much metadata, so it's off by
	// default.
	WithAtime bool

	// Flags controlling change detection. See doc/040_backup.rst for details.
	ChangeIgnoreFlags uint
	// contains filtered or unexported fields
}

Archiver saves a directory structure to the repo.

func New 

func New(repo restic.Repository, fs fs.FS, opts Options) *Archiver

New initializes a new archiver.

func (*Archiver) Save 

func (arch *Archiver) Save(ctx context.Context, snPath, target string, previous *restic.Node) (fn FutureNode, excluded bool, err error)

Save saves a target (file or directory) to the repo. If the item is excluded, this function returns a nil node and error, with excluded set to true.

Errors and completion needs to be handled by the caller.

snPath is the path within the current snapshot.

func (*Archiver) SaveDir 

func (arch *Archiver) SaveDir(ctx context.Context, snPath string, dir string, fi os.FileInfo, previous *restic.Tree, complete CompleteFunc) (d FutureNode, err error)

SaveDir stores a directory in the repo and returns the node. snPath is the path within the current snapshot.

func (*Archiver) SaveTree 

func (arch *Archiver) SaveTree(ctx context.Context, snPath string, atree *Tree, previous *restic.Tree, complete CompleteFunc) (FutureNode, int, error)

SaveTree stores a Tree in the repo, returned is the tree. snPath is the path within the current snapshot.

func (*Archiver) Snapshot 

func (arch *Archiver) Snapshot(ctx context.Context, targets []string, opts SnapshotOptions) (*restic.Snapshot, restic.ID, error)

Snapshot saves several targets and returns a snapshot.

type BlobSaver 

type BlobSaver struct {
	// contains filtered or unexported fields
}

BlobSaver concurrently saves incoming blobs to the repo.

func NewBlobSaver 

func NewBlobSaver(ctx context.Context, wg *errgroup.Group, repo Saver, workers uint) *BlobSaver

NewBlobSaver returns a new blob. A worker pool is started, it is stopped when ctx is cancelled.

func (*BlobSaver) Save 

func (s *BlobSaver) Save(ctx context.Context, t restic.BlobType, buf *Buffer, cb func(res SaveBlobResponse))

Save stores a blob in the repo. It checks the index and the known blobs before saving anything. It takes ownership of the buffer passed in.

func (*BlobSaver) TriggerShutdown 

func (s *BlobSaver) TriggerShutdown()

type Buffer 

type Buffer struct {
	Data []byte
	// contains filtered or unexported fields
}

Buffer is a reusable buffer. After the buffer has been used, Release should be called so the underlying slice is put back into the pool.

func (*Buffer) Release 

func (b *Buffer) Release()

Release puts the buffer back into the pool it came from.

type BufferPool 

type BufferPool struct {
	// contains filtered or unexported fields
}

BufferPool implements a limited set of reusable buffers.

func NewBufferPool 

func NewBufferPool(max int, defaultSize int) *BufferPool

NewBufferPool initializes a new buffer pool. The pool stores at most max items. New buffers are created with defaultSize. Buffers that have grown larger are not put back.

func (*BufferPool) Get 

func (pool *BufferPool) Get() *Buffer

Get returns a new buffer, either from the pool or newly allocated.

type CompleteFunc 

type CompleteFunc func(*restic.Node, ItemStats)

CompleteFunc is called when the file has been saved.

type ErrorFunc 

type ErrorFunc func(file string, err error) error

ErrorFunc is called when an error during archiving occurs. When nil is returned, the archiver continues, otherwise it aborts and passes the error up the call stack.

type FileSaver 

type FileSaver struct {
	CompleteBlob func(bytes uint64)

	NodeFromFileInfo func(snPath, filename string, fi os.FileInfo) (*restic.Node, error)
	// contains filtered or unexported fields
}

FileSaver concurrently saves incoming files to the repo.

func NewFileSaver 

func NewFileSaver(ctx context.Context, wg *errgroup.Group, save SaveBlobFn, pol chunker.Pol, fileWorkers, blobWorkers uint) *FileSaver

NewFileSaver returns a new file saver. A worker pool with fileWorkers is started, it is stopped when ctx is cancelled.

func (*FileSaver) Save 

func (s *FileSaver) Save(ctx context.Context, snPath string, target string, file fs.File, fi os.FileInfo, start func(), completeReading func(), complete CompleteFunc) FutureNode

Save stores the file f and returns the data once it has been completed. The file is closed by Save. completeReading is only called if the file was read successfully. complete is always called. If completeReading is called, then this will always happen before calling complete.

func (*FileSaver) TriggerShutdown 

func (s *FileSaver) TriggerShutdown()

type FutureNode 

type FutureNode struct {
	// contains filtered or unexported fields
}

FutureNode holds a reference to a channel that returns a FutureNodeResult or a reference to an already existing result. If the result is available immediatelly, then storing a reference directly requires less memory than using the indirection via a channel.

type ItemStats 

type ItemStats struct {
	DataBlobs      int    // number of new data blobs added for this item
	DataSize       uint64 // sum of the sizes of all new data blobs
	DataSizeInRepo uint64 // sum of the bytes added to the repo (including compression and crypto overhead)
	TreeBlobs      int    // number of new tree blobs added for this item
	TreeSize       uint64 // sum of the sizes of all new tree blobs
	TreeSizeInRepo uint64 // sum of the bytes added to the repo (including compression and crypto overhead)
}

ItemStats collects some statistics about a particular file or directory.

func (*ItemStats) Add 

func (s *ItemStats) Add(other ItemStats)

Add adds other to the current ItemStats.

type Options 

type Options struct {
	// ReadConcurrency sets how many files are read in concurrently. If
	// it's set to zero, at most two files are read in concurrently (which
	// turned out to be a good default for most situations).
	ReadConcurrency uint

	// SaveBlobConcurrency sets how many blobs are hashed and saved
	// concurrently. If it's set to zero, the default is the number of CPUs
	// available in the system.
	SaveBlobConcurrency uint

	// SaveTreeConcurrency sets how many trees are marshalled and saved to the
	// repo concurrently.
	SaveTreeConcurrency uint
}

Options is used to configure the archiver.

func (Options) ApplyDefaults 

func (o Options) ApplyDefaults() Options

ApplyDefaults returns a copy of o with the default options set for all unset fields.

type SaveBlobFn 

type SaveBlobFn func(context.Context, restic.BlobType, *Buffer, func(res SaveBlobResponse))

SaveBlobFn saves a blob to a repo.

type SaveBlobResponse 

type SaveBlobResponse struct {
	// contains filtered or unexported fields
}

type Saver 

type Saver interface {
	SaveBlob(ctx context.Context, t restic.BlobType, data []byte, id restic.ID, storeDuplicate bool) (restic.ID, bool, int, error)
}

Saver allows saving a blob.

type ScanStats 

type ScanStats struct {
	Files, Dirs, Others uint
	Bytes               uint64
}

ScanStats collect statistics.

type Scanner 

type Scanner struct {
	FS           fs.FS
	SelectByName SelectByNameFunc
	Select       SelectFunc
	Error        ErrorFunc
	Result       func(item string, s ScanStats)
}

Scanner traverses the targets and calls the function Result with cumulated stats concerning the files and folders found. Select is used to decide which items should be included. Error is called when an error occurs.

func NewScanner 

func NewScanner(fs fs.FS) *Scanner

NewScanner initializes a new Scanner.

func (*Scanner) Scan 

func (s *Scanner) Scan(ctx context.Context, targets []string) error

Scan traverses the targets. The function Result is called for each new item found, the complete result is also returned by Scan.

type SelectByNameFunc 

type SelectByNameFunc func(item string) bool

SelectByNameFunc returns true for all items that should be included (files and dirs). If false is returned, files are ignored and dirs are not even walked.

type SelectFunc 

type SelectFunc func(item string, fi os.FileInfo) bool

SelectFunc returns true for all items that should be included (files and dirs). If false is returned, files are ignored and dirs are not even walked.

type SnapshotOptions 

type SnapshotOptions struct {
	Tags           restic.TagList
	Hostname       string
	Excludes       []string
	Time           time.Time
	ParentSnapshot *restic.Snapshot
	ProgramVersion string
}

SnapshotOptions collect attributes for a new snapshot.

type TestDir 

type TestDir map[string]interface{}

TestDir describes a directory structure to create for a test.

func (TestDir) String 

func (d TestDir) String() string

type TestFile 

type TestFile struct {
	Content string
}

TestFile describes a file created for a test.

func (TestFile) String 

func (f TestFile) String() string
type TestSymlink struct {
	Target string
}

TestSymlink describes a symlink created for a test.

func (TestSymlink) String 

func (s TestSymlink) String() string

type TestWalkFunc 

type TestWalkFunc func(path string, item interface{}) error

TestWalkFunc is used by TestWalkFiles to traverse the dir. When an error is returned, traversal stops and the surrounding test is marked as failed.

type Tree 

type Tree struct {
	Nodes        map[string]Tree
	Path         string // where the files/dirs to be saved are found
	FileInfoPath string // where the dir can be found that is not included itself, but its subdirs
	Root         string // parent directory of the tree
}

Tree recursively defines how a snapshot should look like when archived.

When `Path` is set, this is a leaf node and the contents of `Path` should be inserted at this point in the tree.

The attribute `Root` is used to distinguish between files/dirs which have the same name, but live in a separate directory on the local file system.

`FileInfoPath` is used to extract metadata for intermediate (=non-leaf) trees.

func NewTree 

func NewTree(fs fs.FS, targets []string) (*Tree, error)

NewTree creates a Tree from the target files/directories.

func (*Tree) Add 

func (t *Tree) Add(fs fs.FS, path string) error

Add adds a new file or directory to the tree.

func (Tree) Leaf 

func (t Tree) Leaf() bool

Leaf returns true if this is a leaf node, which means Path is set to a non-empty string and the contents of Path should be inserted at this point in the tree.

func (Tree) NodeNames 

func (t Tree) NodeNames() []string

NodeNames returns the sorted list of subtree names.

func (Tree) String 

func (t Tree) String() string

type TreeSaver 

type TreeSaver struct {
	// contains filtered or unexported fields
}

TreeSaver concurrently saves incoming trees to the repo.

func NewTreeSaver 

func NewTreeSaver(ctx context.Context, wg *errgroup.Group, treeWorkers uint, saveBlob func(ctx context.Context, t restic.BlobType, buf *Buffer, cb func(res SaveBlobResponse)), errFn ErrorFunc) *TreeSaver

NewTreeSaver returns a new tree saver. A worker pool with treeWorkers is started, it is stopped when ctx is cancelled.

func (*TreeSaver) Save 

func (s *TreeSaver) Save(ctx context.Context, snPath string, target string, node *restic.Node, nodes []FutureNode, complete CompleteFunc) FutureNode

Save stores the dir d and returns the data once it has been completed.

func (*TreeSaver) TriggerShutdown 

func (s *TreeSaver) TriggerShutdown()

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.