common

Documentation

Index

• Constants
• func SortParts(parts []Part)
• type OriginFS
• type Part
  • func PartsDifference(a, b []Part) []Part
  • func PartsIntersect(a, b []Part) []Part
  • func (p *Part) ParseFromRemotePath(remotePath string) bool
  • func (p *Part) RemotePath(prefix string) string
  • func (p *Part) String() string
• type RemoteFS

Constants

const MaxPartSize = 128 * 1024 * 1024

MaxPartSize is the maximum size for each part.

The MaxPartSize reduces bandwidth usage during retires on network errors when transferring multi-TB files.

Functions

func SortParts

func SortParts(parts []Part)

SortParts sorts parts by (Path, Offset)

Types

type OriginFS

type OriginFS interface {
	// MustStop must be called when the RemoteFS is no longer needed.
	MustStop()

	// String must return human-readable representation of OriginFS.
	String() string

	// ListParts must return all the parts for the OriginFS.
	ListParts() ([]Part, error)
}

OriginFS is an interface for remote origin filesystem.

This filesystem is used for performing server-side file copies instead of uploading data from local filesystem.

type Part

type Part struct {
	// Path is the path to file for backup.
	Path string

	// FileSize is the size of the whole file for the given part.
	FileSize uint64

	// Offset is offset in the file to backup.
	Offset uint64

	// Size is the size of the part to backup starting from Offset.
	Size uint64

	// ActualSize is the actual size of the part.
	//
	// The part is considered broken if it isn't equal to Size.
	// Such a part must be removed from remote storage.
	ActualSize uint64
}

Part is an atomic unit for transfer during backup / restore.

Each source file can be split into parts with up to MaxPartSize sizes.

func PartsDifference

func PartsDifference(a, b []Part) []Part

PartsDifference returns a - b

func PartsIntersect

func PartsIntersect(a, b []Part) []Part

PartsIntersect returns the intersection of a and b

func (*Part) ParseFromRemotePath

func (p *Part) ParseFromRemotePath(remotePath string) bool

ParseFromRemotePath parses p from remotePath.

Returns true on success.

func (*Part) RemotePath

func (p *Part) RemotePath(prefix string) string

RemotePath returns remote path for the part p and the given prefix.

func (*Part) String

func (p *Part) String() string

String returns human-readable representation of the part.

type RemoteFS

type RemoteFS interface {
	// MustStop must be called when the RemoteFS is no longer needed.
	MustStop()

	// String must return human-readable representation of RemoteFS.
	String() string

	// ListParts must return all the parts for the RemoteFS.
	ListParts() ([]Part, error)

	// DeletePart must delete part p from RemoteFS.
	DeletePart(p Part) error

	// RemoveEmptyDirs must recursively remove empty directories in RemoteFS.
	RemoveEmptyDirs() error

	// CopyPart must copy part p from dstFS to RemoteFS.
	CopyPart(dstFS OriginFS, p Part) error

	// DownloadPart must download part p from RemoteFS to w.
	DownloadPart(p Part, w io.Writer) error

	// UploadPart must upload part p from r to RemoteFS.
	UploadPart(p Part, r io.Reader) error

	// DeleteFile deletes filePath at RemoteFS
	DeleteFile(filePath string) error

	// CreateFile creates filePath at RemoteFS and puts data into it.
	CreateFile(filePath string, data []byte) error

	// HasFile returns true if filePath exists at RemoteFS.
	HasFile(filePath string) (bool, error)
}

RemoteFS is a filesystem where backups are stored.