common

Documentation

Index

• Constants
• Variables
• func SortParts(parts []Part)
• func ToCanonicalPath(path string) string
• type OriginFS
• type Part
  • func PartsDifference(a, b []Part) []Part
  • func PartsIntersect(a, b []Part) []Part
  • func (p *Part) LocalPath(dir string) string
  • func (p *Part) ParseFromRemotePath(remotePath string) bool
  • func (p *Part) RemotePath(prefix string) string
  • func (p *Part) String() string
• type RemoteFS

Constants

const MaxPartSize = 1024 * 1024 * 1024

MaxPartSize is the maximum size for each part.

The MaxPartSize reduces bandwidth usage during retires on network errors when transferring multi-TB files.

Variables

var (
	// DeleteAllObjectVersions is a flag for whether to prune previous object versions when deleting an object.
	DeleteAllObjectVersions = flag.Bool("deleteAllObjectVersions", false, "Whether to prune previous object versions when deleting an object. "+
		"By default, when object storage has versioning enabled deleting the file removes only current version. "+
		"This option forces removal of all previous versions. "+
		"See: https://docs.victoriametrics.com/vmbackup.html#permanent-deletion-of-objects-in-s3-compatible-storages")
)

Functions

func SortParts

func SortParts(parts []Part)

SortParts sorts parts by (Path, Offset)

func ToCanonicalPath

func ToCanonicalPath(path string) string

ToCanonicalPath returns canonical path by replacing local directory separators with `/`.

Types

type OriginFS

type OriginFS interface {
	// MustStop must be called when the RemoteFS is no longer needed.
	MustStop()

	// String must return human-readable representation of OriginFS.
	String() string

	// ListParts must return all the parts for the OriginFS.
	ListParts() ([]Part, error)
}

OriginFS is an interface for remote origin filesystem.

This filesystem is used for performing server-side file copies instead of uploading data from local filesystem.

type Part

type Part struct {
	// Path is the path to file for backup.
	//
	// Path must consistently use `/` as directory separator.
	// Use ToCanonicalPath() function for converting local directory separators to `/`.
	Path string

	// FileSize is the size of the whole file for the given part.
	FileSize uint64

	// Offset is offset in the file to backup.
	Offset uint64

	// Size is the size of the part to backup starting from Offset.
	Size uint64

	// ActualSize is the actual size of the part.
	//
	// The part is considered broken if it isn't equal to Size.
	// Such a part must be removed from remote storage.
	ActualSize uint64
}

Part is an atomic unit for transfer during backup / restore.

Each source file can be split into parts with up to MaxPartSize sizes.

func PartsDifference

func PartsDifference(a, b []Part) []Part

PartsDifference returns a - b

func PartsIntersect

func PartsIntersect(a, b []Part) []Part

PartsIntersect returns the intersection of a and b

func (*Part) LocalPath

func (p *Part) LocalPath(dir string) string

LocalPath returns local path for p at the given dir.

func (*Part) ParseFromRemotePath

func (p *Part) ParseFromRemotePath(remotePath string) bool

ParseFromRemotePath parses p from remotePath.

Returns true on success.

remotePath must be in canonical form received from ToCanonicalPath().

func (*Part) RemotePath

func (p *Part) RemotePath(prefix string) string

RemotePath returns remote path for the part p and the given prefix.

func (*Part) String

func (p *Part) String() string

String returns human-readable representation of the part.

type RemoteFS

type RemoteFS interface {
	// MustStop must be called when the RemoteFS is no longer needed.
	MustStop()

	// String must return human-readable representation of RemoteFS.
	String() string

	// ListParts must return all the parts for the RemoteFS.
	ListParts() ([]Part, error)

	// DeletePart must delete part p from RemoteFS.
	DeletePart(p Part) error

	// RemoveEmptyDirs must recursively remove empty directories in RemoteFS.
	RemoveEmptyDirs() error

	// CopyPart must copy part p from dstFS to RemoteFS.
	CopyPart(dstFS OriginFS, p Part) error

	// DownloadPart must download part p from RemoteFS to w.
	DownloadPart(p Part, w io.Writer) error

	// UploadPart must upload part p from r to RemoteFS.
	UploadPart(p Part, r io.Reader) error

	// DeleteFile deletes filePath at RemoteFS
	DeleteFile(filePath string) error

	// CreateFile creates filePath at RemoteFS and puts data into it.
	CreateFile(filePath string, data []byte) error

	// HasFile returns true if filePath exists at RemoteFS.
	HasFile(filePath string) (bool, error)

	// ReadFile returns file contents at the given filePath.
	ReadFile(filePath string) ([]byte, error)
}

RemoteFS is a filesystem where backups are stored.