backup

package
v1.1.0-alpha-8 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 28, 2017 License: Apache-2.0 Imports: 28 Imported by: 2

Documentation

Index

Constants

View Source 
const (
	CryptoKeySize = 32 // 256-bit key
)

static sizes

View Source 
const (
	// DefaultBlockSize is the default block size,
	// used for the deduped blocks stored as a backup.
	DefaultBlockSize = 1024 * 128 // 128 KiB
)

Variables

View Source 
var (
	// ErrInvalidConfig is the error returned when giving an invalid
	// storage (driver) config to the NewStorageDriver function.
	ErrInvalidConfig = errors.New("invalid storage driver config")

	// ErrDataDidNotExist is returned from a ServerDriver's Getter
	// method in case the requested data does not exist on the server.
	ErrDataDidNotExist = errors.New("requested data did not exist")
)
View Source 
var (
	// ErrNilResource is returned when for a given config function
	// a nil resource was given by the user.
	ErrNilResource = errors.New("invalid nil resource given")
)

Functions

func Decrypt 

func Decrypt(key *CryptoKey, src io.Reader, dst io.Writer) error

Decrypt a given cipher text, previously encrypted using AES256 in Galois Counter Mode.

func Encrypt 

func Encrypt(key *CryptoKey, src io.Reader, dst io.Writer) error

Encrypt a given plain text using AES256 in Galois Counter Mode, with a random nonce.

func Export 

func Export(ctx context.Context, cfg Config) error

Export a block storage to aa FTP Server, in a secure and space efficient manner, in order to provide a backup (snapshot) for later usage.

func Import 

func Import(ctx context.Context, cfg Config) error

Import a block storage from a FTP Server, decrypting and decompressing its blocks on the go.

Types

type CompressionType 

type CompressionType uint8

CompressionType defines a type of a compression. 

const (
	// LZ4Compression represents the LZ4 Compression Type,
	// and is also the Default (nil) value of the Compression Type.
	// See https://github.com/pierrec/lz4 for more information.
	LZ4Compression CompressionType = iota
	// XZCompression represents the XZ Compression Type.
	// See https://github.com/ulikunitz/xz for more information.
	XZCompression
)

func (*CompressionType) Set 

func (ct *CompressionType) Set(str string) error

Set implements Flag.Set

func (*CompressionType) String 

func (ct *CompressionType) String() string

String implements Stringer.String

func (*CompressionType) Type 

func (ct *CompressionType) Type() string

Type implements PValue.Type

type Compressor 

type Compressor interface {
	Compress(src io.Reader, dst io.Writer) error
}

Compressor defines the API for any type of compressor.

func LZ4Compressor 

func LZ4Compressor() Compressor

LZ4Compressor creates an LZ4 Compressor. See `LZ4Compression` for more information.

func NewCompressor 

func NewCompressor(ct CompressionType) (Compressor, error)

NewCompressor automatically creates the correct compressor (if possible), based on the given compression type.

func XZCompressor 

func XZCompressor() Compressor

XZCompressor creates an XZ Compressor. See `XZCompression` for more information.

type Config 

type Config struct {
	// Required: VdiskID to export from or import into
	VdiskID string
	// Optional: ID of the snapshot (same as VdiskID by default)
	SnapshotID string

	// Optional: Snapshot BlockSize (128KiB by default)
	BlockSize int64

	// Required: SourceConfig to configure the storage with
	BlockStorageConfig config.SourceConfig
	// Required: BackupStorageConfig used to configure the backup storage driver.
	BackupStorageConfig StorageConfig

	// Optional: Amount of jobs (goroutines) to run simultaneously
	//           (to import/export in parallel)
	//           By default it equals the amount of CPUs available.
	JobCount int

	// Type of Compression to use for compressing/decompressing.
	// Note: this should be the same value for an import/export pair
	CompressionType CompressionType
	// CryptoKey to use for encryption/decryption.
	// Note: this should be the same value for an import/export pair
	CryptoKey CryptoKey

	// Optional: Only used for exporting at the moment.
	// When true, a new deduped map will be created in case
	// the existing deduped map couldn't be loaded.
	// This can happen due to the fact that the existing map's data is corrupt,
	// or the data was encrypted/compressed using a different
	// key/compressionType than the one given.
	Force bool
}

Config used to export/import a backup.

type CryptoKey 

type CryptoKey [CryptoKeySize]byte

CryptoKey defines the type of a CryptoKey

func (*CryptoKey) Set 

func (key *CryptoKey) Set(value string) error

Set implements Value.Set

func (*CryptoKey) String 

func (key *CryptoKey) String() string

String implements Value.String

func (*CryptoKey) Type 

func (key *CryptoKey) Type() string

Type implements PValue.Type

type Decompressor 

type Decompressor interface {
	Decompress(src io.Reader, dst io.Writer) error
}

Decompressor defines the API for any type of decompressor.

func LZ4Decompressor 

func LZ4Decompressor() Decompressor

LZ4Decompressor creates an LZ4 Decompressor. See `LZ4Compression` for more information.

func NewDecompressor 

func NewDecompressor(ct CompressionType) (Decompressor, error)

NewDecompressor automatically creates the correct decompression (if possible), based on the given compression type.

func XZDecompressor 

func XZDecompressor() Decompressor

XZDecompressor creates an XZ Decompressor. See `XZCompression` for more information.

type Decrypter 

type Decrypter interface {
	Decrypt(src io.Reader, dst io.Writer) error
}

Decrypter defines the API, which allows you to decrypt a given cipher text, which was previously encrypted by the Encrypter which acts as the counterpart of this interface.

func NewDecrypter 

func NewDecrypter(key *CryptoKey) (Decrypter, error)

NewDecrypter creates an object using the given private key, which allows you to decrypt cipher text, which was previously encrypted using AES256 in Galois Counter Mode.

type DedupedMap 

type DedupedMap struct {
	// contains filtered or unexported fields
}

DedupedMap contains all hashes for a vdisk's backup, where each hash is mapped to its (export) block index. NOTE: DedupedMap is not thread-safe, 

and should only be used on one goroutine at a time.

func DeserializeDedupedMap 

func DeserializeDedupedMap(key *CryptoKey, ct CompressionType, src io.Reader) (*DedupedMap, error)

DeserializeDedupedMap allows you to deserialize a deduped map from a given reader. It is expected that all the data in the reader is available, and is compressed and (only than) encrypted. This function will attempt to decrypt and decompress the read data, using the given private (AES) key and compression type. The given compression type and private key has to match the information, used to serialize this DedupedMap in the first place. See `DedupedMap` for more information.

func ExistingOrNewDedupedMap 

func ExistingOrNewDedupedMap(id string, src StorageDriver, key *CryptoKey, ct CompressionType, force bool) (*DedupedMap, error)

ExistingOrNewDedupedMap tries to first fetch an existing deduped map from a given server, if it doesn't exist yet, a new one will be created in-memory instead. If it did exist already, it will be decrypted, decompressed and loaded in-memory as a DedupedMap. When `force` is `true`, a new map will be created, even if one existed already but couldn't be loaded. When `force` is `false`, and a map exists but can't be a loaded, the error of why it couldn't be loaded, is returned instead.

func LoadDedupedMap 

func LoadDedupedMap(id string, src StorageDriver, key *CryptoKey, ct CompressionType) (*DedupedMap, error)

LoadDedupedMap a deduped map from a given (backup) server.

func NewDedupedMap 

func NewDedupedMap() *DedupedMap

NewDedupedMap creates a new deduped map, which contains all the metadata stored for a(n) (exported) backup. See `DedupedMap` for more information.

func (*DedupedMap) GetHash 

func (dm *DedupedMap) GetHash(index int64) (zerodisk.Hash, bool)

GetHash returns the hash which is mapped to the given (export block) index. `false` is returned in case no hash is mapped to the given (export block) index.

func (*DedupedMap) Serialize 

func (dm *DedupedMap) Serialize(key *CryptoKey, ct CompressionType, dst io.Writer) error

Serialize allows you to write all data of this map in a binary encoded manner, to the given writer. The encoded data will be compressed and encrypted before being writen to the given writer. You can re-load this map in memory using the `DeserializeDedupedMap` function.

func (*DedupedMap) SetHash 

func (dm *DedupedMap) SetHash(index int64, hash zerodisk.Hash) bool

SetHash sets the given hash, mapped to the given (export block) index. If there is already a hash mapped to the given (export block) index, and the hash equals the given hash, the given hash won't be used and `false` wil be returned. Otherwise the given hash is mapped to the given index and `true“ will be returned.

type Encrypter 

type Encrypter interface {
	Encrypt(src io.Reader, dst io.Writer) error
}

Encrypter defines the API, which allows you to encrypt a given plain text into cipher text. By default we use AES256 in Galois Counter Mode.

func NewEncrypter 

func NewEncrypter(key *CryptoKey) (Encrypter, error)

NewEncrypter creates an object using the given private key, which allows you to encrypt plain text using AES256 in Galois Counter Mode.

type FTPStorageConfig 

type FTPStorageConfig struct {
	// Address of the FTP Server
	Address string

	// Optional: username of Authorized account
	//           for the given FTP server
	Username string
	// Optional: password of Authorized account
	//           for the given FTP server
	Password string

	RootDir string // optional
}

FTPStorageConfig is used to configure and create an FTP (Storage) Driver.

func NewFTPStorageConfig 

func NewFTPStorageConfig(data string) (cfg FTPStorageConfig, err error)

NewFTPStorageConfig creates a new FTP Storage Config, based on a given string

func (*FTPStorageConfig) Set 

func (cfg *FTPStorageConfig) Set(str string) error

Set implements flag.Value.Set

func (*FTPStorageConfig) String 

func (cfg *FTPStorageConfig) String() string

String implements Stringer.String

func (*FTPStorageConfig) Type 

func (cfg *FTPStorageConfig) Type() string

Type implements PFlag.Type

type StorageConfig 

type StorageConfig struct {
	// The resource used to create the specific storage.
	// Type = local -> Resource defines root directory path.
	// Type = ftp -> Resource defines ftp configuration.
	Resource interface{}
	// Defines the type of (backup) storage to
	// write/read the backup to/from.
	StorageType StorageType
}

StorageConfig is the configuration used to create a (backup) storage based on a given type and resource.

func NewStorageConfig 

func NewStorageConfig(data string) (cfg StorageConfig, err error)

NewStorageConfig creates a new storage config by implicitly infering the storage type based on the given data, and based on it use the data as the storage's config resource.

func (*StorageConfig) Set 

func (cfg *StorageConfig) Set(value string) (err error)

Set implements flag.Value.Set

func (*StorageConfig) String 

func (cfg *StorageConfig) String() string

String implements Stringer.String

func (*StorageConfig) Type 

func (cfg *StorageConfig) Type() string

Type implements pflag.Value.Type

type StorageDriver 

type StorageDriver interface {
	SetDedupedBlock(hash zerodisk.Hash, r io.Reader) error
	SetDedupedMap(id string, r io.Reader) error

	GetDedupedBlock(hash zerodisk.Hash, w io.Writer) error
	GetDedupedMap(id string, w io.Writer) error

	Close() error
}

StorageDriver defines the API of a (storage) driver, which allows us to read/write from/to a (backup) storage, the deduped blocks and map which form a backup.

func FTPStorageDriver 

func FTPStorageDriver(cfg FTPStorageConfig) (StorageDriver, error)

FTPStorageDriver ceates a driver which allows you to read/write deduped blocks/map from/to a FTP server.

func LocalStorageDriver 

func LocalStorageDriver(root string) (StorageDriver, error)

LocalStorageDriver ceates a driver which allows you to read/write deduped blocks/map from/to the local file system.

func NewStorageDriver 

func NewStorageDriver(cfg StorageConfig) (StorageDriver, error)

NewStorageDriver creates a new storage driver, using the given Storage (Driver) Config.

type StorageType 

type StorageType uint8

StorageType specifies the type of (backup) storage to export to or import from. 

const (
	// LocalStorageType defines a local file storage,
	// meaning a backup would be stored/loaded,
	// directly on/from the local file storage.
	// This is also the default (backup) storage.
	LocalStorageType StorageType = 0
	// FTPStorageType defines the FTP storage,
	// meaning a backup would be stored/loaded,
	// on/from an FTP Server.
	// This is the (backup) storage used in production.
	FTPStorageType StorageType = 1
)

func (*StorageType) Set 

func (st *StorageType) Set(str string) error

Set allows you to set this Storage Type using a raw string. Options: {ftp, local}

func (StorageType) String 

func (st StorageType) String() string

String returns the name of the Config Source Type

func (*StorageType) Type 

func (st *StorageType) Type() string

Type returns the flag type for a StorageType.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.