content

package
v0.7.0-rc2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 18, 2020 License: Apache-2.0 Imports: 38 Imported by: 4

Documentation

Overview

Package content implements repository support for content-addressable storage.

Index

Constants

View Source 
const (
	PackBlobIDPrefixRegular blob.ID = "p"
	PackBlobIDPrefixSpecial blob.ID = "q"

	FormatLogModule = "kopia/format"
)

Prefixes for pack blobs.

Variables

View Source 
var AllIDs = IDRange{"", maxIDCharacterPlus1}

AllIDs is an IDRange that contains all valid IDs.

View Source 
var AllNonPrefixedIDs = IDRange{"0", "g"}

AllNonPrefixedIDs is an IDRange that contains all valid IDs non-prefixed IDs ('0' .. 'f').

View Source 
var AllPrefixedIDs = IDRange{"g", maxIDCharacterPlus1}

AllPrefixedIDs is an IDRange that contains all valid IDs prefixed IDs ('g' .. 'z').

View Source 
var ErrContentNotFound = errors.New("content not found")

ErrContentNotFound is returned when content is not found.

View Source 
var PackBlobIDPrefixes = []blob.ID{
	PackBlobIDPrefixRegular,
	PackBlobIDPrefixSpecial,
}

PackBlobIDPrefixes contains all possible prefixes for pack blobs.

Functions

func CreateHashAndEncryptor 

func CreateHashAndEncryptor(f *FormattingOptions) (hashing.HashFunc, encryption.Encryptor, error)

CreateHashAndEncryptor returns new hashing and encrypting functions based on the specified formatting options.

func UsingContentCache 

func UsingContentCache(ctx context.Context, enabled bool) context.Context

UsingContentCache returns a derived context that causes content manager to use cache.

func UsingListCache 

func UsingListCache(ctx context.Context, enabled bool) context.Context

UsingListCache returns a derived context that causes content manager to use cache.

func ValidatePrefix added in v0.6.0 

func ValidatePrefix(prefix ID) error

ValidatePrefix returns an error if a given prefix is invalid.

Types

type CachingOptions 

type CachingOptions struct {
	CacheDirectory            string `json:"cacheDirectory,omitempty"`
	MaxCacheSizeBytes         int64  `json:"maxCacheSize,omitempty"`
	MaxMetadataCacheSizeBytes int64  `json:"maxMetadataCacheSize,omitempty"`
	MaxListCacheDurationSec   int    `json:"maxListCacheDuration,omitempty"`
	HMACSecret                []byte `json:"-"`
	// contains filtered or unexported fields
}

CachingOptions specifies configuration of local cache.

func (*CachingOptions) CloneOrDefault added in v0.6.0 

func (c *CachingOptions) CloneOrDefault() *CachingOptions

CloneOrDefault returns a clone of the caching options or empty options for nil.

type CompactOptions 

type CompactOptions struct {
	MaxSmallBlobs     int
	AllIndexes        bool
	DropDeletedBefore time.Time
	DropContents      []ID
}

CompactOptions provides options for compaction.

type Format 

type Format struct {
	Version    byte   // format version number must be 0x01
	KeySize    byte   // size of each key in bytes
	EntrySize  uint16 // size of each entry in bytes, big-endian
	EntryCount uint32 // number of sorted (key,value) entries that follow

	Entries []struct {
		Key   []byte // key bytes (KeySize)
		Entry entry
	}

	ExtraData []byte // extra data
}

Format describes a format of a single pack index. The actual structure is not used, it's purely for documentation purposes. The struct is byte-aligned.

type FormattingOptions 

type FormattingOptions struct {
	Version     int    `json:"version,omitempty"`     // version number, must be "1"
	Hash        string `json:"hash,omitempty"`        // identifier of the hash algorithm used
	Encryption  string `json:"encryption,omitempty"`  // identifier of the encryption algorithm used
	HMACSecret  []byte `json:"secret,omitempty"`      // HMAC secret used to generate encryption keys
	MasterKey   []byte `json:"masterKey,omitempty"`   // master encryption key (SIV-mode encryption only)
	MaxPackSize int    `json:"maxPackSize,omitempty"` // maximum size of a pack object
}

FormattingOptions describes the rules for formatting contents in repository.

func (*FormattingOptions) GetEncryptionAlgorithm added in v0.5.2 

func (f *FormattingOptions) GetEncryptionAlgorithm() string

GetEncryptionAlgorithm implements encryption.Parameters.

func (*FormattingOptions) GetHMACSecret added in v0.5.2 

func (f *FormattingOptions) GetHMACSecret() []byte

GetHMACSecret implements hashing.Parameters.

func (*FormattingOptions) GetHashFunction added in v0.5.2 

func (f *FormattingOptions) GetHashFunction() string

GetHashFunction implements hashing.Parameters.

func (*FormattingOptions) GetMasterKey added in v0.5.2 

func (f *FormattingOptions) GetMasterKey() []byte

GetMasterKey implements encryption.Parameters.

type ID 

type ID string

ID is an identifier of content in content-addressable storage.

func (ID) HasPrefix 

func (i ID) HasPrefix() bool

HasPrefix determines if the given ID has a non-empty prefix.

func (ID) Prefix 

func (i ID) Prefix() ID

Prefix returns a one-character prefix of a content ID or an empty string.

type IDRange added in v0.6.0 

type IDRange struct {
	StartID ID // inclusive
	EndID   ID // exclusive
}

IDRange represents a range of IDs.

func PrefixRange added in v0.6.0 

func PrefixRange(prefix ID) IDRange

PrefixRange returns ID range that contains all IDs with a given prefix.

func (IDRange) Contains added in v0.6.0 

func (r IDRange) Contains(id ID) bool

Contains determines whether given ID is in the range.

type IndexBlobInfo 

type IndexBlobInfo struct {
	blob.Metadata
	Superseded []blob.Metadata
}

IndexBlobInfo is an information about a single index blob managed by Manager.

type Info 

type Info struct {
	ID               ID      `json:"contentID"`
	Length           uint32  `json:"length"`
	TimestampSeconds int64   `json:"time"`
	PackBlobID       blob.ID `json:"packFile,omitempty"`
	PackOffset       uint32  `json:"packOffset,omitempty"`
	Deleted          bool    `json:"deleted"`
	FormatVersion    byte    `json:"formatVersion"`
}

Info is an information about a single piece of content managed by Manager.

func (*Info) Timestamp 

func (i *Info) Timestamp() time.Time

Timestamp returns the time when a content was created or deleted.

type IterateCallback 

type IterateCallback func(Info) error

IterateCallback is the function type used as a callback during content iteration.

type IterateOptions 

type IterateOptions struct {
	Range          IDRange
	IncludeDeleted bool
	Parallel       int
}

IterateOptions contains the options used for iterating over content.

type IteratePackOptions 

type IteratePackOptions struct {
	IncludePacksWithOnlyDeletedContent bool
	IncludeContentInfos                bool
	Prefixes                           []blob.ID
}

IteratePackOptions are the options used to iterate over packs.

type IteratePacksCallback 

type IteratePacksCallback func(PackInfo) error

IteratePacksCallback is the function type used as callback during pack iteration.

type Manager 

type Manager struct {
	// contains filtered or unexported fields
}

Manager builds content-addressable storage with encryption, deduplication and packaging on top of BLOB store.

func NewManager 

func NewManager(ctx context.Context, st blob.Storage, f *FormattingOptions, caching *CachingOptions, options ManagerOptions) (*Manager, error)

NewManager creates new content manager with given packing options and a formatter.

func (*Manager) Close 

func (bm *Manager) Close(ctx context.Context) error

Close closes the content manager.

func (*Manager) CompactIndexes 

func (bm *Manager) CompactIndexes(ctx context.Context, opt CompactOptions) error

CompactIndexes performs compaction of index blobs ensuring that # of small index blobs is below opt.maxSmallBlobs.

func (*Manager) ContentInfo 

func (bm *Manager) ContentInfo(ctx context.Context, contentID ID) (Info, error)

ContentInfo returns information about a single content.

func (*Manager) DecryptBlob added in v0.6.0 

func (bm *Manager) DecryptBlob(ctx context.Context, blobID blob.ID) ([]byte, error)

DecryptBlob returns the contents of an encrypted blob that can be decrypted (n,m,l).

func (*Manager) DeleteContent 

func (bm *Manager) DeleteContent(ctx context.Context, contentID ID) error

DeleteContent marks the given contentID as deleted.

NOTE: To avoid race conditions only contents that cannot be possibly re-created should ever be deleted. That means that contents of such contents should include some element of randomness or a contemporaneous timestamp that will never reappear.

func (*Manager) DisableIndexFlush 

func (bm *Manager) DisableIndexFlush(ctx context.Context)

DisableIndexFlush increments the counter preventing automatic index flushes.

func (*Manager) EnableIndexFlush 

func (bm *Manager) EnableIndexFlush(ctx context.Context)

EnableIndexFlush decrements the counter preventing automatic index flushes. The flushes will be reenabled when the index drops to zero.

func (*Manager) Flush 

func (bm *Manager) Flush(ctx context.Context) error

Flush completes writing any pending packs and writes pack indexes to the underlying storage. Any pending writes completed before Flush() has started are guaranteed to be committed to the repository before Flush() returns.

func (*Manager) GetContent 

func (bm *Manager) GetContent(ctx context.Context, contentID ID) (v []byte, err error)

GetContent gets the contents of a given content. If the content is not found returns ErrContentNotFound.

func (*Manager) IndexBlobs 

func (bm *Manager) IndexBlobs(ctx context.Context, includeInactive bool) ([]IndexBlobInfo, error)

IndexBlobs returns the list of active index blobs.

func (*Manager) IterateContents 

func (bm *Manager) IterateContents(ctx context.Context, opts IterateOptions, callback IterateCallback) error

IterateContents invokes the provided callback for each content starting with a specified prefix and possibly including deleted items.

func (*Manager) IteratePacks 

func (bm *Manager) IteratePacks(ctx context.Context, options IteratePackOptions, callback IteratePacksCallback) error

IteratePacks invokes the provided callback for all pack blobs.

func (*Manager) IterateUnreferencedBlobs 

func (bm *Manager) IterateUnreferencedBlobs(ctx context.Context, blobPrefixes []blob.ID, parallellism int, callback func(blob.Metadata) error) error

IterateUnreferencedBlobs returns the list of unreferenced storage blobs.

func (*Manager) ParseIndexBlob added in v0.7.0 

func (bm *Manager) ParseIndexBlob(ctx context.Context, blobID blob.ID) ([]Info, error)

ParseIndexBlob loads entries in a given index blob and returns them.

func (*Manager) RecoverIndexFromPackBlob 

func (bm *Manager) RecoverIndexFromPackBlob(ctx context.Context, packFile blob.ID, packFileLength int64, commit bool) ([]Info, error)

RecoverIndexFromPackBlob attempts to recover index blob entries from a given pack file. Pack file length may be provided (if known) to reduce the number of bytes that are read from the storage.

func (*Manager) Refresh 

func (bm *Manager) Refresh(ctx context.Context) (bool, error)

Refresh reloads the committed content indexes.

func (*Manager) RewriteContent 

func (bm *Manager) RewriteContent(ctx context.Context, contentID ID) error

RewriteContent causes reads and re-writes a given content using the most recent format.

func (*Manager) SyncMetadataCache added in v0.6.0 

func (bm *Manager) SyncMetadataCache(ctx context.Context) error

SyncMetadataCache synchronizes metadata cache with metadata blobs in storage.

func (*Manager) UndeleteContent added in v0.6.0 

func (bm *Manager) UndeleteContent(ctx context.Context, contentID ID) error

UndeleteContent rewrites the content with the given ID if the content exists and is mark deleted. If the content exists and is not marked deleted, this operation is a no-op.

func (*Manager) WriteContent 

func (bm *Manager) WriteContent(ctx context.Context, data []byte, prefix ID) (ID, error)

WriteContent saves a given content of data to a pack group with a provided name and returns a contentID that's based on the contents of data written.

type ManagerOptions added in v0.6.0 

type ManagerOptions struct {
	RepositoryFormatBytes []byte
	TimeNow               func() time.Time // Time provider
}

ManagerOptions are the optional parameters for manager creation.

type PackInfo 

type PackInfo struct {
	PackID       blob.ID
	ContentCount int
	TotalSize    int64
	ContentInfos []Info
}

PackInfo contains the data for a pack.

type Stats 

type Stats struct {
	// contains filtered or unexported fields
}

Stats exposes statistics about content operation.

func (*Stats) DecryptedBytes 

func (s *Stats) DecryptedBytes() int64

DecryptedBytes returns the approximate total number of decrypted bytes.

func (*Stats) EncryptedBytes 

func (s *Stats) EncryptedBytes() int64

EncryptedBytes returns the approximate total number of decrypted bytes.

func (*Stats) HashedContent added in v0.6.0 

func (s *Stats) HashedContent() (count uint32, bytes int64)

HashedContent returns the approximate hashed content count and their total size in bytes.

func (*Stats) InvalidContents 

func (s *Stats) InvalidContents() uint32

InvalidContents returns the approximate count of invalid contents found.

func (*Stats) ReadContent added in v0.6.0 

func (s *Stats) ReadContent() (count uint32, bytes int64)

ReadContent returns the approximate read content count and their total size in bytes.

func (*Stats) Reset 

func (s *Stats) Reset()

Reset clears all content statistics.

func (*Stats) ValidContents 

func (s *Stats) ValidContents() uint32

ValidContents returns the approximate count of valid contents found.

func (*Stats) WrittenContent added in v0.6.0 

func (s *Stats) WrittenContent() (count uint32, bytes int64)

WrittenContent returns the approximate written content count and their total size in bytes.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.