content

package
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 10, 2020 License: Apache-2.0 Imports: 38 Imported by: 7

Documentation

Overview

Package content implements repository support for content-addressable storage.

Index

Constants

View Source 
const (
	PackBlobIDPrefixRegular blob.ID = "p"
	PackBlobIDPrefixSpecial blob.ID = "q"
)

Prefixes for pack blobs

View Source 
const DefaultEncryption = "SALSA20-HMAC"

DefaultEncryption is the name of the default encryption algorithm.

View Source 
const DefaultHash = "BLAKE2B-256-128"

DefaultHash is the name of the default hash algorithm.

Variables

View Source 
var ErrContentNotFound = errors.New("content not found")

ErrContentNotFound is returned when content is not found.

View Source 
var PackBlobIDPrefixes = []blob.ID{
	PackBlobIDPrefixRegular,
	PackBlobIDPrefixSpecial,
}

PackBlobIDPrefixes contains all possible prefixes for pack blobs.

Functions

func CreateHashAndEncryptor 

func CreateHashAndEncryptor(f *FormattingOptions) (HashFunc, Encryptor, error)

CreateHashAndEncryptor returns new hashing and encrypting functions based on the specified formatting options

func RegisterEncryption 

func RegisterEncryption(name string, newEncryptor EncryptorFactory)

RegisterEncryption registers new encryption algorithm.

func RegisterHash 

func RegisterHash(name string, newHashFunc HashFuncFactory)

RegisterHash registers a hash function with a given name.

func SupportedEncryptionAlgorithms 

func SupportedEncryptionAlgorithms() []string

SupportedEncryptionAlgorithms returns the names of the supported encryption methods

func SupportedHashAlgorithms 

func SupportedHashAlgorithms() []string

SupportedHashAlgorithms returns the names of the supported hashing schemes

func UsingContentCache 

func UsingContentCache(ctx context.Context, enabled bool) context.Context

UsingContentCache returns a derived context that causes content manager to use cache.

func UsingListCache 

func UsingListCache(ctx context.Context, enabled bool) context.Context

UsingListCache returns a derived context that causes content manager to use cache.

Types

type CachingOptions 

type CachingOptions struct {
	CacheDirectory            string `json:"cacheDirectory,omitempty"`
	MaxCacheSizeBytes         int64  `json:"maxCacheSize,omitempty"`
	MaxMetadataCacheSizeBytes int64  `json:"maxMetadataCacheSize,omitempty"`
	MaxListCacheDurationSec   int    `json:"maxListCacheDuration,omitempty"`
	IgnoreListCache           bool   `json:"-"`
	HMACSecret                []byte `json:"-"`
}

CachingOptions specifies configuration of local cache.

type CompactOptions 

type CompactOptions struct {
	MinSmallBlobs        int
	MaxSmallBlobs        int
	AllIndexes           bool
	SkipDeletedOlderThan time.Duration
}

CompactOptions provides options for compaction

type Encryptor 

type Encryptor interface {
	// Encrypt returns encrypted bytes corresponding to the given plaintext.
	// Must not clobber the input slice and return ciphertext with additional padding and checksum.
	Encrypt(plainText, contentID []byte) ([]byte, error)

	// Decrypt returns unencrypted bytes corresponding to the given ciphertext.
	// Must not clobber the input slice. If IsAuthenticated() == true, Decrypt will perform
	// authenticity check before decrypting.
	Decrypt(cipherText, contentID []byte) ([]byte, error)

	// IsAuthenticated returns true if encryption is authenticated.
	// In this case Decrypt() is expected to perform authenticity check.
	IsAuthenticated() bool
}

Encryptor performs encryption and decryption of contents of data.

type EncryptorFactory 

type EncryptorFactory func(o *FormattingOptions) (Encryptor, error)

EncryptorFactory creates new Encryptor for given FormattingOptions

type Format 

type Format struct {
	Version    byte   // format version number must be 0x01
	KeySize    byte   // size of each key in bytes
	EntrySize  uint16 // size of each entry in bytes, big-endian
	EntryCount uint32 // number of sorted (key,value) entries that follow

	Entries []struct {
		Key   []byte // key bytes (KeySize)
		Entry entry
	}

	ExtraData []byte // extra data
}

Format describes a format of a single pack index. The actual structure is not used, it's purely for documentation purposes. The struct is byte-aligned.

type FormattingOptions 

type FormattingOptions struct {
	Version     int    `json:"version,omitempty"`     // version number, must be "1"
	Hash        string `json:"hash,omitempty"`        // identifier of the hash algorithm used
	Encryption  string `json:"encryption,omitempty"`  // identifier of the encryption algorithm used
	HMACSecret  []byte `json:"secret,omitempty"`      // HMAC secret used to generate encryption keys
	MasterKey   []byte `json:"masterKey,omitempty"`   // master encryption key (SIV-mode encryption only)
	MaxPackSize int    `json:"maxPackSize,omitempty"` // maximum size of a pack object
}

FormattingOptions describes the rules for formatting contents in repository.

func (*FormattingOptions) DeriveKey 

func (o *FormattingOptions) DeriveKey(purpose []byte, length int) []byte

DeriveKey uses HKDF to derive a key of a given length and a given purpose.

type HashFunc 

type HashFunc func(data []byte) []byte

HashFunc computes hash of content of data using a cryptographic hash function, possibly with HMAC and/or truncation.

type HashFuncFactory 

type HashFuncFactory func(o *FormattingOptions) (HashFunc, error)

HashFuncFactory returns a hash function for given formatting options.

type ID 

type ID string

ID is an identifier of content in content-addressable storage.

func (ID) HasPrefix 

func (i ID) HasPrefix() bool

HasPrefix determines if the given ID has a non-empty prefix.

func (ID) Prefix 

func (i ID) Prefix() ID

Prefix returns a one-character prefix of a content ID or an empty string.

type IndexBlobInfo 

type IndexBlobInfo struct {
	BlobID    blob.ID
	Length    int64
	Timestamp time.Time
}

IndexBlobInfo is an information about a single index blob managed by Manager.

type Info 

type Info struct {
	ID               ID      `json:"contentID"`
	Length           uint32  `json:"length"`
	TimestampSeconds int64   `json:"time"`
	PackBlobID       blob.ID `json:"packFile,omitempty"`
	PackOffset       uint32  `json:"packOffset,omitempty"`
	Deleted          bool    `json:"deleted"`
	Payload          []byte  `json:"payload"` // set for payloads stored inline
	FormatVersion    byte    `json:"formatVersion"`
}

Info is an information about a single piece of content managed by Manager.

func (*Info) Timestamp 

func (i *Info) Timestamp() time.Time

Timestamp returns the time when a content was created or deleted.

type IterateCallback 

type IterateCallback func(Info) error

IterateCallback is the function type used as a callback during content iteration

type IterateOptions 

type IterateOptions struct {
	Prefix         ID
	IncludeDeleted bool
	Parallel       int
}

IterateOptions contains the options used for iterating over content

type IteratePackOptions 

type IteratePackOptions struct {
	IncludePacksWithOnlyDeletedContent bool
	IncludeContentInfos                bool
}

IteratePackOptions are the options used to iterate over packs

type IteratePacksCallback 

type IteratePacksCallback func(PackInfo) error

IteratePacksCallback is the function type used as callback during pack iteration

type Manager 

type Manager struct {
	// contains filtered or unexported fields
}

Manager builds content-addressable storage with encryption, deduplication and packaging on top of BLOB store.

func NewManager 

func NewManager(ctx context.Context, st blob.Storage, f *FormattingOptions, caching CachingOptions, repositoryFormatBytes []byte) (*Manager, error)

NewManager creates new content manager with given packing options and a formatter.

func (*Manager) Close 

func (bm *Manager) Close(ctx context.Context) error

Close closes the content manager.

func (*Manager) CompactIndexes 

func (bm *Manager) CompactIndexes(ctx context.Context, opt CompactOptions) error

CompactIndexes performs compaction of index blobs ensuring that # of small contents is between minSmallContentCount and maxSmallContentCount

func (*Manager) ContentInfo 

func (bm *Manager) ContentInfo(ctx context.Context, contentID ID) (Info, error)

ContentInfo returns information about a single content.

func (*Manager) DeleteContent 

func (bm *Manager) DeleteContent(contentID ID) error

DeleteContent marks the given contentID as deleted.

NOTE: To avoid race conditions only contents that cannot be possibly re-created should ever be deleted. That means that contents of such contents should include some element of randomness or a contemporaneous timestamp that will never reappear.

func (*Manager) DisableIndexFlush 

func (bm *Manager) DisableIndexFlush()

DisableIndexFlush increments the counter preventing automatic index flushes.

func (*Manager) EnableIndexFlush 

func (bm *Manager) EnableIndexFlush()

EnableIndexFlush decrements the counter preventing automatic index flushes. The flushes will be reenabled when the index drops to zero.

func (*Manager) Flush 

func (bm *Manager) Flush(ctx context.Context) error

Flush completes writing any pending packs and writes pack indexes to the underlying storage. Any pending writes completed before Flush() has started are guaranteed to be committed to the repository before Flush() returns.

func (*Manager) GetContent 

func (bm *Manager) GetContent(ctx context.Context, contentID ID) ([]byte, error)

GetContent gets the contents of a given content. If the content is not found returns ErrContentNotFound.

func (*Manager) IndexBlobs 

func (bm *Manager) IndexBlobs(ctx context.Context) ([]IndexBlobInfo, error)

IndexBlobs returns the list of active index blobs.

func (*Manager) IterateContentInShortPacks 

func (bm *Manager) IterateContentInShortPacks(threshold int64, callback IterateCallback) error

IterateContentInShortPacks invokes the provided callback for all contents that are stored in packs shorter than the given threshold.

func (*Manager) IterateContents 

func (bm *Manager) IterateContents(opts IterateOptions, callback IterateCallback) error

IterateContents invokes the provided callback for each content starting with a specified prefix and possibly including deleted items.

func (*Manager) IteratePacks 

func (bm *Manager) IteratePacks(options IteratePackOptions, callback IteratePacksCallback) error

IteratePacks invokes the provided callback for all pack blobs.

func (*Manager) IterateUnreferencedBlobs 

func (bm *Manager) IterateUnreferencedBlobs(ctx context.Context, parallellism int, callback func(blob.Metadata) error) error

IterateUnreferencedBlobs returns the list of unreferenced storage blobs.

func (*Manager) RecoverIndexFromPackBlob 

func (bm *Manager) RecoverIndexFromPackBlob(ctx context.Context, packFile blob.ID, packFileLength int64, commit bool) ([]Info, error)

RecoverIndexFromPackBlob attempts to recover index blob entries from a given pack file. Pack file length may be provided (if known) to reduce the number of bytes that are read from the storage.

func (*Manager) Refresh 

func (bm *Manager) Refresh(ctx context.Context) (bool, error)

Refresh reloads the committed content indexes.

func (*Manager) ResetStats 

func (bm *Manager) ResetStats()

ResetStats resets statistics to zero values.

func (*Manager) RewriteContent 

func (bm *Manager) RewriteContent(ctx context.Context, contentID ID) error

RewriteContent causes reads and re-writes a given content using the most recent format.

func (*Manager) Stats 

func (bm *Manager) Stats() Stats

Stats returns statistics about content manager operations.

func (*Manager) WriteContent 

func (bm *Manager) WriteContent(ctx context.Context, data []byte, prefix ID) (ID, error)

WriteContent saves a given content of data to a pack group with a provided name and returns a contentID that's based on the contents of data written.

type PackInfo 

type PackInfo struct {
	PackID       blob.ID
	ContentCount int
	TotalSize    int64
	ContentInfos []Info
}

PackInfo contains the data for a pack

type Stats 

type Stats struct {
	// Keep int64 fields first to ensure they get aligned to at least 64-bit boundaries
	// which is required for atomic access on ARM and x86-32.
	ReadBytes      int64 `json:"readBytes,omitempty"`
	WrittenBytes   int64 `json:"writtenBytes,omitempty"`
	DecryptedBytes int64 `json:"decryptedBytes,omitempty"`
	EncryptedBytes int64 `json:"encryptedBytes,omitempty"`
	HashedBytes    int64 `json:"hashedBytes,omitempty"`

	ReadContents    int32 `json:"readContents,omitempty"`
	WrittenContents int32 `json:"writtenContents,omitempty"`
	CheckedContents int32 `json:"checkedContents,omitempty"`
	HashedContents  int32 `json:"hashedContents,omitempty"`
	InvalidContents int32 `json:"invalidContents,omitempty"`
	PresentContents int32 `json:"presentContents,omitempty"`
	ValidContents   int32 `json:"validContents,omitempty"`
}

Stats exposes statistics about content operation.

func (*Stats) Reset 

func (s *Stats) Reset()

Reset clears all repository statistics.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.