snapshotfs

package
v0.13.0-rc2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 17, 2023 License: Apache-2.0 Imports: 43 Imported by: 7

Documentation

Overview

Package snapshotfs implements virtual filesystem on top of snapshots in repo.Repository.

Index

Constants

View Source 
const (
	IncompleteReasonCheckpoint   = "checkpoint"
	IncompleteReasonCanceled     = "canceled"
	IncompleteReasonLimitReached = "limit reached"
)

reasons why a snapshot is incomplete.

View Source 
const DefaultCheckpointInterval = 45 * time.Minute

DefaultCheckpointInterval is the default frequency of mid-upload checkpointing.

Variables

This section is empty.

Functions

func AllSourcesEntry 

func AllSourcesEntry(rep repo.Repository) fs.Directory

AllSourcesEntry returns fs.Directory that contains the list of all snapshot sources found in the repository.

func AutoDetectEntryFromObjectID added in v0.7.2 

func AutoDetectEntryFromObjectID(ctx context.Context, rep repo.Repository, oid object.ID, maybeName string) fs.Entry

AutoDetectEntryFromObjectID returns fs.Entry (either file or directory) for the provided object ID. It uses heuristics to determine whether object ID is possibly a directory and treats it as such.

func CalculateStorageStats added in v0.10.6 

func CalculateStorageStats(ctx context.Context, rep repo.Repository, manifests []*snapshot.Manifest, callback func(m *snapshot.Manifest) error) error

CalculateStorageStats calculates the storage statistics for a given list of snapshots, by determining the count and size of unique contents and objects for each snapshot in the slice as well as running total so far for all the previous snapshots.

For each snapshot the provided callback is invoked as soon as the results are available and iteration continues until the callback returns a non-nil error.

func DirectoryEntry 

func DirectoryEntry(rep repo.Repository, objectID object.ID, dirSummary *fs.DirectorySummary) fs.Directory

DirectoryEntry returns fs.Directory based on repository object with the specified ID. The existence or validity of the directory object is not validated until its contents are read.

func EntryFromDirEntry added in v0.4.0 

func EntryFromDirEntry(r repo.Repository, md *snapshot.DirEntry) fs.Entry

EntryFromDirEntry returns a filesystem entry based on the directory entry.

func Estimate added in v0.8.0 

func Estimate(ctx context.Context, entry fs.Directory, policyTree *policy.Tree, progress EstimateProgress, maxExamplesPerBucket int) error

Estimate walks the provided directory tree and invokes provided progress callback as it discovers items to be snapshotted.

func FilesystemDirectoryFromIDWithPath added in v0.7.2 

func FilesystemDirectoryFromIDWithPath(ctx context.Context, rep repo.Repository, rootID string, consistentAttributes bool) (fs.Directory, error)

FilesystemDirectoryFromIDWithPath returns a filesystem directory entry for the provided object ID, which can be a snapshot manifest ID or an object ID with path.

func FilesystemEntryFromIDWithPath added in v0.7.2 

func FilesystemEntryFromIDWithPath(ctx context.Context, rep repo.Repository, rootID string, consistentAttributes bool) (fs.Entry, error)

FilesystemEntryFromIDWithPath returns a filesystem entry for the provided object ID, which can be a snapshot manifest ID or an object ID with path. If multiple snapshots match and they don't agree on root object attributes and consistentAttributes==true the function fails, otherwise it returns the latest of the snapshots.

func GetEntryFromPlaceholder added in v0.9.0 

func GetEntryFromPlaceholder(ctx context.Context, r repo.Repository, defp snapshot.HasDirEntryOrNil) (fs.Entry, error)

GetEntryFromPlaceholder returns a fs.Entry for shallow placeholder defp referencing a real Entry in Repository r.

func GetNestedEntry added in v0.7.2 

func GetNestedEntry(ctx context.Context, startingDir fs.Entry, pathElements []string) (fs.Entry, error)

GetNestedEntry returns nested entry with a given name path.

func IsDirectoryID added in v0.7.2 

func IsDirectoryID(oid object.ID) bool

IsDirectoryID determines whether given object ID represents a directory.

func ParseObjectIDWithPath added in v0.7.2 

func ParseObjectIDWithPath(ctx context.Context, rep repo.Repository, objectIDWithPath string) (object.ID, error)

ParseObjectIDWithPath interprets the given ID string (which could be an object ID optionally followed by nested path specification) and returns corresponding object.ID.

func RewriteFail added in v0.11.0 

func RewriteFail(ctx context.Context, parentPath string, entry *snapshot.DirEntry, err error) (*snapshot.DirEntry, error)

RewriteFail is a callback that fails the entire rewrite process when a directory is unreadable.

func RewriteKeep added in v0.11.0 

func RewriteKeep(ctx context.Context, parentPath string, input *snapshot.DirEntry, err error) (*snapshot.DirEntry, error)

RewriteKeep is a callback that keeps the unreadable entry.

func RewriteRemove added in v0.11.0 

func RewriteRemove(ctx context.Context, parentPath string, entry *snapshot.DirEntry, err error) (*snapshot.DirEntry, error)

RewriteRemove is a callback that removes the entire failed entry.

func SnapshotRoot 

func SnapshotRoot(rep repo.Repository, man *snapshot.Manifest) (fs.Entry, error)

SnapshotRoot returns fs.Entry representing the root of a snapshot.

Types

type CountingUploadProgress added in v0.5.2 

type CountingUploadProgress struct {
	NullUploadProgress
	// contains filtered or unexported fields
}

CountingUploadProgress is an implementation of UploadProgress that accumulates counters.

func (*CountingUploadProgress) CachedFile added in v0.5.2 

func (p *CountingUploadProgress) CachedFile(fname string, numBytes int64)

CachedFile implements UploadProgress.

func (*CountingUploadProgress) Error added in v0.8.0 

func (p *CountingUploadProgress) Error(path string, err error, isIgnored bool)

Error implements UploadProgress.

func (*CountingUploadProgress) EstimatedDataSize added in v0.7.0 

func (p *CountingUploadProgress) EstimatedDataSize(numFiles int, numBytes int64)

EstimatedDataSize implements UploadProgress.

func (*CountingUploadProgress) ExcludedDir added in v0.8.0 

func (p *CountingUploadProgress) ExcludedDir(dirname string)

ExcludedDir implements UploadProgress.

func (*CountingUploadProgress) ExcludedFile added in v0.8.0 

func (p *CountingUploadProgress) ExcludedFile(fname string, numBytes int64)

ExcludedFile implements UploadProgress.

func (*CountingUploadProgress) FinishedFile added in v0.12.0 

func (p *CountingUploadProgress) FinishedFile(fname string, err error)

FinishedFile implements UploadProgress.

func (*CountingUploadProgress) FinishedHashingFile added in v0.5.2 

func (p *CountingUploadProgress) FinishedHashingFile(fname string, numBytes int64)

FinishedHashingFile implements UploadProgress.

func (*CountingUploadProgress) HashedBytes added in v0.5.2 

func (p *CountingUploadProgress) HashedBytes(numBytes int64)

HashedBytes implements UploadProgress.

func (*CountingUploadProgress) Snapshot added in v0.5.2 

func (p *CountingUploadProgress) Snapshot() UploadCounters

Snapshot captures current snapshot of the upload.

func (*CountingUploadProgress) StartedDirectory added in v0.5.2 

func (p *CountingUploadProgress) StartedDirectory(dirname string)

StartedDirectory implements UploadProgress.

func (*CountingUploadProgress) UITaskCounters added in v0.8.0 

func (p *CountingUploadProgress) UITaskCounters(final bool) map[string]uitask.CounterValue

UITaskCounters returns UI task counters.

func (*CountingUploadProgress) UploadStarted added in v0.5.2 

func (p *CountingUploadProgress) UploadStarted()

UploadStarted implements UploadProgress.

func (*CountingUploadProgress) UploadedBytes added in v0.8.0 

func (p *CountingUploadProgress) UploadedBytes(numBytes int64)

UploadedBytes implements UploadProgress.

type DirManifestBuilder added in v0.11.0 

type DirManifestBuilder struct {
	// contains filtered or unexported fields
}

DirManifestBuilder builds directory manifests.

func (*DirManifestBuilder) AddEntry added in v0.11.0 

func (b *DirManifestBuilder) AddEntry(de *snapshot.DirEntry)

AddEntry adds a directory entry to the builder.

func (*DirManifestBuilder) AddFailedEntry added in v0.11.0 

func (b *DirManifestBuilder) AddFailedEntry(relPath string, isIgnoredError bool, err error)

AddFailedEntry adds a failed directory entry to the builder and increments enither ignored or fatal error count.

func (*DirManifestBuilder) Build added in v0.11.0 

func (b *DirManifestBuilder) Build(dirModTime fs.UTCTimestamp, incompleteReason string) *snapshot.DirManifest

Build builds the directory manifest.

func (*DirManifestBuilder) Clone added in v0.11.0 

func (b *DirManifestBuilder) Clone() *DirManifestBuilder

Clone clones the current state of dirManifestBuilder.

type DirRewriter added in v0.11.0 

type DirRewriter struct {
	// contains filtered or unexported fields
}

DirRewriter rewrites contents of directories by walking the snapshot tree recursively.

func NewDirRewriter added in v0.11.0 

func NewDirRewriter(ctx context.Context, rep repo.RepositoryWriter, opts DirRewriterOptions) (*DirRewriter, error)

NewDirRewriter creates a new directory rewriter.

func (*DirRewriter) Close added in v0.11.0 

func (rw *DirRewriter) Close(ctx context.Context)

Close closes the rewriter.

func (*DirRewriter) RewriteSnapshotManifest added in v0.11.0 

func (rw *DirRewriter) RewriteSnapshotManifest(ctx context.Context, man *snapshot.Manifest) (bool, error)

RewriteSnapshotManifest rewrites the directory tree starting at a given manifest.

type DirRewriterOptions added in v0.11.0 

type DirRewriterOptions struct {
	Parallel int

	RewriteEntry RewriteDirEntryCallback

	// when != nil will be invoked to replace directory that can't be read,
	// by default RewriteAsStub()
	OnDirectoryReadFailure RewriteFailedEntryCallback
}

DirRewriterOptions provides options for directory rewriter.

type EntryCallback added in v0.10.6 

type EntryCallback func(ctx context.Context, entry fs.Entry, oid object.ID, entryPath string) error

EntryCallback is invoked when walking the tree of snapshots.

type EstimateProgress added in v0.8.0 

type EstimateProgress interface {
	Processing(ctx context.Context, dirname string)
	Error(ctx context.Context, filename string, err error, isIgnored bool)
	Stats(ctx context.Context, s *snapshot.Stats, includedFiles, excludedFiles SampleBuckets, excludedDirs []string, final bool)
}

EstimateProgress must be provided by the caller of Estimate to report results.

type NullUploadProgress added in v0.5.2 

type NullUploadProgress struct{}

NullUploadProgress is an implementation of UploadProgress that does not produce any output.

func (*NullUploadProgress) CachedFile added in v0.5.2 

func (p *NullUploadProgress) CachedFile(fname string, numBytes int64)

CachedFile implements UploadProgress.

func (*NullUploadProgress) Error added in v0.8.0 

func (p *NullUploadProgress) Error(path string, err error, isIgnored bool)

Error implements UploadProgress.

func (*NullUploadProgress) EstimatedDataSize added in v0.7.0 

func (p *NullUploadProgress) EstimatedDataSize(fileCount int, totalBytes int64)

EstimatedDataSize implements UploadProgress.

func (*NullUploadProgress) ExcludedDir added in v0.8.0 

func (p *NullUploadProgress) ExcludedDir(dirname string)

ExcludedDir implements UploadProgress.

func (*NullUploadProgress) ExcludedFile added in v0.8.0 

func (p *NullUploadProgress) ExcludedFile(fname string, numBytes int64)

ExcludedFile implements UploadProgress.

func (*NullUploadProgress) FinishedDirectory added in v0.5.2 

func (p *NullUploadProgress) FinishedDirectory(dirname string)

FinishedDirectory implements UploadProgress.

func (*NullUploadProgress) FinishedFile added in v0.12.0 

func (p *NullUploadProgress) FinishedFile(fname string, err error)

FinishedFile implements UploadProgress.

func (*NullUploadProgress) FinishedHashingFile added in v0.5.2 

func (p *NullUploadProgress) FinishedHashingFile(fname string, numBytes int64)

FinishedHashingFile implements UploadProgress.

func (*NullUploadProgress) HashedBytes added in v0.5.2 

func (p *NullUploadProgress) HashedBytes(numBytes int64)

HashedBytes implements UploadProgress.

func (*NullUploadProgress) HashingFile added in v0.5.2 

func (p *NullUploadProgress) HashingFile(fname string)

HashingFile implements UploadProgress.

func (*NullUploadProgress) StartedDirectory added in v0.5.2 

func (p *NullUploadProgress) StartedDirectory(dirname string)

StartedDirectory implements UploadProgress.

func (*NullUploadProgress) UploadFinished added in v0.5.2 

func (p *NullUploadProgress) UploadFinished()

UploadFinished implements UploadProgress.

func (*NullUploadProgress) UploadStarted added in v0.5.2 

func (p *NullUploadProgress) UploadStarted()

UploadStarted implements UploadProgress.

func (*NullUploadProgress) UploadedBytes added in v0.5.2 

func (p *NullUploadProgress) UploadedBytes(numBytes int64)

UploadedBytes implements UploadProgress.

type RewriteDirEntryCallback added in v0.11.0 

type RewriteDirEntryCallback func(ctx context.Context, parentPath string, input *snapshot.DirEntry) (*snapshot.DirEntry, error)

RewriteDirEntryCallback returns a replacement for the provided directory entry in the provided path. nil indicates that the entry should be removed.

type RewriteFailedEntryCallback added in v0.11.0 

type RewriteFailedEntryCallback func(ctx context.Context, parentPath string, input *snapshot.DirEntry, err error) (*snapshot.DirEntry, error)

RewriteFailedEntryCallback is invoked rewriting a file or directory.

func RewriteAsStub added in v0.11.0 

func RewriteAsStub(rep repo.RepositoryWriter) RewriteFailedEntryCallback

RewriteAsStub returns a callback that replaces the invalid entry with a stub that describes the error.

type SampleBucket added in v0.8.0 

type SampleBucket struct {
	MinSize   int64    `json:"minSize"`
	Count     int      `json:"count"`
	TotalSize int64    `json:"totalSize"`
	Examples  []string `json:"examples,omitempty"`
}

SampleBucket keeps track of count and total size of files above in certain size range and includes small number of examples of such files.

type SampleBuckets added in v0.8.0 

type SampleBuckets []*SampleBucket

SampleBuckets is a collection of buckets for interesting file sizes sorted in descending order.

type TreeWalker 

type TreeWalker struct {
	// contains filtered or unexported fields
}

TreeWalker processes snapshot filesystem trees by invoking the provided callback once for each object found in the tree.

func NewTreeWalker 

func NewTreeWalker(ctx context.Context, options TreeWalkerOptions) (*TreeWalker, error)

NewTreeWalker creates new tree walker.

func (*TreeWalker) Close added in v0.10.6 

func (w *TreeWalker) Close(ctx context.Context)

Close closes the tree walker.

func (*TreeWalker) Err added in v0.10.6 

func (w *TreeWalker) Err() error

Err returns the error encountered when walking the tree.

func (*TreeWalker) Process added in v0.10.6 

func (w *TreeWalker) Process(ctx context.Context, e fs.Entry, entryPath string) error

Process processes the snapshot tree entry.

func (*TreeWalker) ReportError added in v0.10.6 

func (w *TreeWalker) ReportError(ctx context.Context, entryPath string, err error)

ReportError reports the error.

func (*TreeWalker) TooManyErrors added in v0.10.6 

func (w *TreeWalker) TooManyErrors() bool

TooManyErrors reports true if there are too many errors already reported.

type TreeWalkerOptions added in v0.10.6 

type TreeWalkerOptions struct {
	EntryCallback EntryCallback

	Parallelism int
	MaxErrors   int
}

TreeWalkerOptions provides optional fields for TreeWalker.

type UnreadableDirEntryReplacement added in v0.11.0 

type UnreadableDirEntryReplacement struct {
	Info  string             `json:"info"`
	Error string             `json:"error"`
	Entry *snapshot.DirEntry `json:"entry"`
}

UnreadableDirEntryReplacement is serialized as a stub object replacing unreadable file or directory.

type UploadCounters added in v0.5.2 

type UploadCounters struct {
	// +checkatomic
	TotalCachedBytes int64 `json:"cachedBytes"`
	// +checkatomic
	TotalHashedBytes int64 `json:"hashedBytes"`
	// +checkatomic
	TotalUploadedBytes int64 `json:"uploadedBytes"`

	// +checkatomic
	EstimatedBytes int64 `json:"estimatedBytes"`

	// +checkatomic
	TotalCachedFiles int32 `json:"cachedFiles"`
	// +checkatomic
	TotalHashedFiles int32 `json:"hashedFiles"`

	// +checkatomic
	TotalExcludedFiles int32 `json:"excludedFiles"`
	// +checkatomic
	TotalExcludedDirs int32 `json:"excludedDirs"`

	// +checkatomic
	FatalErrorCount int32 `json:"errors"`
	// +checkatomic
	IgnoredErrorCount int32 `json:"ignoredErrors"`
	// +checkatomic
	EstimatedFiles int32 `json:"estimatedFiles"`

	CurrentDirectory string `json:"directory"`

	LastErrorPath string `json:"lastErrorPath"`
	LastError     string `json:"lastError"`
}

UploadCounters represents a snapshot of upload counters.

type UploadProgress 

type UploadProgress interface {
	// UploadStarted is emitted once at the start of an upload
	UploadStarted()

	// UploadFinished is emitted once at the end of an upload
	UploadFinished()

	// CachedFile is emitted whenever uploader reuses previously uploaded entry without hashing the file.
	CachedFile(path string, size int64)

	// HashingFile is emitted at the beginning of hashing of a given file.
	HashingFile(fname string)

	// ExcludedFile is emitted when a file is excluded.
	ExcludedFile(fname string, size int64)

	// ExcludedDir is emitted when a directory is excluded.
	ExcludedDir(dirname string)

	// FinishedHashingFile is emitted at the end of hashing of a given file.
	FinishedHashingFile(fname string, numBytes int64)

	// FinishedFile is emitted when the uploader is done with a file, regardless of if it was hashed
	// or cached. If an error was encountered it reports that too. A call to FinishedFile gives no
	// information about the reachability of the file in checkpoints that may occur close to the
	// time this function is called.
	FinishedFile(fname string, err error)

	// HashedBytes is emitted while hashing any blocks of bytes.
	HashedBytes(numBytes int64)

	// Error is emitted when an error is encountered.
	Error(path string, err error, isIgnored bool)

	// UploadedBytes is emitted whenever bytes are written to the blob storage.
	UploadedBytes(numBytes int64)

	// StartedDirectory is emitted whenever a directory starts being uploaded.
	StartedDirectory(dirname string)

	// FinishedDirectory is emitted whenever a directory is finished uploading.
	FinishedDirectory(dirname string)

	// EstimatedDataSize is emitted whenever the size of upload is estimated.
	EstimatedDataSize(fileCount int, totalBytes int64)
}

UploadProgress is invoked by uploader to report status of file and directory uploads.

type Uploader 

type Uploader struct {
	Progress UploadProgress

	// automatically cancel the Upload after certain number of bytes
	MaxUploadBytes int64

	// probability with cached entries will be ignored, must be [0..100]
	// 0=always use cached object entries if possible
	// 100=never use cached entries
	ForceHashPercentage float64

	// Number of files to hash and upload in parallel.
	ParallelUploads int

	// Enable snapshot actions
	EnableActions bool

	// override the directory log level and entry log verbosity.
	OverrideDirLogDetail   *policy.LogDetail
	OverrideEntryLogDetail *policy.LogDetail

	// Fail the entire snapshot on source file/directory error.
	FailFast bool

	// How frequently to create checkpoint snapshot entries.
	CheckpointInterval time.Duration

	// When set to true, do not ignore any files, regardless of policy settings.
	DisableIgnoreRules bool

	// Labels to apply to every checkpoint made for this snapshot.
	CheckpointLabels map[string]string
	// contains filtered or unexported fields
}

Uploader supports efficient uploading files and directories to repository.

func NewUploader 

func NewUploader(r repo.RepositoryWriter) *Uploader

NewUploader creates new Uploader object for a given repository.

func (*Uploader) Cancel 

func (u *Uploader) Cancel()

Cancel requests cancellation of an upload that's in progress. Will typically result in an incomplete snapshot.

func (*Uploader) IsCanceled added in v0.6.0 

func (u *Uploader) IsCanceled() bool

IsCanceled returns true if the upload is canceled.

func (*Uploader) Upload 

func (u *Uploader) Upload(
	ctx context.Context,
	source fs.Entry,
	policyTree *policy.Tree,
	sourceInfo snapshot.SourceInfo,
	previousManifests ...*snapshot.Manifest,
) (*snapshot.Manifest, error)

Upload uploads contents of the specified filesystem entry (file or directory) to the repository and returns snapshot.Manifest with statistics. Old snapshot manifest, when provided can be used to speed up uploads by utilizing hash cache.

type Verifier added in v0.11.0 

type Verifier struct {
	// contains filtered or unexported fields
}

Verifier allows efficient verification of large amounts of filesystem entries in parallel.

func NewVerifier added in v0.11.0 

func NewVerifier(ctx context.Context, rep repo.Repository, opts VerifierOptions) *Verifier

NewVerifier creates a verifier.

func (*Verifier) InParallel added in v0.11.0 

func (v *Verifier) InParallel(ctx context.Context, enqueue func(tw *TreeWalker) error) error

InParallel starts parallel verification and invokes the provided function which can call Process() on in the provided TreeWalker.

func (*Verifier) ShowStats added in v0.11.0 

func (v *Verifier) ShowStats(ctx context.Context)

ShowStats logs verification statistics.

func (*Verifier) VerifyFile added in v0.11.0 

func (v *Verifier) VerifyFile(ctx context.Context, oid object.ID, entryPath string) error

VerifyFile verifies a single file object (using content check, blob map check or full read).

type VerifierOptions added in v0.11.0 

type VerifierOptions struct {
	VerifyFilesPercent float64
	FileQueueLength    int
	Parallelism        int
	MaxErrors          int
	BlobMap            map[blob.ID]blob.Metadata
}

VerifierOptions provides options for the verifier.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.