snapshotfs

package
v0.9.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 7, 2021 License: Apache-2.0 Imports: 36 Imported by: 7

Documentation

Overview

Package snapshotfs implements virtual filesystem on top of snapshots in repo.Repository.

Index

Constants

View Source 
const (
	IncompleteReasonCheckpoint   = "checkpoint"
	IncompleteReasonCanceled     = "canceled"
	IncompleteReasonLimitReached = "limit reached"
)

reasons why a snapshot is incomplete.

View Source 
const DefaultCheckpointInterval = 45 * time.Minute

DefaultCheckpointInterval is the default frequency of mid-upload checkpointing.

Variables

This section is empty.

Functions

func AllSourcesEntry 

func AllSourcesEntry(rep repo.Repository) fs.Directory

AllSourcesEntry returns fs.Directory that contains the list of all snapshot sources found in the repository.

func AutoDetectEntryFromObjectID added in v0.7.2 

func AutoDetectEntryFromObjectID(ctx context.Context, rep repo.Repository, oid object.ID, maybeName string) fs.Entry

AutoDetectEntryFromObjectID returns fs.Entry (either file or directory) for the provided object ID. It uses heuristics to determine whether object ID is possibly a directory and treats it as such.

func DirectoryEntry 

func DirectoryEntry(rep repo.Repository, objectID object.ID, dirSummary *fs.DirectorySummary) fs.Directory

DirectoryEntry returns fs.Directory based on repository object with the specified ID. The existence or validity of the directory object is not validated until its contents are read.

func EntryFromDirEntry added in v0.4.0 

func EntryFromDirEntry(r repo.Repository, md *snapshot.DirEntry) fs.Entry

EntryFromDirEntry returns a filesystem entry based on the directory entry.

func Estimate added in v0.8.0 

func Estimate(ctx context.Context, rep repo.Repository, entry fs.Directory, policyTree *policy.Tree, progress EstimateProgress, maxExamplesPerBucket int) error

Estimate walks the provided directory tree and invokes provided progress callback as it discovers items to be snapshotted.

func FilesystemDirectoryFromIDWithPath added in v0.7.2 

func FilesystemDirectoryFromIDWithPath(ctx context.Context, rep repo.Repository, rootID string, consistentAttributes bool) (fs.Directory, error)

FilesystemDirectoryFromIDWithPath returns a filesystem directory entry for the provided object ID, which can be a snapshot manifest ID or an object ID with path.

func FilesystemEntryFromIDWithPath added in v0.7.2 

func FilesystemEntryFromIDWithPath(ctx context.Context, rep repo.Repository, rootID string, consistentAttributes bool) (fs.Entry, error)

FilesystemEntryFromIDWithPath returns a filesystem entry for the provided object ID, which can be a snapshot manifest ID or an object ID with path. If multiple snapshots match and they don't agree on root object attributes and consistentAttributes==true the function fails, otherwise it returns the latest of the snapshots.

func GetEntryFromPlaceholder added in v0.9.0 

func GetEntryFromPlaceholder(ctx context.Context, r repo.Repository, defp snapshot.HasDirEntryOrNil) (fs.Entry, error)

GetEntryFromPlaceholder returns a fs.Entry for shallow placeholder defp referencing a real Entry in Repository r.

func GetNestedEntry added in v0.7.2 

func GetNestedEntry(ctx context.Context, startingDir fs.Entry, pathElements []string) (fs.Entry, error)

GetNestedEntry returns nested entry with a given name path.

func IsDirectoryID added in v0.7.2 

func IsDirectoryID(oid object.ID) bool

IsDirectoryID determines whether given object ID represents a directory.

func ParseObjectIDWithPath added in v0.7.2 

func ParseObjectIDWithPath(ctx context.Context, rep repo.Repository, objectIDWithPath string) (object.ID, error)

ParseObjectIDWithPath interprets the given ID string (which could be an object ID optionally followed by nested path specification) and returns corresponding object.ID.

func SnapshotRoot 

func SnapshotRoot(rep repo.Repository, man *snapshot.Manifest) (fs.Entry, error)

SnapshotRoot returns fs.Entry representing the root of a snapshot.

Types

type CountingUploadProgress added in v0.5.2 

type CountingUploadProgress struct {
	NullUploadProgress
	// contains filtered or unexported fields
}

CountingUploadProgress is an implementation of UploadProgress that accumulates counters.

func (*CountingUploadProgress) CachedFile added in v0.5.2 

func (p *CountingUploadProgress) CachedFile(fname string, numBytes int64)

CachedFile implements UploadProgress.

func (*CountingUploadProgress) Error added in v0.8.0 

func (p *CountingUploadProgress) Error(path string, err error, isIgnored bool)

Error implements UploadProgress.

func (*CountingUploadProgress) EstimatedDataSize added in v0.7.0 

func (p *CountingUploadProgress) EstimatedDataSize(numFiles int, numBytes int64)

EstimatedDataSize implements UploadProgress.

func (*CountingUploadProgress) ExcludedDir added in v0.8.0 

func (p *CountingUploadProgress) ExcludedDir(dirname string)

ExcludedDir implements UploadProgress.

func (*CountingUploadProgress) ExcludedFile added in v0.8.0 

func (p *CountingUploadProgress) ExcludedFile(fname string, numBytes int64)

ExcludedFile implements UploadProgress.

func (*CountingUploadProgress) FinishedHashingFile added in v0.5.2 

func (p *CountingUploadProgress) FinishedHashingFile(fname string, numBytes int64)

FinishedHashingFile implements UploadProgress.

func (*CountingUploadProgress) HashedBytes added in v0.5.2 

func (p *CountingUploadProgress) HashedBytes(numBytes int64)

HashedBytes implements UploadProgress.

func (*CountingUploadProgress) Snapshot added in v0.5.2 

func (p *CountingUploadProgress) Snapshot() UploadCounters

Snapshot captures current snapshot of the upload.

func (*CountingUploadProgress) StartedDirectory added in v0.5.2 

func (p *CountingUploadProgress) StartedDirectory(dirname string)

StartedDirectory implements UploadProgress.

func (*CountingUploadProgress) UITaskCounters added in v0.8.0 

func (p *CountingUploadProgress) UITaskCounters(final bool) map[string]uitask.CounterValue

UITaskCounters returns UI task counters.

func (*CountingUploadProgress) UploadStarted added in v0.5.2 

func (p *CountingUploadProgress) UploadStarted()

UploadStarted implements UploadProgress.

func (*CountingUploadProgress) UploadedBytes added in v0.8.0 

func (p *CountingUploadProgress) UploadedBytes(numBytes int64)

UploadedBytes implements UploadProgress.

type EstimateProgress added in v0.8.0 

type EstimateProgress interface {
	Processing(ctx context.Context, dirname string)
	Error(ctx context.Context, filename string, err error, isIgnored bool)
	Stats(ctx context.Context, s *snapshot.Stats, includedFiles, excludedFiles SampleBuckets, excludedDirs []string, final bool)
}

EstimateProgress must be provided by the caller of Estimate to report results.

type NullUploadProgress added in v0.5.2 

type NullUploadProgress struct{}

NullUploadProgress is an implementation of UploadProgress that does not produce any output.

func (*NullUploadProgress) CachedFile added in v0.5.2 

func (p *NullUploadProgress) CachedFile(fname string, numBytes int64)

CachedFile implements UploadProgress.

func (*NullUploadProgress) Error added in v0.8.0 

func (p *NullUploadProgress) Error(path string, err error, isIgnored bool)

Error implements UploadProgress.

func (*NullUploadProgress) EstimatedDataSize added in v0.7.0 

func (p *NullUploadProgress) EstimatedDataSize(fileCount int, totalBytes int64)

EstimatedDataSize implements UploadProgress.

func (*NullUploadProgress) ExcludedDir added in v0.8.0 

func (p *NullUploadProgress) ExcludedDir(dirname string)

ExcludedDir implements UploadProgress.

func (*NullUploadProgress) ExcludedFile added in v0.8.0 

func (p *NullUploadProgress) ExcludedFile(fname string, numBytes int64)

ExcludedFile implements UploadProgress.

func (*NullUploadProgress) FinishedDirectory added in v0.5.2 

func (p *NullUploadProgress) FinishedDirectory(dirname string)

FinishedDirectory implements UploadProgress.

func (*NullUploadProgress) FinishedHashingFile added in v0.5.2 

func (p *NullUploadProgress) FinishedHashingFile(fname string, numBytes int64)

FinishedHashingFile implements UploadProgress.

func (*NullUploadProgress) HashedBytes added in v0.5.2 

func (p *NullUploadProgress) HashedBytes(numBytes int64)

HashedBytes implements UploadProgress.

func (*NullUploadProgress) HashingFile added in v0.5.2 

func (p *NullUploadProgress) HashingFile(fname string)

HashingFile implements UploadProgress.

func (*NullUploadProgress) StartedDirectory added in v0.5.2 

func (p *NullUploadProgress) StartedDirectory(dirname string)

StartedDirectory implements UploadProgress.

func (*NullUploadProgress) UploadFinished added in v0.5.2 

func (p *NullUploadProgress) UploadFinished()

UploadFinished implements UploadProgress.

func (*NullUploadProgress) UploadStarted added in v0.5.2 

func (p *NullUploadProgress) UploadStarted()

UploadStarted implements UploadProgress.

func (*NullUploadProgress) UploadedBytes added in v0.5.2 

func (p *NullUploadProgress) UploadedBytes(numBytes int64)

UploadedBytes implements UploadProgress.

type SampleBucket added in v0.8.0 

type SampleBucket struct {
	MinSize   int64    `json:"minSize"`
	Count     int      `json:"count"`
	TotalSize int64    `json:"totalSize"`
	Examples  []string `json:"examples,omitempty"`
}

SampleBucket keeps track of count and total size of files above in certain size range and includes small number of examples of such files.

type SampleBuckets added in v0.8.0 

type SampleBuckets []*SampleBucket

SampleBuckets is a collection of buckets for interesting file sizes sorted in descending order.

type TreeWalker 

type TreeWalker struct {
	Parallelism    int
	RootEntries    []fs.Entry
	ObjectCallback func(entry fs.Entry) error
	// EntryID extracts or generates an id from an fs.Entry.
	// It can be used to eliminate duplicate entries when in a FS
	EntryID func(entry fs.Entry) interface{}
	// contains filtered or unexported fields
}

TreeWalker holds information for concurrently walking down FS trees specified by their roots.

func NewTreeWalker 

func NewTreeWalker() *TreeWalker

NewTreeWalker creates new tree walker.

func (*TreeWalker) Run 

func (w *TreeWalker) Run(ctx context.Context) error

Run walks the given tree roots.

type UploadCounters added in v0.5.2 

type UploadCounters struct {
	TotalCachedBytes   int64 `json:"cachedBytes"`
	TotalHashedBytes   int64 `json:"hashedBytes"`
	TotalUploadedBytes int64 `json:"uploadedBytes"`

	EstimatedBytes int64 `json:"estimatedBytes"`

	TotalCachedFiles int32 `json:"cachedFiles"`
	TotalHashedFiles int32 `json:"hashedFiles"`

	TotalExcludedFiles int32 `json:"excludedFiles"`
	TotalExcludedDirs  int32 `json:"excludedDirs"`

	FatalErrorCount   int32 `json:"errors"`
	IgnoredErrorCount int32 `json:"ignoredErrors"`
	EstimatedFiles    int32 `json:"estimatedFiles"`

	CurrentDirectory string `json:"directory"`

	LastErrorPath string `json:"lastErrorPath"`
	LastError     string `json:"lastError"`
}

UploadCounters represents a snapshot of upload counters.

type UploadProgress 

type UploadProgress interface {
	// UploadStarted is emitted once at the start of an upload
	UploadStarted()

	// UploadFinished is emitted once at the end of an upload
	UploadFinished()

	// CachedFile is emitted whenever uploader reuses previously uploaded entry without hashing the file.
	CachedFile(path string, size int64)

	// HashingFile is emitted at the beginning of hashing of a given file.
	HashingFile(fname string)

	// ExcludedFile is emitted when a file is excluded.
	ExcludedFile(fname string, size int64)

	// ExcludedDir is emitted when a directory is excluded.
	ExcludedDir(dirname string)

	// FinishedHashingFile is emitted at the end of hashing of a given file.
	FinishedHashingFile(fname string, numBytes int64)

	// HashedBytes is emitted while hashing any blocks of bytes.
	HashedBytes(numBytes int64)

	// Error is emitted when an error is encountered.
	Error(path string, err error, isIgnored bool)

	// UploadedBytes is emitted whenever bytes are written to the blob storage.
	UploadedBytes(numBytes int64)

	// StartedDirectory is emitted whenever a directory starts being uploaded.
	StartedDirectory(dirname string)

	// FinishedDirectory is emitted whenever a directory is finished uploading.
	FinishedDirectory(dirname string)

	// EstimatedDataSize is emitted whenever the size of upload is estimated.
	EstimatedDataSize(fileCount int, totalBytes int64)
}

UploadProgress is invoked by by uploader to report status of file and directory uploads.

type Uploader 

type Uploader struct {
	Progress UploadProgress

	// automatically cancel the Upload after certain number of bytes
	MaxUploadBytes int64

	// probability with cached entries will be ignored, must be [0..100]
	// 0=always use cached object entries if possible
	// 100=never use cached entries
	ForceHashPercentage float64

	// Number of files to hash and upload in parallel.
	ParallelUploads int

	// Enable snapshot actions
	EnableActions bool

	// override the directory log level and entry log verbosity.
	OverrideDirLogDetail   *policy.LogDetail
	OverrideEntryLogDetail *policy.LogDetail

	// Fail the entire snapshot on source file/directory error.
	FailFast bool

	// How frequently to create checkpoint snapshot entries.
	CheckpointInterval time.Duration

	// When set to true, do not ignore any files, regardless of policy settings.
	DisableIgnoreRules bool
	// contains filtered or unexported fields
}

Uploader supports efficient uploading files and directories to repository.

func NewUploader 

func NewUploader(r repo.RepositoryWriter) *Uploader

NewUploader creates new Uploader object for a given repository.

func (*Uploader) Cancel 

func (u *Uploader) Cancel()

Cancel requests cancellation of an upload that's in progress. Will typically result in an incomplete snapshot.

func (*Uploader) IsCanceled added in v0.6.0 

func (u *Uploader) IsCanceled() bool

IsCanceled returns true if the upload is canceled.

func (*Uploader) Upload 

func (u *Uploader) Upload(
	ctx context.Context,
	source fs.Entry,
	policyTree *policy.Tree,
	sourceInfo snapshot.SourceInfo,
	previousManifests ...*snapshot.Manifest,
) (*snapshot.Manifest, error)

Upload uploads contents of the specified filesystem entry (file or directory) to the repository and returns snapshot.Manifest with statistics. Old snapshot manifest, when provided can be used to speed up uploads by utilizing hash cache.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.