snapshotfs

package
v0.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 11, 2020 License: Apache-2.0 Imports: 26 Imported by: 7

Documentation

Overview

Package snapshotfs implements virtual filesystem on top of snapshots in repo.Repository.

Index

Constants

View Source 
const (
	IncompleteReasonCheckpoint   = "checkpoint"
	IncompleteReasonCanceled     = "canceled"
	IncompleteReasonLimitReached = "limit reached"
)

reasons why a snapshot is incomplete

View Source 
const DefaultCheckpointInterval = 45 * time.Minute

DefaultCheckpointInterval is the default frequency of mid-upload checkpointing.

Variables

This section is empty.

Functions

func AllSourcesEntry 

func AllSourcesEntry(rep repo.Repository) fs.Directory

AllSourcesEntry returns fs.Directory that contains the list of all snapshot sources found in the repository.

func DirectoryEntry 

func DirectoryEntry(rep repo.Repository, objectID object.ID, dirSummary *fs.DirectorySummary) fs.Directory

DirectoryEntry returns fs.Directory based on repository object with the specified ID. The existence or validity of the directory object is not validated until its contents are read.

func EntryFromDirEntry added in v0.4.0 

func EntryFromDirEntry(r repo.Repository, md *snapshot.DirEntry) (fs.Entry, error)

EntryFromDirEntry returns a filesystem entry based on the directory entry.

func SnapshotRoot 

func SnapshotRoot(rep repo.Repository, man *snapshot.Manifest) (fs.Entry, error)

SnapshotRoot returns fs.Entry representing the root of a snapshot.

Types

type CountingUploadProgress added in v0.5.2 

type CountingUploadProgress struct {
	NullUploadProgress
	// contains filtered or unexported fields
}

CountingUploadProgress is an implementation of UploadProgress that accumulates counters.

func (*CountingUploadProgress) CachedFile added in v0.5.2 

func (p *CountingUploadProgress) CachedFile(fname string, numBytes int64)

CachedFile implements UploadProgress

func (*CountingUploadProgress) FinishedHashingFile added in v0.5.2 

func (p *CountingUploadProgress) FinishedHashingFile(fname string, numBytes int64)

FinishedHashingFile implements UploadProgress

func (*CountingUploadProgress) HashedBytes added in v0.5.2 

func (p *CountingUploadProgress) HashedBytes(numBytes int64)

HashedBytes implements UploadProgress

func (*CountingUploadProgress) IgnoredError added in v0.6.0 

func (p *CountingUploadProgress) IgnoredError(path string, err error)

IgnoredError implements UploadProgress

func (*CountingUploadProgress) Snapshot added in v0.5.2 

func (p *CountingUploadProgress) Snapshot() UploadCounters

Snapshot captures current snapshot of the upload.

func (*CountingUploadProgress) StartedDirectory added in v0.5.2 

func (p *CountingUploadProgress) StartedDirectory(dirname string)

StartedDirectory implements UploadProgress

func (*CountingUploadProgress) UploadStarted added in v0.5.2 

func (p *CountingUploadProgress) UploadStarted(previousFileCount int, previousTotalFileSize int64)

UploadStarted implements UploadProgress

type NullUploadProgress added in v0.5.2 

type NullUploadProgress struct {
}

NullUploadProgress is an implementation of UploadProgress that does not produce any output.

func (*NullUploadProgress) CachedFile added in v0.5.2 

func (p *NullUploadProgress) CachedFile(fname string, numBytes int64)

CachedFile implements UploadProgress

func (*NullUploadProgress) Checkpoint added in v0.6.0 

func (p *NullUploadProgress) Checkpoint()

Checkpoint implements UploadProgress

func (*NullUploadProgress) FinishedDirectory added in v0.5.2 

func (p *NullUploadProgress) FinishedDirectory(dirname string)

FinishedDirectory implements UploadProgress

func (*NullUploadProgress) FinishedHashingFile added in v0.5.2 

func (p *NullUploadProgress) FinishedHashingFile(fname string, numBytes int64)

FinishedHashingFile implements UploadProgress

func (*NullUploadProgress) HashedBytes added in v0.5.2 

func (p *NullUploadProgress) HashedBytes(numBytes int64)

HashedBytes implements UploadProgress

func (*NullUploadProgress) HashingFile added in v0.5.2 

func (p *NullUploadProgress) HashingFile(fname string)

HashingFile implements UploadProgress

func (*NullUploadProgress) IgnoredError added in v0.6.0 

func (p *NullUploadProgress) IgnoredError(path string, err error)

IgnoredError implements UploadProgress

func (*NullUploadProgress) StartedDirectory added in v0.5.2 

func (p *NullUploadProgress) StartedDirectory(dirname string)

StartedDirectory implements UploadProgress

func (*NullUploadProgress) UploadFinished added in v0.5.2 

func (p *NullUploadProgress) UploadFinished()

UploadFinished implements UploadProgress

func (*NullUploadProgress) UploadStarted added in v0.5.2 

func (p *NullUploadProgress) UploadStarted(previousFileCount int, previousTotalSize int64)

UploadStarted implements UploadProgress

func (*NullUploadProgress) UploadedBytes added in v0.5.2 

func (p *NullUploadProgress) UploadedBytes(numBytes int64)

UploadedBytes implements UploadProgress

type TreeWalker 

type TreeWalker struct {
	Parallelism    int
	RootEntries    []fs.Entry
	ObjectCallback func(entry fs.Entry) error
	// EntryID extracts or generates an id from an fs.Entry.
	// It can be used to eliminate duplicate entries when in a FS
	EntryID func(entry fs.Entry) interface{}
	// contains filtered or unexported fields
}

TreeWalker holds information for concurrently walking down FS trees specified by their roots

func NewTreeWalker 

func NewTreeWalker() *TreeWalker

NewTreeWalker creates new tree walker.

func (*TreeWalker) Run 

func (w *TreeWalker) Run(ctx context.Context) error

Run walks the given tree roots

type UploadCounters added in v0.5.2 

type UploadCounters struct {
	TotalCachedBytes int64 `json:"cachedBytes"`
	TotalHashedBytes int64 `json:"hashedBytes"`

	TotalCachedFiles int32 `json:"cachedFiles"`
	TotalHashedFiles int32 `json:"hashedFiles"`

	TotalIgnoredErrors int32 `json:"ignoredErrors"`

	CurrentDirectory string `json:"directory"`

	LastErrorPath string `json:"lastErrorPath"`
	LastError     string `json:"lastError"`
}

UploadCounters represents a snapshot of upload counters.

type UploadProgress 

type UploadProgress interface {
	// UploadStarted is emitted once at the start of an upload
	UploadStarted(previousFileCount int, previousTotalSize int64)

	// UploadFinished is emitted once at the end of an upload
	UploadFinished()

	// CachedFile is emitted whenever uploader reuses previously uploaded entry without hashing the file.
	CachedFile(path string, size int64)

	// HashingFile is emitted at the beginning of hashing of a given file.
	HashingFile(fname string)

	// FinishedHashingFile is emitted at the end of hashing of a given file.
	FinishedHashingFile(fname string, numBytes int64)

	// HashedBytes is emitted while hashing any blocks of bytes.
	HashedBytes(numBytes int64)

	// IgnoredError is emitted when an error is encountered and ignored
	IgnoredError(path string, err error)

	// UploadedBytes is emitted whenever bytes are written to the blob storage.
	UploadedBytes(numBytes int64)

	// StartedDirectory is emitted whenever a directory starts being uploaded.
	StartedDirectory(dirname string)

	// FinishedDirectory is emitted whenever a directory is finished uploading.
	FinishedDirectory(dirname string)

	// Checkpoint is emitted whenever snapshot is checkpointed.
	Checkpoint()
}

UploadProgress is invoked by by uploader to report status of file and directory uploads.

type Uploader 

type Uploader struct {
	Progress UploadProgress

	// automatically cancel the Upload after certain number of bytes
	MaxUploadBytes int64

	// ignore read errors
	IgnoreReadErrors bool

	// probability with cached entries will be ignored, must be [0..100]
	// 0=always use cached object entries if possible
	// 100=never use cached entries
	ForceHashPercentage int

	// Number of files to hash and upload in parallel.
	ParallelUploads int

	// How frequently to create checkpoint snapshot entries.
	CheckpointInterval time.Duration
	// contains filtered or unexported fields
}

Uploader supports efficient uploading files and directories to repository.

func NewUploader 

func NewUploader(r repo.Repository) *Uploader

NewUploader creates new Uploader object for a given repository.

func (*Uploader) Cancel 

func (u *Uploader) Cancel()

Cancel requests cancellation of an upload that's in progress. Will typically result in an incomplete snapshot.

func (*Uploader) IsCanceled added in v0.6.0 

func (u *Uploader) IsCanceled() bool

IsCanceled returns true if the upload is canceled.

func (*Uploader) Upload 

func (u *Uploader) Upload(
	ctx context.Context,
	source fs.Entry,
	policyTree *policy.Tree,
	sourceInfo snapshot.SourceInfo,
	previousManifests ...*snapshot.Manifest,
) (*snapshot.Manifest, error)

Upload uploads contents of the specified filesystem entry (file or directory) to the repository and returns snapshot.Manifest with statistics. Old snapshot manifest, when provided can be used to speed up uploads by utilizing hash cache.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.