chunker

package
v0.0.0-...-b4aeae5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 7, 2024 License: MIT Imports: 26 Imported by: 0

Documentation

Overview

Package chunker provides wrappers for Fs and Object which split large files in chunks

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrChunkOverflow = errors.New("chunk number overflow")
	ErrMetaTooBig    = errors.New("metadata is too big")
	ErrMetaUnknown   = errors.New("unknown metadata, please upgrade rclone")
)

standard chunker errors

Functions

func NewFs 

func NewFs(ctx context.Context, name, rpath string, m configmap.Mapper) (fs.Fs, error)

NewFs constructs an Fs from the path, container:path

Types

type Fs 

type Fs struct {
	// contains filtered or unexported fields
}

Fs represents a wrapped fs.Fs

func (*Fs) About 

func (f *Fs) About(ctx context.Context) (*fs.Usage, error)

About gets quota information from the Fs

func (*Fs) CanQuickRename 

func (f *Fs) CanQuickRename() bool

CanQuickRename returns true if the Fs supports a quick rename operation

func (*Fs) ChangeNotify 

func (f *Fs) ChangeNotify(ctx context.Context, notifyFunc func(string, fs.EntryType), pollIntervalChan <-chan time.Duration)

ChangeNotify calls the passed function with a path that has had changes. If the implementation uses polling, it should adhere to the given interval.

Replace data chunk names by the name of composite file. Ignore temporary and control chunks.

func (*Fs) CleanUp 

func (f *Fs) CleanUp(ctx context.Context) error

CleanUp the trash in the Fs

Implement this if you have a way of emptying the trash or otherwise cleaning up old versions of files.

func (*Fs) Copy 

func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object, error)

Copy src to this remote using server-side copy operations.

This is stored with the remote path given.

It returns the destination Object and a possible error.

Will only be called if src.Fs().Name() == f.Name()

If it isn't possible then return fs.ErrorCantCopy

func (*Fs) DirMove 

func (f *Fs) DirMove(ctx context.Context, src fs.Fs, srcRemote, dstRemote string) error

DirMove moves src, srcRemote to this remote at dstRemote using server-side move operations.

Will only be called if src.Fs().Name() == f.Name()

If it isn't possible then return fs.ErrorCantDirMove

If destination exists then return fs.ErrorDirExists

func (*Fs) DirSetModTime 

func (f *Fs) DirSetModTime(ctx context.Context, dir string, modTime time.Time) error

DirSetModTime sets the directory modtime for dir

func (*Fs) Features 

func (f *Fs) Features() *fs.Features

Features returns the optional features of this Fs

func (*Fs) Hashes 

func (f *Fs) Hashes() hash.Set

Hashes returns the supported hash sets. Chunker advertises a hash type if and only if it can be calculated for files of any size, non-chunked or composite.

func (*Fs) List 

func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err error)

List the objects and directories in dir into entries. The entries can be returned in any order but should be for a complete directory.

dir should be "" to list the root, and should not have trailing slashes.

This should return ErrDirNotFound if the directory isn't found.

Commands normally cleanup all temporary chunks in case of a failure. However, if rclone dies unexpectedly, it can leave behind a bunch of hidden temporary chunks. List and its underlying chunkEntries() silently skip all temporary chunks in the directory. It's okay if they belong to an unfinished command running in parallel.

However, there is no way to discover dead temporary chunks atm. As a workaround users can use `purge` to forcibly remove the whole directory together with dead chunks. In future a flag named like `--chunker-list-hidden` may be added to rclone that will tell List to reveal hidden chunks.

func (*Fs) ListR 

func (f *Fs) ListR(ctx context.Context, dir string, callback fs.ListRCallback) (err error)

ListR lists the objects and directories of the Fs starting from dir recursively into out.

dir should be "" to start from the root, and should not have trailing slashes.

This should return ErrDirNotFound if the directory isn't found.

It should call callback for each tranche of entries read. These need not be returned in any particular order. If callback returns an error then the listing will stop immediately.

Don't implement this unless you have a more efficient way of listing recursively than doing a directory traversal.

func (*Fs) Mkdir 

func (f *Fs) Mkdir(ctx context.Context, dir string) error

Mkdir makes the directory (container, bucket)

Shouldn't return an error if it already exists

func (*Fs) MkdirMetadata 

func (f *Fs) MkdirMetadata(ctx context.Context, dir string, metadata fs.Metadata) (fs.Directory, error)

MkdirMetadata makes the root directory of the Fs object

func (*Fs) Move 

func (f *Fs) Move(ctx context.Context, src fs.Object, remote string) (fs.Object, error)

Move src to this remote using server-side move operations.

This is stored with the remote path given.

It returns the destination Object and a possible error.

Will only be called if src.Fs().Name() == f.Name()

If it isn't possible then return fs.ErrorCantMove

func (*Fs) Name 

func (f *Fs) Name() string

Name of the remote (as passed into NewFs)

func (*Fs) NewObject 

func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error)

NewObject finds the Object at remote.

Please note that every NewObject invocation will scan the whole directory. Using here something like fs.DirCache might improve performance (yet making the logic more complex).

Note that chunker prefers analyzing file names rather than reading the content of meta object assuming that directory scans are fast but opening even a small file can be slow on some backends.

func (*Fs) Precision 

func (f *Fs) Precision() time.Duration

Precision returns the precision of this Fs

func (*Fs) Purge 

func (f *Fs) Purge(ctx context.Context, dir string) error

Purge all files in the directory

Implement this if you have a way of deleting all the files quicker than just running Remove() on the result of List()

Return an error if it doesn't exist.

This command will chain to `purge` from wrapped remote. As a result it removes not only composite chunker files with their active chunks but also all hidden temporary chunks in the directory.

func (*Fs) Put 

func (f *Fs) Put(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error)

Put into the remote path with the given modTime and size.

May create the object even if it returns an error - if so will return the object and the error, otherwise will return nil and the error

func (*Fs) PutStream 

func (f *Fs) PutStream(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error)

PutStream uploads to the remote path with the modTime given of indeterminate size

func (*Fs) PutUnchecked 

func (f *Fs) PutUnchecked(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error)

PutUnchecked uploads the object

This will create a duplicate if we upload a new file without checking to see if there is one already - use Put() for that.

func (*Fs) Rmdir 

func (f *Fs) Rmdir(ctx context.Context, dir string) error

Rmdir removes the directory (container, bucket) if empty

Return an error if it doesn't exist or isn't empty

func (*Fs) Root 

func (f *Fs) Root() string

Root of the remote (as passed into NewFs)

func (*Fs) SetWrapper 

func (f *Fs) SetWrapper(wrapper fs.Fs)

SetWrapper sets the Fs that is wrapping this Fs

func (*Fs) Shutdown 

func (f *Fs) Shutdown(ctx context.Context) error

Shutdown the backend, closing any background tasks and any cached connections.

func (*Fs) String 

func (f *Fs) String() string

String returns a description of the FS

func (*Fs) UnWrap 

func (f *Fs) UnWrap() fs.Fs

UnWrap returns the Fs that this Fs is wrapping

func (*Fs) WrapFs 

func (f *Fs) WrapFs() fs.Fs

WrapFs returns the Fs that is wrapping this Fs

type Object 

type Object struct {
	// contains filtered or unexported fields
}

Object represents a composite file wrapping one or more data chunks

func (*Object) Fs 

func (o *Object) Fs() fs.Info

Fs returns read only access to the Fs that this object is part of

func (*Object) Hash 

func (o *Object) Hash(ctx context.Context, hashType hash.Type) (string, error)

Hash returns the selected checksum of the file. If no checksum is available it returns "".

Hash won't fail with `unsupported` error but return empty hash string if a particular hashsum type is not supported

Hash takes hashsum from metadata if available or requests it from wrapped remote for non-chunked files. Metadata (if meta format is not 'none') is by default kept only for composite files. In the "All" hashing mode chunker will force metadata on all files if particular hashsum type is not supported by wrapped remote.

Note that Hash prefers the wrapped hashsum for non-chunked file, then tries to read it from metadata. This in theory handles the unusual case when a small file has been tampered on the level of wrapped remote but chunker is unaware of that.

func (*Object) ID 

func (o *Object) ID() string

ID returns the ID of the Object if known, or "" if not

func (*Object) ModTime 

func (o *Object) ModTime(ctx context.Context) time.Time

ModTime returns the modification time of the file

func (*Object) Open 

func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (rc io.ReadCloser, err error)

Open opens the file for read. Call Close() on the returned io.ReadCloser

func (*Object) Remote 

func (o *Object) Remote() string

Remote returns the remote path

func (*Object) Remove 

func (o *Object) Remove(ctx context.Context) (err error)

Remove an object (chunks and metadata, if any)

Remove deletes only active chunks of the composite object. It does not try to look for temporary chunks because they could belong to another command modifying this composite file in parallel.

Commands normally cleanup all temporary chunks in case of a failure. However, if rclone dies unexpectedly, it can leave hidden temporary chunks, which cannot be discovered using the `list` command. Remove does not try to search for such chunks or to delete them. Sometimes this can lead to strange results e.g. when `list` shows that directory is empty but `rmdir` refuses to remove it because on the level of wrapped remote it's actually *not* empty. As a workaround users can use `purge` to forcibly remove it.

In future, a flag `--chunker-delete-hidden` may be added which tells Remove to search directory for hidden chunks and remove them too (at the risk of breaking parallel commands).

Remove is the only operation allowed on the composite files with invalid or future metadata format. We don't let user copy/move/update unsupported composite files. Let's at least let her get rid of them, just complain loudly.

This can litter directory with orphan chunks of unsupported types, but as long as we remove meta object, even future releases will treat the composite file as removed and refuse to act upon it.

Disclaimer: corruption can still happen if unsupported file is removed and then recreated with the same name. Unsupported control chunks will get re-picked by a more recent rclone version with unexpected results. This can be helped by the `delete hidden` flag above or at least the user has been warned.

func (*Object) SetModTime 

func (o *Object) SetModTime(ctx context.Context, mtime time.Time) error

SetModTime sets the modification time of the file

func (*Object) Size 

func (o *Object) Size() int64

Size returns the size of the file

func (*Object) Storable 

func (o *Object) Storable() bool

Storable returns whether object is storable

func (*Object) String 

func (o *Object) String() string

Return a string version

func (*Object) UnWrap 

func (o *Object) UnWrap() fs.Object

UnWrap returns the wrapped Object

func (*Object) Update 

func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) error

Update in to the object with the modTime given of the given size

type ObjectInfo 

type ObjectInfo struct {
	// contains filtered or unexported fields
}

ObjectInfo describes a wrapped fs.ObjectInfo for being the source

func (*ObjectInfo) Fs 

func (oi *ObjectInfo) Fs() fs.Info

Fs returns read only access to the Fs that this object is part of

func (*ObjectInfo) Hash 

func (oi *ObjectInfo) Hash(ctx context.Context, hashType hash.Type) (string, error)

Hash returns the selected checksum of the wrapped file It returns "" if no checksum is available or if this info doesn't wrap the complete file.

func (*ObjectInfo) ModTime 

func (oi *ObjectInfo) ModTime(ctx context.Context) time.Time

ModTime returns the modification time

func (*ObjectInfo) Remote 

func (oi *ObjectInfo) Remote() string

Remote returns the remote path

func (*ObjectInfo) Size 

func (oi *ObjectInfo) Size() int64

Size returns the size of the file

func (*ObjectInfo) Storable 

func (oi *ObjectInfo) Storable() bool

Storable returns whether object is storable

func (*ObjectInfo) String 

func (oi *ObjectInfo) String() string

String returns string representation

type Options 

type Options struct {
	Remote       string        `config:"remote"`
	ChunkSize    fs.SizeSuffix `config:"chunk_size"`
	NameFormat   string        `config:"name_format"`
	StartFrom    int           `config:"start_from"`
	MetaFormat   string        `config:"meta_format"`
	HashType     string        `config:"hash_type"`
	FailHard     bool          `config:"fail_hard"`
	Transactions string        `config:"transactions"`
}

Options defines the configuration for this backend

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.