Documentation
¶
Overview ¶
Package chunker provides wrappers for Fs and Object which split large files in chunks
Index ¶
- Variables
- func NewFs(name, rpath string, m configmap.Mapper) (fs.Fs, error)
- type Fs
- func (f *Fs) About(ctx context.Context) (*fs.Usage, error)
- func (f *Fs) ChangeNotify(ctx context.Context, notifyFunc func(string, fs.EntryType), ...)
- func (f *Fs) CleanUp(ctx context.Context) error
- func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object, error)
- func (f *Fs) DirMove(ctx context.Context, src fs.Fs, srcRemote, dstRemote string) error
- func (f *Fs) Features() *fs.Features
- func (f *Fs) Hashes() hash.Set
- func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err error)
- func (f *Fs) ListR(ctx context.Context, dir string, callback fs.ListRCallback) (err error)
- func (f *Fs) Mkdir(ctx context.Context, dir string) error
- func (f *Fs) Move(ctx context.Context, src fs.Object, remote string) (fs.Object, error)
- func (f *Fs) Name() string
- func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error)
- func (f *Fs) Precision() time.Duration
- func (f *Fs) Purge(ctx context.Context) error
- func (f *Fs) Put(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error)
- func (f *Fs) PutStream(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error)
- func (f *Fs) PutUnchecked(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error)
- func (f *Fs) Rmdir(ctx context.Context, dir string) error
- func (f *Fs) Root() string
- func (f *Fs) SetWrapper(wrapper fs.Fs)
- func (f *Fs) String() string
- func (f *Fs) UnWrap() fs.Fs
- func (f *Fs) WrapFs() fs.Fs
- type Object
- func (o *Object) Fs() fs.Info
- func (o *Object) Hash(ctx context.Context, hashType hash.Type) (string, error)
- func (o *Object) ID() string
- func (o *Object) ModTime(ctx context.Context) time.Time
- func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (rc io.ReadCloser, err error)
- func (o *Object) Remote() string
- func (o *Object) Remove(ctx context.Context) (err error)
- func (o *Object) SetModTime(ctx context.Context, mtime time.Time) error
- func (o *Object) Size() int64
- func (o *Object) Storable() bool
- func (o *Object) String() string
- func (o *Object) UnWrap() fs.Object
- func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) error
- type ObjectInfo
- func (oi *ObjectInfo) Fs() fs.Info
- func (oi *ObjectInfo) Hash(ctx context.Context, hashType hash.Type) (string, error)
- func (oi *ObjectInfo) ModTime(ctx context.Context) time.Time
- func (oi *ObjectInfo) Remote() string
- func (oi *ObjectInfo) Size() int64
- func (oi *ObjectInfo) Storable() bool
- func (oi *ObjectInfo) String() string
- type Options
Constants ¶
This section is empty.
Variables ¶
var (
ErrChunkOverflow = errors.New("chunk number overflow")
)
standard chunker errors
Functions ¶
Types ¶
type Fs ¶
type Fs struct {
// contains filtered or unexported fields
}
Fs represents a wrapped fs.Fs
func (*Fs) ChangeNotify ¶
func (f *Fs) ChangeNotify(ctx context.Context, notifyFunc func(string, fs.EntryType), pollIntervalChan <-chan time.Duration)
ChangeNotify calls the passed function with a path that has had changes. If the implementation uses polling, it should adhere to the given interval.
Replace data chunk names by the name of composite file. Ignore temporary and control chunks.
func (*Fs) CleanUp ¶
CleanUp the trash in the Fs
Implement this if you have a way of emptying the trash or otherwise cleaning up old versions of files.
func (*Fs) Copy ¶
Copy src to this remote using server side copy operations.
This is stored with the remote path given ¶
It returns the destination Object and a possible error ¶
Will only be called if src.Fs().Name() == f.Name()
If it isn't possible then return fs.ErrorCantCopy
func (*Fs) DirMove ¶
DirMove moves src, srcRemote to this remote at dstRemote using server side move operations.
Will only be called if src.Fs().Name() == f.Name()
If it isn't possible then return fs.ErrorCantDirMove
If destination exists then return fs.ErrorDirExists
func (*Fs) Hashes ¶
Hashes returns the supported hash sets. Chunker advertises a hash type if and only if it can be calculated for files of any size, non-chunked or composite.
func (*Fs) List ¶
List the objects and directories in dir into entries. The entries can be returned in any order but should be for a complete directory.
dir should be "" to list the root, and should not have trailing slashes.
This should return ErrDirNotFound if the directory isn't found.
Commands normally cleanup all temporary chunks in case of a failure. However, if rclone dies unexpectedly, it can leave behind a bunch of hidden temporary chunks. List and its underlying chunkEntries() silently skip all temporary chunks in the directory. It's okay if they belong to an unfinished command running in parallel.
However, there is no way to discover dead temporary chunks atm. As a workaround users can use `purge` to forcibly remove the whole directory together with dead chunks. In future a flag named like `--chunker-list-hidden` may be added to rclone that will tell List to reveal hidden chunks.
func (*Fs) ListR ¶
ListR lists the objects and directories of the Fs starting from dir recursively into out.
dir should be "" to start from the root, and should not have trailing slashes.
This should return ErrDirNotFound if the directory isn't found.
It should call callback for each tranche of entries read. These need not be returned in any particular order. If callback returns an error then the listing will stop immediately.
Don't implement this unless you have a more efficient way of listing recursively than doing a directory traversal.
func (*Fs) Mkdir ¶
Mkdir makes the directory (container, bucket)
Shouldn't return an error if it already exists
func (*Fs) Move ¶
Move src to this remote using server side move operations.
This is stored with the remote path given ¶
It returns the destination Object and a possible error ¶
Will only be called if src.Fs().Name() == f.Name()
If it isn't possible then return fs.ErrorCantMove
func (*Fs) NewObject ¶
NewObject finds the Object at remote.
Please note that every NewObject invocation will scan the whole directory. Using here something like fs.DirCache might improve performance (yet making the logic more complex).
Note that chunker prefers analyzing file names rather than reading the content of meta object assuming that directory scans are fast but opening even a small file can be slow on some backends.
func (*Fs) Purge ¶
Purge all files in the root and the root directory
Implement this if you have a way of deleting all the files quicker than just running Remove() on the result of List()
Return an error if it doesn't exist.
This command will chain to `purge` from wrapped remote. As a result it removes not only composite chunker files with their active chunks but also all hidden temporary chunks in the directory.
func (*Fs) Put ¶
func (f *Fs) Put(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error)
Put into the remote path with the given modTime and size.
May create the object even if it returns an error - if so will return the object and the error, otherwise will return nil and the error
func (*Fs) PutStream ¶
func (f *Fs) PutStream(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error)
PutStream uploads to the remote path with the modTime given of indeterminate size
func (*Fs) PutUnchecked ¶
func (f *Fs) PutUnchecked(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error)
PutUnchecked uploads the object
This will create a duplicate if we upload a new file without checking to see if there is one already - use Put() for that.
func (*Fs) Rmdir ¶
Rmdir removes the directory (container, bucket) if empty
Return an error if it doesn't exist or isn't empty
func (*Fs) SetWrapper ¶
SetWrapper sets the Fs that is wrapping this Fs
type Object ¶
type Object struct {
// contains filtered or unexported fields
}
Object represents a composite file wrapping one or more data chunks
func (*Object) Hash ¶
Hash returns the selected checksum of the file. If no checksum is available it returns "".
Hash won't fail with `unsupported` error but return empty hash string if a particular hashsum type is not supported
Hash takes hashsum from metadata if available or requests it from wrapped remote for non-chunked files. Metadata (if meta format is not 'none') is by default kept only for composite files. In the "All" hashing mode chunker will force metadata on all files if particular hashsum type is not supported by wrapped remote.
Note that Hash prefers the wrapped hashsum for non-chunked file, then tries to read it from metadata. This in theory handles the unusual case when a small file has been tampered on the level of wrapped remote but chunker is unaware of that.
func (*Object) Open ¶
func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (rc io.ReadCloser, err error)
Open opens the file for read. Call Close() on the returned io.ReadCloser
func (*Object) Remove ¶
Remove an object (chunks and metadata, if any)
Remove deletes only active chunks of the composite object. It does not try to look for temporary chunks because they could belong to another command modifying this composite file in parallel.
Commands normally cleanup all temporary chunks in case of a failure. However, if rclone dies unexpectedly, it can leave hidden temporary chunks, which cannot be discovered using the `list` command. Remove does not try to search for such chunks or to delete them. Sometimes this can lead to strange results eg. when `list` shows that directory is empty but `rmdir` refuses to remove it because on the level of wrapped remote it's actually *not* empty. As a workaround users can use `purge` to forcibly remove it.
In future, a flag `--chunker-delete-hidden` may be added which tells Remove to search directory for hidden chunks and remove them too (at the risk of breaking parallel commands).
Remove is the only operation allowed on the composite files with invalid or future metadata format. We don't let user copy/move/update unsupported composite files. Let's at least let her get rid of them, just complain loudly.
This can litter directory with orphan chunks of unsupported types, but as long as we remove meta object, even future releases will treat the composite file as removed and refuse to act upon it.
Disclaimer: corruption can still happen if unsupported file is removed and then recreated with the same name. Unsupported control chunks will get re-picked by a more recent rclone version with unexpected results. This can be helped by the `delete hidden` flag above or at least the user has been warned.
func (*Object) SetModTime ¶
SetModTime sets the modification time of the file
type ObjectInfo ¶
type ObjectInfo struct {
// contains filtered or unexported fields
}
ObjectInfo describes a wrapped fs.ObjectInfo for being the source
func (*ObjectInfo) Fs ¶
func (oi *ObjectInfo) Fs() fs.Info
Fs returns read only access to the Fs that this object is part of
func (*ObjectInfo) Hash ¶
Hash returns the selected checksum of the wrapped file It returns "" if no checksum is available or if this info doesn't wrap the complete file.
func (*ObjectInfo) ModTime ¶
func (oi *ObjectInfo) ModTime(ctx context.Context) time.Time
ModTime returns the modification time
func (*ObjectInfo) Storable ¶
func (oi *ObjectInfo) Storable() bool
Storable returns whether object is storable
func (*ObjectInfo) String ¶
func (oi *ObjectInfo) String() string
String returns string representation
type Options ¶
type Options struct {
Remote string `config:"remote"`
ChunkSize fs.SizeSuffix `config:"chunk_size"`
NameFormat string `config:"name_format"`
StartFrom int `config:"start_from"`
MetaFormat string `config:"meta_format"`
HashType string `config:"hash_type"`
FailHard bool `config:"fail_hard"`
}
Options defines the configuration for this backend
Source Files