vfs

package
v1.59.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 22, 2022 License: MIT Imports: 26 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	ENOENT  = os.ErrNotExist
	EEXIST  = os.ErrExist
	EPERM   = os.ErrPermission
	EINVAL  = os.ErrInvalid
	ECLOSED = os.ErrClosed
)

Errors which have exact counterparts in os

View Source 
var Help = strings.ReplaceAll(`
### VFS - Virtual File System

This command uses the VFS layer. This adapts the cloud storage objects
that rclone uses into something which looks much more like a disk
filing system.

Cloud storage objects have lots of properties which aren't like disk
files - you can't extend them or write to the middle of them, so the
VFS layer has to deal with that. Because there is no one right way of
doing this there are various options explained below.

The VFS layer also implements a directory cache - this caches info
about files and directories (but not the data) in memory.

### VFS Directory Cache

Using the !--dir-cache-time! flag, you can control how long a
directory should be considered up to date and not refreshed from the
backend. Changes made through the VFS will appear immediately or
invalidate the cache.

    --dir-cache-time duration   Time to cache directory entries for (default 5m0s)
    --poll-interval duration    Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)

However, changes made directly on the cloud storage by the web
interface or a different copy of rclone will only be picked up once
the directory cache expires if the backend configured does not support
polling for changes. If the backend supports polling, changes will be
picked up within the polling interval.

You can send a !SIGHUP! signal to rclone for it to flush all
directory caches, regardless of how old they are.  Assuming only one
rclone instance is running, you can reset the cache like this:

    kill -SIGHUP $(pidof rclone)

If you configure rclone with a [remote control](/rc) then you can use
rclone rc to flush the whole directory cache:

    rclone rc vfs/forget

Or individual files or directories:

    rclone rc vfs/forget file=path/to/file dir=path/to/dir

### VFS File Buffering

The !--buffer-size! flag determines the amount of memory,
that will be used to buffer data in advance.

Each open file will try to keep the specified amount of data in memory
at all times. The buffered data is bound to one open file and won't be
shared.

This flag is a upper limit for the used memory per open file.  The
buffer will only use memory for data that is downloaded but not not
yet read. If the buffer is empty, only a small amount of memory will
be used.

The maximum memory used by rclone for buffering can be up to
!--buffer-size * open files!.

### VFS File Caching

These flags control the VFS file caching options. File caching is
necessary to make the VFS layer appear compatible with a normal file
system. It can be disabled at the cost of some compatibility.

For example you'll need to enable VFS caching if you want to read and
write simultaneously to a file.  See below for more details.

Note that the VFS cache is separate from the cache backend and you may
find that you need one or the other or both.

    --cache-dir string                   Directory rclone will use for caching.
    --vfs-cache-mode CacheMode           Cache mode off|minimal|writes|full (default off)
    --vfs-cache-max-age duration         Max age of objects in the cache (default 1h0m0s)
    --vfs-cache-max-size SizeSuffix      Max total size of objects in the cache (default off)
    --vfs-cache-poll-interval duration   Interval to poll the cache for stale objects (default 1m0s)
    --vfs-write-back duration            Time to writeback files after last use when using cache (default 5s)

If run with !-vv! rclone will print the location of the file cache.  The
files are stored in the user cache file area which is OS dependent but
can be controlled with !--cache-dir! or setting the appropriate
environment variable.

The cache has 4 different modes selected by !--vfs-cache-mode!.
The higher the cache mode the more compatible rclone becomes at the
cost of using disk space.

Note that files are written back to the remote only when they are
closed and if they haven't been accessed for !--vfs-write-back!
seconds. If rclone is quit or dies with files that haven't been
uploaded, these will be uploaded next time rclone is run with the same
flags.

If using !--vfs-cache-max-size! note that the cache may exceed this size
for two reasons.  Firstly because it is only checked every
!--vfs-cache-poll-interval!.  Secondly because open files cannot be
evicted from the cache.

You **should not** run two copies of rclone using the same VFS cache
with the same or overlapping remotes if using !--vfs-cache-mode > off!.
This can potentially cause data corruption if you do. You can work
around this by giving each rclone its own cache hierarchy with
!--cache-dir!. You don't need to worry about this if the remotes in
use don't overlap.

#### --vfs-cache-mode off

In this mode (the default) the cache will read directly from the remote and write
directly to the remote without caching anything on disk.

This will mean some operations are not possible

  * Files can't be opened for both read AND write
  * Files opened for write can't be seeked
  * Existing files opened for write must have O_TRUNC set
  * Files open for read with O_TRUNC will be opened write only
  * Files open for write only will behave as if O_TRUNC was supplied
  * Open modes O_APPEND, O_TRUNC are ignored
  * If an upload fails it can't be retried

#### --vfs-cache-mode minimal

This is very similar to "off" except that files opened for read AND
write will be buffered to disk.  This means that files opened for
write will be a lot more compatible, but uses the minimal disk space.

These operations are not possible

  * Files opened for write only can't be seeked
  * Existing files opened for write must have O_TRUNC set
  * Files opened for write only will ignore O_APPEND, O_TRUNC
  * If an upload fails it can't be retried

#### --vfs-cache-mode writes

In this mode files opened for read only are still read directly from
the remote, write only and read/write files are buffered to disk
first.

This mode should support all normal file system operations.

If an upload fails it will be retried at exponentially increasing
intervals up to 1 minute.

#### --vfs-cache-mode full

In this mode all reads and writes are buffered to and from disk. When
data is read from the remote this is buffered to disk as well.

In this mode the files in the cache will be sparse files and rclone
will keep track of which bits of the files it has downloaded.

So if an application only reads the starts of each file, then rclone
will only buffer the start of the file. These files will appear to be
their full size in the cache, but they will be sparse files with only
the data that has been downloaded present in them.

This mode should support all normal file system operations and is
otherwise identical to !--vfs-cache-mode! writes.

When reading a file rclone will read !--buffer-size! plus
!--vfs-read-ahead! bytes ahead.  The !--buffer-size! is buffered in memory
whereas the !--vfs-read-ahead! is buffered on disk.

When using this mode it is recommended that !--buffer-size! is not set
too large and !--vfs-read-ahead! is set large if required.

**IMPORTANT** not all file systems support sparse files. In particular
FAT/exFAT do not. Rclone will perform very badly if the cache
directory is on a filesystem which doesn't support sparse files and it
will log an ERROR message if one is detected.

#### Fingerprinting

Various parts of the VFS use fingerprinting to see if a local file
copy has changed relative to a remote file. Fingerprints are made
from:

- size
- modification time
- hash

where available on an object.

On some backends some of these attributes are slow to read (they take
an extra API call per object, or extra work per object).

For example !hash! is slow with the !local! and !sftp! backends as
they have to read the entire file and hash it, and !modtime! is slow
with the !s3!, !swift!, !ftp! and !qinqstor! backends because they
need to do an extra API call to fetch it.

If you use the !--vfs-fast-fingerprint! flag then rclone will not
include the slow operations in the fingerprint. This makes the
fingerprinting less accurate but much faster and will improve the
opening time of cached files.

If you are running a vfs cache over !local!, !s3! or !swift! backends
then using this flag is recommended.

Note that if you change the value of this flag, the fingerprints of
the files in the cache may be invalidated and the files will need to
be downloaded again.

### VFS Chunked Reading

When rclone reads files from a remote it reads them in chunks. This
means that rather than requesting the whole file rclone reads the
chunk specified.  This can reduce the used download quota for some
remotes by requesting only chunks from the remote that are actually
read, at the cost of an increased number of requests.

These flags control the chunking:

    --vfs-read-chunk-size SizeSuffix        Read the source objects in chunks (default 128M)
    --vfs-read-chunk-size-limit SizeSuffix  Max chunk doubling size (default off)

Rclone will start reading a chunk of size !--vfs-read-chunk-size!,
and then double the size for each read. When !--vfs-read-chunk-size-limit! is
specified, and greater than !--vfs-read-chunk-size!, the chunk size for each
open file will get doubled only until the specified value is reached. If the
value is "off", which is the default, the limit is disabled and the chunk size
will grow indefinitely.

With !--vfs-read-chunk-size 100M! and !--vfs-read-chunk-size-limit 0!
the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on.
When !--vfs-read-chunk-size-limit 500M! is specified, the result would be
0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.

Setting !--vfs-read-chunk-size! to !0! or "off" disables chunked reading.

### VFS Performance

These flags may be used to enable/disable features of the VFS for
performance or other reasons. See also the [chunked reading](#vfs-chunked-reading)
feature.

In particular S3 and Swift benefit hugely from the !--no-modtime! flag
(or use !--use-server-modtime! for a slightly different effect) as each
read of the modification time takes a transaction.

    --no-checksum     Don't compare checksums on up/download.
    --no-modtime      Don't read/write the modification time (can speed things up).
    --no-seek         Don't allow seeking in files.
    --read-only       Only allow read-only access.

Sometimes rclone is delivered reads or writes out of order. Rather
than seeking rclone will wait a short time for the in sequence read or
write to come in. These flags only come into effect when not using an
on disk cache file.

    --vfs-read-wait duration   Time to wait for in-sequence read before seeking (default 20ms)
    --vfs-write-wait duration  Time to wait for in-sequence write before giving error (default 1s)

When using VFS write caching (!--vfs-cache-mode! with value writes or full),
the global flag !--transfers! can be set to adjust the number of parallel uploads of
modified files from the cache (the related global flag !--checkers! has no effect on the VFS).

    --transfers int  Number of file transfers to run in parallel (default 4)

### VFS Case Sensitivity

Linux file systems are case-sensitive: two files can differ only
by case, and the exact case must be used when opening a file.

File systems in modern Windows are case-insensitive but case-preserving:
although existing files can be opened using any case, the exact case used
to create the file is preserved and available for programs to query.
It is not allowed for two files in the same directory to differ only by case.

Usually file systems on macOS are case-insensitive. It is possible to make macOS
file systems case-sensitive but that is not the default.

The !--vfs-case-insensitive! VFS flag controls how rclone handles these
two cases. If its value is "false", rclone passes file names to the remote
as-is. If the flag is "true" (or appears without a value on the
command line), rclone may perform a "fixup" as explained below.

The user may specify a file name to open/delete/rename/etc with a case
different than what is stored on the remote. If an argument refers
to an existing file with exactly the same name, then the case of the existing
file on the disk will be used. However, if a file name with exactly the same
name is not found but a name differing only by case exists, rclone will
transparently fixup the name. This fixup happens only when an existing file
is requested. Case sensitivity of file names created anew by rclone is
controlled by the underlying remote.

Note that case sensitivity of the operating system running rclone (the target)
may differ from case sensitivity of a file system presented by rclone (the source).
The flag controls whether "fixup" is performed to satisfy the target.

If the flag is not provided on the command line, then its default value depends
on the operating system where rclone runs: "true" on Windows and macOS, "false"
otherwise. If the flag is provided without a value, then it is "true".

### VFS Disk Options

This flag allows you to manually set the statistics about the filing system.
It can be useful when those statistics cannot be read correctly automatically.

    --vfs-disk-space-total-size    Manually set the total disk space size (example: 256G, default: -1)

### Alternate report of used bytes

Some backends, most notably S3, do not report the amount of bytes used.
If you need this information to be available when running !df! on the
filesystem, then pass the flag !--vfs-used-is-size! to rclone.
With this flag set, instead of relying on the backend to report this
information, rclone will scan the whole remote similar to !rclone size!
and compute the total used space itself.

_WARNING._ Contrary to !rclone size!, this flag ignores filters so that the
result is accurate. However, this is very inefficient and may cost lots of API
calls resulting in extra charges. Use it as a last resort and only with caching.
`, "!", "`")

Help contains text describing file and directory caching to add to the command help. Warning: "!" (sic) will be replaced by backticks below, 

but the pipe character "|" can be used as is.

Functions

This section is empty.

Types

type Dir 

type Dir struct {
	// contains filtered or unexported fields
}

Dir represents a directory entry

func (*Dir) AddVirtual added in v1.50.20 

func (d *Dir) AddVirtual(leaf string, size int64, isDir bool)

AddVirtual adds a virtual object of name and size to the directory

This will be replaced with a real object when it is read back from the remote.

This is used to add directory entries while things are uploading

func (*Dir) Create 

func (d *Dir) Create(name string, flags int) (*File, error)

Create makes a new file node

func (*Dir) DelVirtual added in v1.50.20 

func (d *Dir) DelVirtual(leaf string)

DelVirtual removes an object from the directory listing

It marks it as removed until it has confirmed the object is missing when the directory entries are re-read in.

This is used to remove directory entries after things have been deleted or renamed but before we've had confirmation from the backend.

func (*Dir) DirEntry 

func (d *Dir) DirEntry() (entry fs.DirEntry)

DirEntry returns the underlying fs.DirEntry

func (*Dir) ForgetAll 

func (d *Dir) ForgetAll() (hasVirtual bool)

ForgetAll forgets directory entries for this directory and any children.

It does not invalidate or clear the cache of the parent directory.

It returns true if the directory or any of its children had virtual entries so could not be forgotten. Children which didn't have virtual entries and children with virtual entries will be forgotten even if true is returned.

func (*Dir) ForgetPath 

func (d *Dir) ForgetPath(relativePath string, entryType fs.EntryType)

ForgetPath clears the cache for itself and all subdirectories if they match the given path. The path is specified relative from the directory it is called from. The cache of the parent directory is marked as stale, but not cleared otherwise. It is not possible to traverse the directory tree upwards, i.e. you cannot clear the cache for the Dir's ancestors or siblings.

func (*Dir) Fs added in v1.50.20 

func (d *Dir) Fs() fs.Fs

Fs returns the Fs that the Dir is on

func (*Dir) Inode 

func (d *Dir) Inode() uint64

Inode returns the inode number - satisfies Node interface

func (*Dir) IsDir 

func (d *Dir) IsDir() bool

IsDir returns true for Dir - satisfies Node interface

func (*Dir) IsFile 

func (d *Dir) IsFile() bool

IsFile returns false for Dir - satisfies Node interface

func (*Dir) Mkdir 

func (d *Dir) Mkdir(name string) (*Dir, error)

Mkdir creates a new directory

func (*Dir) ModTime 

func (d *Dir) ModTime() time.Time

ModTime returns the modification time of the directory

func (*Dir) Mode 

func (d *Dir) Mode() (mode os.FileMode)

Mode bits of the directory - satisfies Node interface

func (*Dir) Name 

func (d *Dir) Name() (name string)

Name (base) of the directory - satisfies Node interface

func (*Dir) Node 

func (d *Dir) Node() Node

Node returns the Node associated with this - satisfies Noder interface

func (*Dir) Open 

func (d *Dir) Open(flags int) (fd Handle, err error)

Open the directory according to the flags provided

func (*Dir) Path 

func (d *Dir) Path() (name string)

Path of the directory - satisfies Node interface

func (*Dir) ReadDirAll 

func (d *Dir) ReadDirAll() (items Nodes, err error)

ReadDirAll reads the contents of the directory sorted

func (*Dir) Remove 

func (d *Dir) Remove() error

Remove the directory

func (*Dir) RemoveAll 

func (d *Dir) RemoveAll() error

RemoveAll removes the directory and any contents recursively

func (*Dir) RemoveName 

func (d *Dir) RemoveName(name string) error

RemoveName removes the entry with the given name from the receiver, which must be a directory. The entry to be removed may correspond to a file (unlink) or to a directory (rmdir).

func (*Dir) Rename 

func (d *Dir) Rename(oldName, newName string, destDir *Dir) error

Rename the file

func (*Dir) SetModTime 

func (d *Dir) SetModTime(modTime time.Time) error

SetModTime sets the modTime for this dir

func (*Dir) SetSys added in v1.50.20 

func (d *Dir) SetSys(x interface{})

SetSys sets the underlying data source (can be nil) - satisfies Node interface

func (*Dir) Size 

func (d *Dir) Size() int64

Size of the directory

func (*Dir) Stat 

func (d *Dir) Stat(name string) (node Node, err error)

Stat looks up a specific entry in the receiver.

Stat should return a Node corresponding to the entry. If the name does not exist in the directory, Stat should return ENOENT.

Stat need not to handle the names "." and "..".

func (*Dir) String 

func (d *Dir) String() string

String converts it to printable

func (*Dir) Sync 

func (d *Dir) Sync() error

Sync the directory

Note that we don't do anything except return OK

func (*Dir) Sys 

func (d *Dir) Sys() interface{}

Sys returns underlying data source (can be nil) - satisfies Node interface

func (*Dir) Truncate 

func (d *Dir) Truncate(size int64) error

Truncate changes the size of the named file.

func (*Dir) VFS 

func (d *Dir) VFS() *VFS

VFS returns the instance of the VFS

type DirHandle 

type DirHandle struct {
	// contains filtered or unexported fields
}

DirHandle represents an open directory

func (DirHandle) Chdir 

func (h DirHandle) Chdir() error

func (DirHandle) Chmod 

func (h DirHandle) Chmod(mode os.FileMode) error

func (DirHandle) Chown 

func (h DirHandle) Chown(uid, gid int) error

func (*DirHandle) Close 

func (fh *DirHandle) Close() (err error)

Close closes the handle

func (DirHandle) Fd 

func (h DirHandle) Fd() uintptr

func (DirHandle) Flush 

func (h DirHandle) Flush() (err error)

func (DirHandle) Name 

func (h DirHandle) Name() string

func (*DirHandle) Node 

func (fh *DirHandle) Node() Node

Node returns the Node associated with this - satisfies Noder interface

func (DirHandle) Read 

func (h DirHandle) Read(b []byte) (n int, err error)

func (DirHandle) ReadAt 

func (h DirHandle) ReadAt(b []byte, off int64) (n int, err error)

func (*DirHandle) Readdir 

func (fh *DirHandle) Readdir(n int) (fis []os.FileInfo, err error)

Readdir reads the contents of the directory associated with file and returns a slice of up to n FileInfo values, as would be returned by Lstat, in directory order. Subsequent calls on the same file will yield further FileInfos.

If n > 0, Readdir returns at most n FileInfo structures. In this case, if Readdir returns an empty slice, it will return a non-nil error explaining why. At the end of a directory, the error is io.EOF.

If n <= 0, Readdir returns all the FileInfo from the directory in a single slice. In this case, if Readdir succeeds (reads all the way to the end of the directory), it returns the slice and a nil error. If it encounters an error before the end of the directory, Readdir returns the FileInfo read until that point and a non-nil error.

func (*DirHandle) Readdirnames 

func (fh *DirHandle) Readdirnames(n int) (names []string, err error)

Readdirnames reads and returns a slice of names from the directory f.

If n > 0, Readdirnames returns at most n names. In this case, if Readdirnames returns an empty slice, it will return a non-nil error explaining why. At the end of a directory, the error is io.EOF.

If n <= 0, Readdirnames returns all the names from the directory in a single slice. In this case, if Readdirnames succeeds (reads all the way to the end of the directory), it returns the slice and a nil error. If it encounters an error before the end of the directory, Readdirnames returns the names read until that point and a non-nil error.

func (DirHandle) Release 

func (h DirHandle) Release() (err error)

func (DirHandle) Seek 

func (h DirHandle) Seek(offset int64, whence int) (ret int64, err error)

func (*DirHandle) Stat 

func (fh *DirHandle) Stat() (fi os.FileInfo, err error)

Stat returns info about the current directory

func (*DirHandle) String 

func (fh *DirHandle) String() string

String converts it to printable

func (DirHandle) Sync 

func (h DirHandle) Sync() error

func (DirHandle) Truncate 

func (h DirHandle) Truncate(size int64) error

func (DirHandle) Write 

func (h DirHandle) Write(b []byte) (n int, err error)

func (DirHandle) WriteAt 

func (h DirHandle) WriteAt(b []byte, off int64) (n int, err error)

func (DirHandle) WriteString 

func (h DirHandle) WriteString(s string) (n int, err error)

type Error 

type Error byte

Error describes low level errors in a cross platform way. 

const (
	OK Error = iota
	ENOTEMPTY
	ESPIPE
	EBADF
	EROFS
	ENOSYS
)

Low level errors

func (Error) Error 

func (e Error) Error() string

Error renders the error as a string

type File 

type File struct {
	// contains filtered or unexported fields
}

File represents a file

func (*File) Dir 

func (f *File) Dir() *Dir

Dir returns the directory this file is in

func (*File) DirEntry 

func (f *File) DirEntry() (entry fs.DirEntry)

DirEntry returns the underlying fs.DirEntry - may be nil

func (*File) Fs added in v1.50.20 

func (f *File) Fs() fs.Fs

Fs returns the underlying Fs for the file

func (*File) Inode 

func (f *File) Inode() uint64

Inode returns the inode number - satisfies Node interface

func (*File) IsDir 

func (f *File) IsDir() bool

IsDir returns false for File - satisfies Node interface

func (*File) IsFile 

func (f *File) IsFile() bool

IsFile returns true for File - satisfies Node interface

func (*File) ModTime 

func (f *File) ModTime() (modTime time.Time)

ModTime returns the modified time of the file

if NoModTime is set then it returns the mod time of the directory

func (*File) Mode 

func (f *File) Mode() (mode os.FileMode)

Mode bits of the file or directory - satisfies Node interface

func (*File) Name 

func (f *File) Name() (name string)

Name (base) of the directory - satisfies Node interface

func (*File) Node 

func (f *File) Node() Node

Node returns the Node associated with this - satisfies Noder interface

func (*File) Open 

func (f *File) Open(flags int) (fd Handle, err error)

Open a file according to the flags provided 

O_RDONLY open the file read-only.
O_WRONLY open the file write-only.
O_RDWR   open the file read-write.

O_APPEND append data to the file when writing.
O_CREATE create a new file if none exists.
O_EXCL   used with O_CREATE, file must not exist
O_SYNC   open for synchronous I/O.
O_TRUNC  if possible, truncate file when opened

We ignore O_SYNC and O_EXCL

func (*File) Path 

func (f *File) Path() string

Path returns the full path of the file

func (*File) Remove 

func (f *File) Remove() (err error)

Remove the file

func (*File) RemoveAll 

func (f *File) RemoveAll() error

RemoveAll the file - same as remove for files

func (*File) SetModTime 

func (f *File) SetModTime(modTime time.Time) error

SetModTime sets the modtime for the file

if NoModTime is set then it does nothing

func (*File) SetSys added in v1.50.20 

func (f *File) SetSys(x interface{})

SetSys sets the underlying data source (can be nil) - satisfies Node interface

func (*File) Size 

func (f *File) Size() int64

Size of the file

func (*File) String 

func (f *File) String() string

String converts it to printable

func (*File) Sync 

func (f *File) Sync() error

Sync the file

Note that we don't do anything except return OK

func (*File) Sys 

func (f *File) Sys() interface{}

Sys returns underlying data source (can be nil) - satisfies Node interface

func (*File) Truncate 

func (f *File) Truncate(size int64) (err error)

Truncate changes the size of the named file.

func (*File) VFS 

func (f *File) VFS() *VFS

VFS returns the instance of the VFS

type Handle 

type Handle interface {
	OsFiler
	// Additional methods useful for FUSE filesystems
	Flush() error
	Release() error
	Node() Node
}

Handle is the interface satisfied by open files or directories. It is the methods on *os.File, plus a few more useful for FUSE filingsystems. Not all of them are supported.

type Node 

type Node interface {
	os.FileInfo
	IsFile() bool
	Inode() uint64
	SetModTime(modTime time.Time) error
	Sync() error
	Remove() error
	RemoveAll() error
	DirEntry() fs.DirEntry
	VFS() *VFS
	Open(flags int) (Handle, error)
	Truncate(size int64) error
	Path() string
	SetSys(interface{})
}

Node represents either a directory (*Dir) or a file (*File)

type Noder 

type Noder interface {
	fmt.Stringer
	Node() Node
}

Noder represents something which can return a node

type Nodes 

type Nodes []Node

Nodes is a slice of Node

func (Nodes) Len 

func (ns Nodes) Len() int

Sort functions

func (Nodes) Less 

func (ns Nodes) Less(i, j int) bool

func (Nodes) Swap 

func (ns Nodes) Swap(i, j int)

type OsFiler 

type OsFiler interface {
	Chdir() error
	Chmod(mode os.FileMode) error
	Chown(uid, gid int) error
	Close() error
	Fd() uintptr
	Name() string
	Read(b []byte) (n int, err error)
	ReadAt(b []byte, off int64) (n int, err error)
	Readdir(n int) ([]os.FileInfo, error)
	Readdirnames(n int) (names []string, err error)
	Seek(offset int64, whence int) (ret int64, err error)
	Stat() (os.FileInfo, error)
	Sync() error
	Truncate(size int64) error
	Write(b []byte) (n int, err error)
	WriteAt(b []byte, off int64) (n int, err error)
	WriteString(s string) (n int, err error)
}

OsFiler is the methods on *os.File

type RWFileHandle 

type RWFileHandle struct {
	// contains filtered or unexported fields
}

RWFileHandle is a handle that can be open for read and write.

It will be open to a temporary file which, when closed, will be transferred to the remote.

func (*RWFileHandle) Chdir added in v1.50.20 

func (fh *RWFileHandle) Chdir() error

Chdir changes the current working directory to the file, which must be a directory.

func (*RWFileHandle) Chmod added in v1.50.20 

func (fh *RWFileHandle) Chmod(mode os.FileMode) error

Chmod changes the mode of the file to mode.

func (*RWFileHandle) Chown added in v1.50.20 

func (fh *RWFileHandle) Chown(uid, gid int) error

Chown changes the numeric uid and gid of the named file.

func (*RWFileHandle) Close 

func (fh *RWFileHandle) Close() error

Close closes the file

func (*RWFileHandle) Fd added in v1.50.20 

func (fh *RWFileHandle) Fd() uintptr

Fd returns the integer Unix file descriptor referencing the open file.

func (*RWFileHandle) Flush 

func (fh *RWFileHandle) Flush() error

Flush is called each time the file or directory is closed. Because there can be multiple file descriptors referring to a single opened file, Flush can be called multiple times.

func (*RWFileHandle) Name added in v1.50.20 

func (fh *RWFileHandle) Name() string

Name returns the name of the file from the underlying Object.

func (*RWFileHandle) Node 

func (fh *RWFileHandle) Node() Node

Node returns the Node associated with this - satisfies Noder interface

func (*RWFileHandle) Read 

func (fh *RWFileHandle) Read(b []byte) (n int, err error)

Read bytes from the file

func (*RWFileHandle) ReadAt 

func (fh *RWFileHandle) ReadAt(b []byte, off int64) (n int, err error)

ReadAt bytes from the file at off

func (*RWFileHandle) Readdir added in v1.50.20 

func (fh *RWFileHandle) Readdir(n int) ([]os.FileInfo, error)

Readdir reads the contents of the directory associated with file.

func (*RWFileHandle) Readdirnames added in v1.50.20 

func (fh *RWFileHandle) Readdirnames(n int) (names []string, err error)

Readdirnames reads the contents of the directory associated with file.

func (*RWFileHandle) Release 

func (fh *RWFileHandle) Release() error

Release is called when we are finished with the file handle

It isn't called directly from userspace so the error is ignored by the kernel

func (*RWFileHandle) Seek 

func (fh *RWFileHandle) Seek(offset int64, whence int) (ret int64, err error)

Seek to new file position

func (*RWFileHandle) Size 

func (fh *RWFileHandle) Size() int64

Size returns the size of the underlying file

func (*RWFileHandle) Stat 

func (fh *RWFileHandle) Stat() (os.FileInfo, error)

Stat returns info about the file

func (*RWFileHandle) String 

func (fh *RWFileHandle) String() string

String converts it to printable

func (*RWFileHandle) Sync 

func (fh *RWFileHandle) Sync() error

Sync commits the current contents of the file to stable storage. Typically, this means flushing the file system's in-memory copy of recently written data to disk.

func (*RWFileHandle) Truncate 

func (fh *RWFileHandle) Truncate(size int64) (err error)

Truncate file to given size

func (*RWFileHandle) Write 

func (fh *RWFileHandle) Write(b []byte) (n int, err error)

Write bytes to the file

func (*RWFileHandle) WriteAt 

func (fh *RWFileHandle) WriteAt(b []byte, off int64) (n int, err error)

WriteAt bytes to the file at off

func (*RWFileHandle) WriteString 

func (fh *RWFileHandle) WriteString(s string) (n int, err error)

WriteString a string to the file

type ReadFileHandle 

type ReadFileHandle struct {
	// contains filtered or unexported fields
}

ReadFileHandle is an open for read file handle on a File

func (ReadFileHandle) Chdir 

func (h ReadFileHandle) Chdir() error

func (ReadFileHandle) Chmod 

func (h ReadFileHandle) Chmod(mode os.FileMode) error

func (ReadFileHandle) Chown 

func (h ReadFileHandle) Chown(uid, gid int) error

func (*ReadFileHandle) Close 

func (fh *ReadFileHandle) Close() error

Close closes the file

func (ReadFileHandle) Fd 

func (h ReadFileHandle) Fd() uintptr

func (*ReadFileHandle) Flush 

func (fh *ReadFileHandle) Flush() error

Flush is called each time the file or directory is closed. Because there can be multiple file descriptors referring to a single opened file, Flush can be called multiple times.

func (ReadFileHandle) Name 

func (h ReadFileHandle) Name() string

func (*ReadFileHandle) Node 

func (fh *ReadFileHandle) Node() Node

Node returns the Node associated with this - satisfies Noder interface

func (*ReadFileHandle) Read 

func (fh *ReadFileHandle) Read(p []byte) (n int, err error)

Read reads up to len(p) bytes into p. It returns the number of bytes read (0 <= n <= len(p)) and any error encountered. Even if Read returns n < len(p), it may use all of p as scratch space during the call. If some data is available but not len(p) bytes, Read conventionally returns what is available instead of waiting for more.

When Read encounters an error or end-of-file condition after successfully reading n > 0 bytes, it returns the number of bytes read. It may return the (non-nil) error from the same call or return the error (and n == 0) from a subsequent call. An instance of this general case is that a Reader returning a non-zero number of bytes at the end of the input stream may return either err == EOF or err == nil. The next Read should return 0, EOF.

Callers should always process the n > 0 bytes returned before considering the error err. Doing so correctly handles I/O errors that happen after reading some bytes and also both of the allowed EOF behaviors.

Implementations of Read are discouraged from returning a zero byte count with a nil error, except when len(p) == 0. Callers should treat a return of 0 and nil as indicating that nothing happened; in particular it does not indicate EOF.

Implementations must not retain p.

func (*ReadFileHandle) ReadAt 

func (fh *ReadFileHandle) ReadAt(p []byte, off int64) (n int, err error)

ReadAt reads len(p) bytes into p starting at offset off in the underlying input source. It returns the number of bytes read (0 <= n <= len(p)) and any error encountered.

When ReadAt returns n < len(p), it returns a non-nil error explaining why more bytes were not returned. In this respect, ReadAt is stricter than Read.

Even if ReadAt returns n < len(p), it may use all of p as scratch space during the call. If some data is available but not len(p) bytes, ReadAt blocks until either all the data is available or an error occurs. In this respect ReadAt is different from Read.

If the n = len(p) bytes returned by ReadAt are at the end of the input source, ReadAt may return either err == EOF or err == nil.

If ReadAt is reading from an input source with a seek offset, ReadAt should not affect nor be affected by the underlying seek offset.

Clients of ReadAt can execute parallel ReadAt calls on the same input source.

Implementations must not retain p.

func (ReadFileHandle) Readdir 

func (h ReadFileHandle) Readdir(n int) ([]os.FileInfo, error)

func (ReadFileHandle) Readdirnames 

func (h ReadFileHandle) Readdirnames(n int) (names []string, err error)

func (*ReadFileHandle) Release 

func (fh *ReadFileHandle) Release() error

Release is called when we are finished with the file handle

It isn't called directly from userspace so the error is ignored by the kernel

func (*ReadFileHandle) Seek 

func (fh *ReadFileHandle) Seek(offset int64, whence int) (n int64, err error)

Seek the file - returns ESPIPE if seeking isn't possible

func (*ReadFileHandle) Size 

func (fh *ReadFileHandle) Size() int64

Size returns the size of the underlying file

func (*ReadFileHandle) Stat 

func (fh *ReadFileHandle) Stat() (os.FileInfo, error)

Stat returns info about the file

func (*ReadFileHandle) String 

func (fh *ReadFileHandle) String() string

String converts it to printable

func (ReadFileHandle) Sync 

func (h ReadFileHandle) Sync() error

func (ReadFileHandle) Truncate 

func (h ReadFileHandle) Truncate(size int64) error

func (ReadFileHandle) Write 

func (h ReadFileHandle) Write(b []byte) (n int, err error)

func (ReadFileHandle) WriteAt 

func (h ReadFileHandle) WriteAt(b []byte, off int64) (n int, err error)

func (ReadFileHandle) WriteString 

func (h ReadFileHandle) WriteString(s string) (n int, err error)

type VFS 

type VFS struct {
	Opt vfscommon.Options
	// contains filtered or unexported fields
}

VFS represents the top level filing system

func New 

func New(f fs.Fs, opt *vfscommon.Options) *VFS

New creates a new VFS and root directory. If opt is nil, then DefaultOpt will be used

func (*VFS) AddVirtual added in v1.50.20 

func (vfs *VFS) AddVirtual(remote string, size int64, isDir bool) error

AddVirtual adds the object (file or dir) to the directory cache

func (*VFS) Chtimes added in v1.50.20 

func (vfs *VFS) Chtimes(name string, atime time.Time, mtime time.Time) error

Chtimes changes the access and modification times of the named file, similar to the Unix utime() or utimes() functions.

The underlying filesystem may truncate or round the values to a less precise time unit.

func (*VFS) CleanUp 

func (vfs *VFS) CleanUp() error

CleanUp deletes the contents of the on disk cache

func (*VFS) Create added in v1.50.20 

func (vfs *VFS) Create(name string) (Handle, error)

Create creates the named file with mode 0666 (before umask), truncating it if it already exists. If successful, methods on the returned File can be used for I/O; the associated file descriptor has mode O_RDWR.

func (*VFS) FlushDirCache 

func (vfs *VFS) FlushDirCache()

FlushDirCache empties the directory cache

func (*VFS) Fs added in v1.6.1 

func (vfs *VFS) Fs() fs.Fs

Fs returns the Fs passed into the New call

func (*VFS) Mkdir added in v1.50.20 

func (vfs *VFS) Mkdir(name string, perm os.FileMode) error

Mkdir creates a new directory with the specified name and permission bits (before umask).

func (*VFS) Open added in v1.50.20 

func (vfs *VFS) Open(name string) (Handle, error)

Open opens the named file for reading. If successful, methods on the returned file can be used for reading; the associated file descriptor has mode O_RDONLY.

func (*VFS) OpenFile 

func (vfs *VFS) OpenFile(name string, flags int, perm os.FileMode) (fd Handle, err error)

OpenFile a file according to the flags and perm provided

func (*VFS) ReadDir added in v1.50.20 

func (vfs *VFS) ReadDir(dirname string) ([]os.FileInfo, error)

ReadDir reads the directory named by dirname and returns a list of directory entries sorted by filename.

func (*VFS) ReadFile added in v1.50.20 

func (vfs *VFS) ReadFile(filename string) (b []byte, err error)

ReadFile reads the file named by filename and returns the contents. A successful call returns err == nil, not err == EOF. Because ReadFile reads the whole file, it does not treat an EOF from Read as an error to be reported.

func (*VFS) Remove added in v1.50.20 

func (vfs *VFS) Remove(name string) error

Remove removes the named file or (empty) directory.

func (*VFS) Rename 

func (vfs *VFS) Rename(oldName, newName string) error

Rename oldName to newName

func (*VFS) Root 

func (vfs *VFS) Root() (*Dir, error)

Root returns the root node

func (*VFS) SetCacheMode 

func (vfs *VFS) SetCacheMode(cacheMode vfscommon.CacheMode)

SetCacheMode change the cache mode

func (*VFS) Shutdown 

func (vfs *VFS) Shutdown()

Shutdown stops any background go-routines and removes the VFS from the active ache.

func (*VFS) Stat 

func (vfs *VFS) Stat(path string) (node Node, err error)

Stat finds the Node by path starting from the root

It is the equivalent of os.Stat - Node contains the os.FileInfo interface.

func (*VFS) StatParent 

func (vfs *VFS) StatParent(name string) (dir *Dir, leaf string, err error)

StatParent finds the parent directory and the leaf name of a path

func (*VFS) Statfs 

func (vfs *VFS) Statfs() (total, used, free int64)

Statfs returns into about the filing system if known

The values will be -1 if they aren't known

This information is cached for the DirCacheTime interval

func (*VFS) Stats added in v1.57.1 

func (vfs *VFS) Stats() (out rc.Params)

Stats returns info about the VFS

func (*VFS) WaitForWriters 

func (vfs *VFS) WaitForWriters(timeout time.Duration)

WaitForWriters sleeps until all writers have finished or time.Duration has elapsed

type WriteFileHandle 

type WriteFileHandle struct {
	// contains filtered or unexported fields
}

WriteFileHandle is an open for write handle on a File

func (WriteFileHandle) Chdir 

func (h WriteFileHandle) Chdir() error

func (WriteFileHandle) Chmod 

func (h WriteFileHandle) Chmod(mode os.FileMode) error

func (WriteFileHandle) Chown 

func (h WriteFileHandle) Chown(uid, gid int) error

func (*WriteFileHandle) Close 

func (fh *WriteFileHandle) Close() error

Close closes the file

func (WriteFileHandle) Fd 

func (h WriteFileHandle) Fd() uintptr

func (*WriteFileHandle) Flush 

func (fh *WriteFileHandle) Flush() error

Flush is called on each close() of a file descriptor. So if a filesystem wants to return write errors in close() and the file has cached dirty data, this is a good place to write back data and return any errors. Since many applications ignore close() errors this is not always useful.

NOTE: The flush() method may be called more than once for each open(). This happens if more than one file descriptor refers to an opened file due to dup(), dup2() or fork() calls. It is not possible to determine if a flush is final, so each flush should be treated equally. Multiple write-flush sequences are relatively rare, so this shouldn't be a problem.

Filesystems shouldn't assume that flush will always be called after some writes, or that if will be called at all.

func (WriteFileHandle) Name 

func (h WriteFileHandle) Name() string

func (*WriteFileHandle) Node 

func (fh *WriteFileHandle) Node() Node

Node returns the Node associated with this - satisfies Noder interface

func (*WriteFileHandle) Offset 

func (fh *WriteFileHandle) Offset() (offset int64)

Offset returns the offset of the file pointer

func (*WriteFileHandle) Read 

func (fh *WriteFileHandle) Read(p []byte) (n int, err error)

Read reads up to len(p) bytes into p.

func (*WriteFileHandle) ReadAt 

func (fh *WriteFileHandle) ReadAt(p []byte, off int64) (n int, err error)

ReadAt reads len(p) bytes into p starting at offset off in the underlying input source. It returns the number of bytes read (0 <= n <= len(p)) and any error encountered.

func (WriteFileHandle) Readdir 

func (h WriteFileHandle) Readdir(n int) ([]os.FileInfo, error)

func (WriteFileHandle) Readdirnames 

func (h WriteFileHandle) Readdirnames(n int) (names []string, err error)

func (*WriteFileHandle) Release 

func (fh *WriteFileHandle) Release() error

Release is called when we are finished with the file handle

It isn't called directly from userspace so the error is ignored by the kernel

func (WriteFileHandle) Seek 

func (h WriteFileHandle) Seek(offset int64, whence int) (ret int64, err error)

func (*WriteFileHandle) Stat 

func (fh *WriteFileHandle) Stat() (os.FileInfo, error)

Stat returns info about the file

func (*WriteFileHandle) String 

func (fh *WriteFileHandle) String() string

String converts it to printable

func (*WriteFileHandle) Sync 

func (fh *WriteFileHandle) Sync() error

Sync commits the current contents of the file to stable storage. Typically, this means flushing the file system's in-memory copy of recently written data to disk.

func (*WriteFileHandle) Truncate 

func (fh *WriteFileHandle) Truncate(size int64) (err error)

Truncate file to given size

func (*WriteFileHandle) Write 

func (fh *WriteFileHandle) Write(p []byte) (n int, err error)

Write writes len(p) bytes from p to the underlying data stream. It returns the number of bytes written from p (0 <= n <= len(p)) and any error encountered that caused the write to stop early. Write must return a non-nil error if it returns n < len(p). Write must not modify the slice data, even temporarily.

Implementations must not retain p.

func (*WriteFileHandle) WriteAt 

func (fh *WriteFileHandle) WriteAt(p []byte, off int64) (n int, err error)

WriteAt writes len(p) bytes from p to the underlying data stream at offset off. It returns the number of bytes written from p (0 <= n <= len(p)) and any error encountered that caused the write to stop early. WriteAt must return a non-nil error if it returns n < len(p).

If WriteAt is writing to a destination with a seek offset, WriteAt should not affect nor be affected by the underlying seek offset.

Clients of WriteAt can execute parallel WriteAt calls on the same destination if the ranges do not overlap.

Implementations must not retain p.

func (*WriteFileHandle) WriteString 

func (fh *WriteFileHandle) WriteString(s string) (n int, err error)

WriteString a string to the file

Source Files

View all Source files

Directories

Path Synopsis
Test the VFS to exhaustion, specifically looking for deadlocks
 Test the VFS to exhaustion, specifically looking for deadlocks
Package vfscache deals with caching of files locally for the VFS layer
 Package vfscache deals with caching of files locally for the VFS layer
downloaders
writeback
Package writeback keeps track of the files which need to be written back to storage
 Package writeback keeps track of the files which need to be written back to storage
Package vfsflags implements command line flags to set up a vfs
 Package vfsflags implements command line flags to set up a vfs

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.