fs

package
v0.0.0-...-1b1593a Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 13, 2017 License: MIT Imports: 52 Imported by: 0

Documentation

Overview

Package fs is a generic file system interface for rclone object storage systems

Index

Constants

View Source 
const (

	// ConfigToken is the key used to store the token under
	ConfigToken = "token"

	// ConfigClientID is the config key used to store the client id
	ConfigClientID = "client_id"

	// ConfigClientSecret is the config key used to store the client secret
	ConfigClientSecret = "client_secret"

	// ConfigAuthURL is the config key used to store the auth server endpoint
	ConfigAuthURL = "auth_url"

	// ConfigTokenURL is the config key used to store the token server endpoint
	ConfigTokenURL = "token_url"

	// ConfigAutomatic indicates that we want non-interactive configuration
	ConfigAutomatic = "config_automatic"
)
View Source 
const (
	// ModTimeNotSupported is a very large precision value to show
	// mod time isn't supported on this Fs
	ModTimeNotSupported = 100 * 365 * 24 * time.Hour
	// MaxLevel is a sentinel representing an infinite depth for listings
	MaxLevel = math.MaxInt32
)

Constants

Variables

View Source 
var (

	// ConfigPath points to the config file
	ConfigPath = makeConfigPath()

	// CacheDir points to the cache directory.  Users of this
	// should make a subdirectory and use MkdirAll() to create it
	// and any parents.
	CacheDir = makeCacheDir()

	// Config is the global config
	Config = &ConfigInfo{}

	AskPassword = BoolP("ask-password", "", true, "Allow prompt for password for encrypted configuration.")
)

Global

View Source 
var (

	// ErrorNotFoundInConfigFile is returned by NewFs if not found in config file
	ErrorNotFoundInConfigFile        = errors.New("didn't find section in config file")
	ErrorCantPurge                   = errors.New("can't purge directory")
	ErrorCantCopy                    = errors.New("can't copy object - incompatible remotes")
	ErrorCantMove                    = errors.New("can't move object - incompatible remotes")
	ErrorCantDirMove                 = errors.New("can't move directory - incompatible remotes")
	ErrorDirExists                   = errors.New("can't copy directory - destination already exists")
	ErrorCantSetModTime              = errors.New("can't set modified time")
	ErrorCantSetModTimeWithoutDelete = errors.New("can't set modified time without deleting existing object")
	ErrorDirNotFound                 = errors.New("directory not found")
	ErrorObjectNotFound              = errors.New("object not found")
	ErrorLevelNotSupported           = errors.New("level value not supported")
	ErrorListAborted                 = errors.New("list aborted")
	ErrorListBucketRequired          = errors.New("bucket or container name is needed in remote")
	ErrorIsFile                      = errors.New("is a file not a directory")
	ErrorNotAFile                    = errors.New("is a not a regular file")
	ErrorNotDeleting                 = errors.New("not deleting files as there were IO errors")
	ErrorNotDeletingDirs             = errors.New("not deleting directories as there were IO errors")
	ErrorCantMoveOverlapping         = errors.New("can't move files on overlapping remotes")
	ErrorDirectoryNotEmpty           = errors.New("directory not empty")
	ErrorImmutableModified           = errors.New("immutable file modified")
)

Globals

View Source 
var ErrHashUnsupported = errors.New("hash type not supported")

ErrHashUnsupported should be returned by filesystem, if it is requested to deliver an unsupported hash type.

View Source 
var ErrorCantListR = errors.New("recursive directory listing not available")

ErrorCantListR is returned by WalkR if the underlying Fs isn't capable of doing a recursive listing.

View Source 
var ErrorSkipDir = errors.New("skip this directory")

ErrorSkipDir is used as a return value from Walk to indicate that the directory named in the call is to be skipped. It is not returned as an error by any function.

View Source 
var HashWidth = map[HashType]int{
	HashMD5:     32,
	HashSHA1:    40,
	HashDropbox: 64,
}

HashWidth returns the width in characters for any HashType

View Source 
var MemoryFs memoryFs

MemoryFs is an in memory Fs, it only supports FsInfo and Put

View Source 
var ReadLine = func() string {
	buf := bufio.NewReader(os.Stdin)
	line, err := buf.ReadString('\n')
	if err != nil {
		log.Fatalf("Failed to read line: %v", err)
	}
	return strings.TrimSpace(line)
}

ReadLine reads some input

View Source 
var (
	Stats = NewStats()
)

Globals

View Source 
var SupportedHashes = NewHashSet(HashMD5, HashSHA1, HashDropbox)

SupportedHashes returns a set of all the supported hashes by HashStream and MultiHasher.

View Source 
var Version = "v1.38-DEV"

Version of rclone

Functions

func AccountPart 

func AccountPart(obj Object, in io.Reader) io.Reader

AccountPart accounts for part of a transfer

It disables the whole file counter and returns an io.Reader to wrap a segment of the transfer.

func Authorize 

func Authorize(args []string)

Authorize is for remote authorization of headless machines.

It expects 1 or 3 arguments 

rclone authorize "fs name"
rclone authorize "fs name" "client id" "client secret"

func BoolP 

func BoolP(name, shorthand string, value bool, usage string) (out *bool)

BoolP defines a flag which can be overridden by an environment variable

It is a thin wrapper around pflag.BoolP

func BoolVarP 

func BoolVarP(flags *pflag.FlagSet, p *bool, name, shorthand string, value bool, usage string)

BoolVarP defines a flag which can be overridden by an environment variable

It is a thin wrapper around pflag.BoolVarP

func CalculateModifyWindow 

func CalculateModifyWindow(fs ...Fs)

CalculateModifyWindow works out modify window for Fses passed in - sets Config.ModifyWindow

This is the largest modify window of all the fses in use, and the user configured value

func CanServerSideMove 

func CanServerSideMove(fdst Fs) bool

CanServerSideMove returns true if fdst support server side moves or server side copies

Some remotes simulate rename by server-side copy and delete, so include remotes that implements either Mover or Copier.

func Cat 

func Cat(f Fs, w io.Writer, offset, count int64) error

Cat any files to the io.Writer

if offset == 0 it will be ignored if offset > 0 then the file will be seeked to that offset if offset < 0 then the file will be seeked that far from the end

if count < 0 then it will be ignored if count >= 0 then only that many characters will be output

func Cause 

func Cause(cause error) (retriable bool, err error)

Cause is a souped up errors.Cause which can unwrap some standard library errors too. It returns true if any of the intermediate errors had a Timeout() or Temporary() method which returned true.

func ChangePassword 

func ChangePassword(name string) string

ChangePassword will query the user twice for the named password. If the same password is entered it is returned.

func Check 

func Check(fdst, fsrc Fs) error

Check the files in fsrc and fdst according to Size and hash

func CheckClose 

func CheckClose(c io.Closer, err *error)

CheckClose is a utility function used to check the return from Close in a defer statement.

func CheckDownload 

func CheckDownload(fdst, fsrc Fs) error

CheckDownload checks the files in fsrc and fdst according to Size and the actual contents of the files.

func CheckEqualReaders 

func CheckEqualReaders(in1, in2 io.Reader) (differ bool, err error)

CheckEqualReaders checks to see if in1 and in2 have the same content when read.

it returns true if differences were found

func CheckFn 

func CheckFn(fdst, fsrc Fs, check checkFn) error

CheckFn checks the files in fsrc and fdst according to Size and hash using checkFunction on each file to check the hashes.

checkFunction sees if dst and src are identical

it returns true if differences were found it also returns whether it couldn't be hashed

func CheckIdentical 

func CheckIdentical(dst, src Object) (differ bool, err error)

CheckIdentical checks to see if dst and src are identical by reading all their bytes if necessary.

it returns true if differences were found

func Choose 

func Choose(what string, defaults, help []string, newOk bool) string

Choose one of the defaults or type a new string if newOk is set

func ChooseNumber 

func ChooseNumber(what string, min, max int) int

ChooseNumber asks the user to enter a number between min and max inclusive prompting them with what.

func ChooseOption 

func ChooseOption(o *Option) string

ChooseOption asks the user to choose an option

func ChooseRemote 

func ChooseRemote() string

ChooseRemote chooses a remote name

func CleanUp 

func CleanUp(f Fs) error

CleanUp removes the trash for the Fs

func Command 

func Command(commands []string) byte

Command - choose one

func ConfigDump 

func ConfigDump() error

ConfigDump dumps all the config as a JSON file

func ConfigFileDeleteKey 

func ConfigFileDeleteKey(section, key string) bool

ConfigFileDeleteKey deletes the config key in the config file. It returns true if the key was deleted, or returns false if the section or key didn't exist.

func ConfigFileGet 

func ConfigFileGet(section, key string, defaultVal ...string) string

ConfigFileGet gets the config key under section returning the default or empty string if not set.

It looks up defaults in the environment if they are present

func ConfigFileGetBool 

func ConfigFileGetBool(section, key string, defaultVal ...bool) bool

ConfigFileGetBool gets the config key under section returning the default or false if not set.

It looks up defaults in the environment if they are present

func ConfigFileGetInt 

func ConfigFileGetInt(section, key string, defaultVal ...int) int

ConfigFileGetInt gets the config key under section returning the default or 0 if not set.

It looks up defaults in the environment if they are present

func ConfigFileSections 

func ConfigFileSections() []string

ConfigFileSections returns the sections in the config file including any defined by environment variables.

func ConfigFileSet 

func ConfigFileSet(section, key, value string)

ConfigFileSet sets the key in section to value. It doesn't save the config file.

func ConfigMaxDepth 

func ConfigMaxDepth(recursive bool) int

ConfigMaxDepth returns the depth to use for a recursive or non recursive listing.

func ConfigSetValueAndSave 

func ConfigSetValueAndSave(name, key, value string) (err error)

ConfigSetValueAndSave sets the key to the value and saves just that value in the config file. It loads the old config file in from disk first and overwrites the given value only.

func Confirm 

func Confirm() bool

Confirm asks the user for Yes or No and returns true or false

func CopyDir 

func CopyDir(fdst, fsrc Fs) error

CopyDir copies fsrc into fdst

func CopyFile 

func CopyFile(fdst Fs, fsrc Fs, dstFileName string, srcFileName string) (err error)

CopyFile moves a single file possibly to a new name

func CopyRemote 

func CopyRemote(name string)

CopyRemote copies a config section

func Count 

func Count(f Fs) (objects int64, size int64, err error)

Count counts the objects and their sizes in the Fs

Obeys includes and excludes

func CountP 

func CountP(name, shorthand string, usage string) (out *int)

CountP defines a flag which can be overridden by an environment variable

It is a thin wrapper around pflag.CountP

func CountVarP 

func CountVarP(flags *pflag.FlagSet, p *int, name, shorthand string, usage string)

CountVarP defines a flag which can be overridden by an environment variable

It is a thin wrapper around pflag.CountVarP

func CreateRemote 

func CreateRemote(name string, provider string, keyValues []string) error

CreateRemote creates a new remote with name, provider and a list of parameters which are key, value pairs. If update is set then it adds the new keys rather than replacing all of them.

func Debugf 

func Debugf(o interface{}, text string, args ...interface{})

Debugf writes debugging output for this Object or Fs. Use this for debug only. The user must have to specify -vv to see this.

func Deduplicate 

func Deduplicate(f Fs, mode DeduplicateMode) error

Deduplicate interactively finds duplicate files and offers to delete all but one or rename them to be different. Only useful with Google Drive which can have duplicate file names.

func Delete 

func Delete(f Fs) error

Delete removes all the contents of a container. Unlike Purge, it obeys includes and excludes.

func DeleteFile 

func DeleteFile(dst Object) (err error)

DeleteFile deletes a single file respecting --dry-run and accumulating stats and errors.

If useBackupDir is set and --backup-dir is in effect then it moves the file to there instead of deleting

func DeleteFiles 

func DeleteFiles(toBeDeleted ObjectsChan) error

DeleteFiles removes all the files passed in the channel

func DeleteRemote 

func DeleteRemote(name string)

DeleteRemote gets the user to delete a remote

func DirEntryType 

func DirEntryType(d DirEntry) string

DirEntryType returns a string description of the DirEntry, either "object", "directory" or "unknown type XXX"

func DropboxHashSum 

func DropboxHashSum(f Fs, w io.Writer) error

DropboxHashSum list the Fs to the supplied writer

Obeys includes and excludes

Lists in parallel which may get them out of order

func DurationP 

func DurationP(name, shorthand string, value time.Duration, usage string) (out *time.Duration)

DurationP defines a flag which can be overridden by an environment variable

It is a thin wrapper around pflag.DurationP

func DurationVarP 

func DurationVarP(flags *pflag.FlagSet, p *time.Duration, name, shorthand string, value time.Duration, usage string)

DurationVarP defines a flag which can be overridden by an environment variable

It is a thin wrapper around pflag.DurationVarP

func EditConfig 

func EditConfig()

EditConfig edits the config file interactively

func EditRemote 

func EditRemote(fs *RegInfo, name string)

EditRemote gets the user to edit a remote

func Equal 

func Equal(src ObjectInfo, dst Object) bool

Equal checks to see if the src and dst objects are equal by looking at size, mtime and hash

If the src and dst size are different then it is considered to be not equal. If --size-only is in effect then this is the only check that is done. If --ignore-size is in effect then this check is skipped and the files are considered the same size.

If the size is the same and the mtime is the same then it is considered to be equal. This check is skipped if using --checksum.

If the size is the same and mtime is different, unreadable or --checksum is set and the hash is the same then the file is considered to be equal. In this case the mtime on the dst is updated if --checksum is not set.

Otherwise the file is considered to be not equal including if there were errors reading info.

func Errorf 

func Errorf(o interface{}, text string, args ...interface{})

Errorf writes error log output for this Object or Fs. It should always be seen by the user.

func FatalError 

func FatalError(err error) error

FatalError makes an error which indicates it is a fatal error and the sync should stop.

func FileExists 

func FileExists(fs Fs, remote string) (bool, error)

FileExists returns true if a file remote exists. If remote is a directory, FileExists returns false.

func FlagsVarP 

func FlagsVarP(flags *pflag.FlagSet, value pflag.Value, name, shorthand, usage string)

FlagsVarP defines a flag which can be overridden by an environment variable

It is a thin wrapper around pflag.VarP

func Float64P 

func Float64P(name, shorthand string, value float64, usage string) (out *float64)

Float64P defines a flag which can be overridden by an environment variable

It is a thin wrapper around pflag.Float64P

func Float64VarP 

func Float64VarP(flags *pflag.FlagSet, p *float64, name, shorthand string, value float64, usage string)

Float64VarP defines a flag which can be overridden by an environment variable

It is a thin wrapper around pflag.Float64VarP

func GetPassword 

func GetPassword(prompt string) string

GetPassword asks the user for a password with the prompt given.

func HashEquals 

func HashEquals(src, dst string) bool

HashEquals checks to see if src == dst, but ignores empty strings and returns true if either is empty.

func HashStream 

func HashStream(r io.Reader) (map[HashType]string, error)

HashStream will calculate hashes of all supported hash types.

func HashStreamTypes 

func HashStreamTypes(r io.Reader, set HashSet) (map[HashType]string, error)

HashStreamTypes will calculate hashes of the requested hash types.

func Infof 

func Infof(o interface{}, text string, args ...interface{})

Infof writes info on transfers for this Object or Fs. Use this level for logging transfers, deletions and things which should appear with the -v flag.

func InitLogging 

func InitLogging()

InitLogging start the logging as per the command line flags

func IntP 

func IntP(name, shorthand string, value int, usage string) (out *int)

IntP defines a flag which can be overridden by an environment variable

It is a thin wrapper around pflag.IntP

func IntVarP 

func IntVarP(flags *pflag.FlagSet, p *int, name, shorthand string, value int, usage string)

IntVarP defines a flag which can be overridden by an environment variable

It is a thin wrapper around pflag.IntVarP

func IsFatalError 

func IsFatalError(err error) bool

IsFatalError returns true if err conforms to the Fatal interface and calling the Fatal method returns true.

func IsNoRetryError 

func IsNoRetryError(err error) bool

IsNoRetryError returns true if err conforms to the NoRetry interface and calling the NoRetry method returns true.

func IsRetryError 

func IsRetryError(err error) bool

IsRetryError returns true if err conforms to the Retry interface and calling the Retry method returns true.

func JSONListProviders 

func JSONListProviders() error

JSONListProviders prints all the providers and options in JSON format

func List 

func List(f Fs, w io.Writer) error

List the Fs to the supplied writer

Shows size and path - obeys includes and excludes

Lists in parallel which may get them out of order

func ListDir 

func ListDir(f Fs, w io.Writer) error

ListDir lists the directories/buckets/containers in the Fs to the supplied writer

func ListFn 

func ListFn(f Fs, fn func(Object)) error

ListFn lists the Fs to the supplied function

Lists in parallel which may get them out of order

func ListLong 

func ListLong(f Fs, w io.Writer) error

ListLong lists the Fs to the supplied writer

Shows size, mod time and path - obeys includes and excludes

Lists in parallel which may get them out of order

func LoadConfig 

func LoadConfig()

LoadConfig loads the config file

func LogLevelPrintf 

func LogLevelPrintf(level LogLevel, o interface{}, text string, args ...interface{})

LogLevelPrintf writes logs at the given level

func Logf 

func Logf(o interface{}, text string, args ...interface{})

Logf writes log output for this Object or Fs. This should be considered to be Info level logging. It is the default level. By default rclone should not log very much so only use this for important things the user should see. The user can filter these out with the -q flag.

func Md5sum 

func Md5sum(f Fs, w io.Writer) error

Md5sum list the Fs to the supplied writer

Produces the same output as the md5sum command - obeys includes and excludes

Lists in parallel which may get them out of order

func MimeType 

func MimeType(o ObjectInfo) (mimeType string)

MimeType returns the MimeType from the object, either by calling the MimeTyper interface or using MimeTypeFromName

func MimeTypeFromName 

func MimeTypeFromName(remote string) (mimeType string)

MimeTypeFromName returns a guess at the mime type from the name

func Mkdir 

func Mkdir(f Fs, dir string) error

Mkdir makes a destination directory or container

func MoveDir 

func MoveDir(fdst, fsrc Fs, deleteEmptySrcDirs bool) error

MoveDir moves fsrc into fdst

func MoveFile 

func MoveFile(fdst Fs, fsrc Fs, dstFileName string, srcFileName string) (err error)

MoveFile moves a single file possibly to a new name

func MustObscure 

func MustObscure(x string) string

MustObscure obscures a value, exiting with a fatal error if it failed

func MustReveal 

func MustReveal(x string) string

MustReveal reveals an obscured value, exiting with a fatal error if it failed

func NeedTransfer 

func NeedTransfer(dst, src Object) bool

NeedTransfer checks to see if src needs to be copied to dst using the current config.

Returns a flag which indicates whether the file needs to be transferred or not.

func NewRemote 

func NewRemote(name string)

NewRemote make a new remote from its name

func NewRemoteName 

func NewRemoteName() (name string)

NewRemoteName asks the user for a name for a remote

func NoRetryError 

func NoRetryError(err error) error

NoRetryError makes an error which indicates the sync shouldn't be retried.

func Obscure 

func Obscure(x string) (string, error)

Obscure a value

This is done by encrypting with AES-CTR

func OkRemote 

func OkRemote(name string) bool

OkRemote prints the contents of the remote and ask if it is OK

func OpenOptionAddHTTPHeaders 

func OpenOptionAddHTTPHeaders(headers http.Header, options []OpenOption)

OpenOptionAddHTTPHeaders Sets each header found in options to the http.Header map provided the key was non empty.

func OpenOptionAddHeaders 

func OpenOptionAddHeaders(options []OpenOption, headers map[string]string)

OpenOptionAddHeaders adds each header found in options to the headers map provided the key was non empty.

func OpenOptionHeaders 

func OpenOptionHeaders(options []OpenOption) (headers map[string]string)

OpenOptionHeaders adds each header found in options to the headers map provided the key was non empty.

It returns a nil map if options was empty

func Overlapping 

func Overlapping(fdst, fsrc Info) bool

Overlapping returns true if fdst and fsrc point to the same underlying Fs and they overlap.

func ParseDuration 

func ParseDuration(age string) (time.Duration, error)

ParseDuration parses a duration string. Accept ms|s|m|h|d|w|M|y suffixes. Defaults to second if not provided

func PasswordRemote 

func PasswordRemote(name string, keyValues []string) error

PasswordRemote adds the keyValues passed in to the remote of name. keyValues should be key, value pairs.

func Purge 

func Purge(f Fs) error

Purge removes a container and all of its contents

func ReadFill 

func ReadFill(r io.Reader, buf []byte) (n int, err error)

ReadFill reads as much data from r into buf as it can

It reads until the buffer is full or r.Read returned an error.

This is io.ReadFull but when you just want as much data as possible, not an exact size of block.

func ReadPassword 

func ReadPassword() string

ReadPassword reads a password without echoing it to the terminal.

func Register 

func Register(info *RegInfo)

Register a filesystem

Fs modules should use this in an init() function

func RemoteConfig 

func RemoteConfig(name string)

RemoteConfig runs the config helper for the remote if needed

func RemoteSplit 

func RemoteSplit(remote string) (parent string, leaf string)

RemoteSplit splits a remote into a parent and a leaf

if it returns leaf as an empty string then remote is a directory

if it returns parent as an empty string then that means the current directory

The returned values have the property that parent + leaf == remote

func RenameRemote 

func RenameRemote(name string)

RenameRemote renames a config section

func RetryError 

func RetryError(err error) error

RetryError makes an error which indicates it would like to be retried

func RetryErrorf 

func RetryErrorf(format string, a ...interface{}) error

RetryErrorf makes an error which indicates it would like to be retried

func Reveal 

func Reveal(x string) (string, error)

Reveal an obscured value

func Rmdir 

func Rmdir(f Fs, dir string) error

Rmdir removes a container but not if not empty

func Rmdirs 

func Rmdirs(f Fs, dir string, leaveRoot bool) error

Rmdirs removes any empty directories (or directories only containing empty directories) under f, including f.

func Same 

func Same(fdst, fsrc Info) bool

Same returns true if fdst and fsrc point to the same underlying Fs

func SameConfig 

func SameConfig(fdst, fsrc Info) bool

SameConfig returns true if fdst and fsrc are using the same config file entry

func SaveConfig 

func SaveConfig()

SaveConfig saves configuration file. if configKey has been set, the file will be encrypted.

func SetPassword 

func SetPassword()

SetPassword will allow the user to modify the current configuration encryption settings.

func Sha1sum 

func Sha1sum(f Fs, w io.Writer) error

Sha1sum list the Fs to the supplied writer

Obeys includes and excludes

Lists in parallel which may get them out of order

func ShouldRetry 

func ShouldRetry(err error) bool

ShouldRetry looks at an error and tries to work out if retrying the operation that caused it would be a good idea. It returns true if the error implements Timeout() or Temporary() or if the error indicates a premature closing of the connection.

func ShouldRetryHTTP 

func ShouldRetryHTTP(resp *http.Response, retryErrorCodes []int) bool

ShouldRetryHTTP returns a boolean as to whether this resp deserves. It checks to see if the HTTP response code is in the slice retryErrorCodes.

func ShowConfig 

func ShowConfig()

ShowConfig prints the (unencrypted) config options

func ShowConfigLocation 

func ShowConfigLocation()

ShowConfigLocation prints the location of the config file in use

func ShowRemote 

func ShowRemote(name string)

ShowRemote shows the contents of the remote

func ShowRemotes 

func ShowRemotes()

ShowRemotes shows an overview of the config file

func StringArrayP 

func StringArrayP(name, shorthand string, value []string, usage string) (out *[]string)

StringArrayP defines a flag which can be overridden by an environment variable

It sets one value only - command line flags can be used to set more.

It is a thin wrapper around pflag.StringArrayP

func StringArrayVarP 

func StringArrayVarP(flags *pflag.FlagSet, p *[]string, name, shorthand string, value []string, usage string)

StringArrayVarP defines a flag which can be overridden by an environment variable

It sets one value only - command line flags can be used to set more.

It is a thin wrapper around pflag.StringArrayVarP

func StringP 

func StringP(name, shorthand string, value string, usage string) (out *string)

StringP defines a flag which can be overridden by an environment variable

It is a thin wrapper around pflag.StringP

func StringVarP 

func StringVarP(flags *pflag.FlagSet, p *string, name, shorthand string, value string, usage string)

StringVarP defines a flag which can be overridden by an environment variable

It is a thin wrapper around pflag.StringVarP

func Sync 

func Sync(fdst, fsrc Fs) error

Sync fsrc into fdst

func Trace 

func Trace(o interface{}, format string, a ...interface{}) func(string, ...interface{})

Trace debugs the entry and exit of the calling function

It is designed to be used in a defer statement so it returns a function that logs the exit parameters.

Any pointers in the exit function will be dereferenced

func TryRmdir 

func TryRmdir(f Fs, dir string) error

TryRmdir removes a container but not if not empty. It doesn't count errors but may return one.

func Uint32VarP 

func Uint32VarP(flags *pflag.FlagSet, p *uint32, name, shorthand string, value uint32, usage string)

Uint32VarP defines a flag which can be overridden by an environment variable

It is a thin wrapper around pflag.Uint32VarP

func UpdateRemote 

func UpdateRemote(name string, keyValues []string) error

UpdateRemote adds the keyValues passed in to the remote of name. keyValues should be key, value pairs.

func VarP 

func VarP(value pflag.Value, name, shorthand, usage string)

VarP defines a flag which can be overridden by an environment variable

It is a thin wrapper around pflag.VarP

func Walk 

func Walk(f Fs, path string, includeAll bool, maxLevel int, fn WalkFunc) error

Walk lists the directory.

If includeAll is not set it will use the filters defined.

If maxLevel is < 0 then it will recurse indefinitely, else it will only do maxLevel levels.

It calls fn for each tranche of DirEntries read.

Note that fn will not be called concurrently whereas the directory listing will proceed concurrently.

Parent directories are always listed before their children

This is implemented by WalkR if Config.UseRecursiveListing is true and f supports it and level > 1, or WalkN otherwise.

NB (f, path) to be replaced by fs.Dir at some point

func WalkGetAll 

func WalkGetAll(f Fs, path string, includeAll bool, maxLevel int) (objs []Object, dirs []Directory, err error)

WalkGetAll runs Walk getting all the results

func WalkN 

func WalkN(f Fs, path string, includeAll bool, maxLevel int, fn WalkFunc) error

WalkN lists the directory.

It implements Walk using non recursive directory listing.

func WalkR 

func WalkR(f Fs, path string, includeAll bool, maxLevel int, fn WalkFunc) error

WalkR lists the directory.

It implements Walk using recursive directory listing if available, or returns ErrorCantListR if not.

Types

type Account 

type Account struct {
	// contains filtered or unexported fields
}

Account limits and accounts for one transfer

func AccountByPart 

func AccountByPart(obj Object) *Account

AccountByPart turns off whole file accounting

Returns the current account or nil if not found

func NewAccount 

func NewAccount(in io.ReadCloser, obj Object) *Account

NewAccount makes a Account reader for an object

func NewAccountSizeName 

func NewAccountSizeName(in io.ReadCloser, size int64, name string) *Account

NewAccountSizeName makes a Account reader for an io.ReadCloser of the given size and name

func (*Account) Close 

func (acc *Account) Close() error

Close the object

func (*Account) ETA 

func (acc *Account) ETA() (eta time.Duration, ok bool)

ETA returns the ETA of the current operation, rounded to full seconds. If the ETA cannot be determined 'ok' returns false.

func (*Account) GetReader 

func (acc *Account) GetReader() io.ReadCloser

GetReader returns the underlying io.ReadCloser

func (*Account) Progress 

func (acc *Account) Progress() (bytes, size int64)

Progress returns bytes read as well as the size. Size can be <= 0 if the size is unknown.

func (*Account) Read 

func (acc *Account) Read(p []byte) (n int, err error)

Read bytes from the object - see io.Reader

func (*Account) Speed 

func (acc *Account) Speed() (bps, current float64)

Speed returns the speed of the current file transfer in bytes per second, as well a an exponentially weighted moving average If no read has completed yet, 0 is returned for both values.

func (*Account) StopBuffering 

func (acc *Account) StopBuffering()

StopBuffering stops the async buffer doing any more buffering

func (*Account) String 

func (acc *Account) String() string

String produces stats for this file

func (*Account) UpdateReader 

func (acc *Account) UpdateReader(in io.ReadCloser)

UpdateReader updates the underlying io.ReadCloser

func (*Account) WithBuffer 

func (acc *Account) WithBuffer() *Account

WithBuffer - If the file is above a certain size it adds an Async reader

type BwTimeSlot 

type BwTimeSlot struct {
	// contains filtered or unexported fields
}

BwTimeSlot represents a bandwidth configuration at a point in time.

type BwTimetable 

type BwTimetable []BwTimeSlot

BwTimetable contains all configured time slots.

func (BwTimetable) LimitAt 

func (x BwTimetable) LimitAt(tt time.Time) BwTimeSlot

LimitAt returns a BwTimeSlot for the time requested.

func (*BwTimetable) Set 

func (x *BwTimetable) Set(s string) error

Set the bandwidth timetable.

func (BwTimetable) String 

func (x BwTimetable) String() string

String returns a printable representation of BwTimetable.

func (BwTimetable) Type 

func (x BwTimetable) Type() string

Type of the value

type CleanUpper 

type CleanUpper interface {
	// CleanUp the trash in the Fs
	//
	// Implement this if you have a way of emptying the trash or
	// otherwise cleaning up old versions of files.
	CleanUp() error
}

CleanUpper is an optional interfaces for Fs

type ConfigInfo 

type ConfigInfo struct {
	LogLevel              LogLevel
	StatsLogLevel         LogLevel
	DryRun                bool
	CheckSum              bool
	SizeOnly              bool
	IgnoreTimes           bool
	IgnoreExisting        bool
	ModifyWindow          time.Duration
	Checkers              int
	Transfers             int
	ConnectTimeout        time.Duration // Connect timeout
	Timeout               time.Duration // Data channel timeout
	Dump                  DumpFlags
	Filter                *Filter
	InsecureSkipVerify    bool // Skip server certificate verification
	DeleteMode            DeleteMode
	TrackRenames          bool // Track file renames.
	LowLevelRetries       int
	UpdateOlder           bool // Skip files that are newer on the destination
	NoGzip                bool // Disable compression
	MaxDepth              int
	IgnoreSize            bool
	IgnoreChecksum        bool
	NoTraverse            bool
	NoUpdateModTime       bool
	DataRateUnit          string
	BackupDir             string
	Suffix                string
	UseListR              bool
	BufferSize            SizeSuffix
	TPSLimit              float64
	TPSLimitBurst         int
	BindAddr              net.IP
	DisableFeatures       []string
	Immutable             bool
	AutoConfirm           bool
	StreamingUploadCutoff SizeSuffix
}

ConfigInfo is filesystem config options

func (*ConfigInfo) Client 

func (ci *ConfigInfo) Client() *http.Client

Client returns an http.Client with the correct timeouts

func (*ConfigInfo) NewDialer 

func (ci *ConfigInfo) NewDialer() *net.Dialer

NewDialer creates a net.Dialer structure with Timeout, Keepalive and LocalAddr set from rclone flags.

func (*ConfigInfo) Transport 

func (ci *ConfigInfo) Transport() http.RoundTripper

Transport returns an http.RoundTripper with the correct timeouts

type Copier 

type Copier interface {
	// Copy src to this remote using server side copy operations.
	//
	// This is stored with the remote path given
	//
	// It returns the destination Object and a possible error
	//
	// Will only be called if src.Fs().Name() == f.Name()
	//
	// If it isn't possible then return fs.ErrorCantCopy
	Copy(src Object, remote string) (Object, error)
}

Copier is an optional interface for Fs

type CountingReader 

type CountingReader struct {
	// contains filtered or unexported fields
}

CountingReader holds a reader and a read count of how many bytes were read so far.

func NewCountingReader 

func NewCountingReader(in io.Reader) *CountingReader

NewCountingReader returns a CountingReader, which will read from the given reader while keeping track of how many bytes were read.

func (*CountingReader) BytesRead 

func (cr *CountingReader) BytesRead() uint64

BytesRead returns how many bytes were read from the underlying reader so far.

func (*CountingReader) Read 

func (cr *CountingReader) Read(b []byte) (int, error)

Read reads from the underlying reader.

type DeduplicateMode 

type DeduplicateMode int

DeduplicateMode is how the dedupe command chooses what to do 

const (
	DeduplicateInteractive DeduplicateMode = iota // interactively ask the user
	DeduplicateSkip                               // skip all conflicts
	DeduplicateFirst                              // choose the first object
	DeduplicateNewest                             // choose the newest object
	DeduplicateOldest                             // choose the oldest object
	DeduplicateRename                             // rename the objects
)

Deduplicate modes

func (*DeduplicateMode) Set 

func (x *DeduplicateMode) Set(s string) error

Set a DeduplicateMode from a string

func (DeduplicateMode) String 

func (x DeduplicateMode) String() string

func (*DeduplicateMode) Type 

func (x *DeduplicateMode) Type() string

Type of the value

type DeleteMode 

type DeleteMode byte

DeleteMode describes the possible delete modes in the config 

const (
	DeleteModeOff DeleteMode = iota
	DeleteModeBefore
	DeleteModeDuring
	DeleteModeAfter
	DeleteModeOnly
	DeleteModeDefault = DeleteModeAfter
)

DeleteMode constants

type Dir 

type Dir struct {
	// contains filtered or unexported fields
}

Dir describes an unspecialized directory for directory/container/bucket lists

func NewDir 

func NewDir(remote string, modTime time.Time) *Dir

NewDir creates an unspecialized Directory object

func NewDirCopy 

func NewDirCopy(d Directory) *Dir

NewDirCopy creates an unspecialized copy of the Directory object passed in

func (*Dir) ID 

func (d *Dir) ID() string

ID gets the optional ID

func (*Dir) Items 

func (d *Dir) Items() int64

Items returns the count of items in this directory or this directory and subdirectories if known, -1 for unknown

func (*Dir) ModTime 

func (d *Dir) ModTime() time.Time

ModTime returns the modification date of the file It should return a best guess if one isn't available

func (*Dir) Remote 

func (d *Dir) Remote() string

Remote returns the remote path

func (*Dir) SetID 

func (d *Dir) SetID(id string) *Dir

SetID sets the optional ID

func (*Dir) SetItems 

func (d *Dir) SetItems(items int64) *Dir

SetItems sets the number of items in the directory

func (*Dir) SetRemote 

func (d *Dir) SetRemote(remote string) *Dir

SetRemote sets the remote

func (*Dir) SetSize 

func (d *Dir) SetSize(size int64) *Dir

SetSize sets the size of the directory

func (*Dir) Size 

func (d *Dir) Size() int64

Size returns the size of the file

func (*Dir) String 

func (d *Dir) String() string

String returns the name

type DirCacheFlusher 

type DirCacheFlusher interface {
	// DirCacheFlush resets the directory cache - used in testing
	// as an optional interface
	DirCacheFlush()
}

DirCacheFlusher is an optional interface for Fs

type DirChangeNotifier 

type DirChangeNotifier interface {
	// DirChangeNotify calls the passed function with a path
	// of a directory that has had changes. If the implementation
	// uses polling, it should adhere to the given interval.
	DirChangeNotify(func(string), time.Duration) chan bool
}

DirChangeNotifier is an optional interface for Fs

type DirEntries 

type DirEntries []DirEntry

DirEntries is a slice of Object or *Dir

func ListDirSorted 

func ListDirSorted(fs Fs, includeAll bool, dir string) (entries DirEntries, err error)

ListDirSorted reads Object and *Dir into entries for the given Fs.

dir is the start directory, "" for root

If includeAll is specified all files will be added, otherwise only files and directories passing the filter will be added.

Files will be returned in sorted order

func (DirEntries) ForDir 

func (ds DirEntries) ForDir(fn func(dir Directory))

ForDir runs the function supplied on every Directory in the entries

func (DirEntries) ForDirError 

func (ds DirEntries) ForDirError(fn func(dir Directory) error) error

ForDirError runs the function supplied on every Directory in the entries

func (DirEntries) ForObject 

func (ds DirEntries) ForObject(fn func(o Object))

ForObject runs the function supplied on every object in the entries

func (DirEntries) ForObjectError 

func (ds DirEntries) ForObjectError(fn func(o Object) error) error

ForObjectError runs the function supplied on every object in the entries

func (DirEntries) Len 

func (ds DirEntries) Len() int

Len is part of sort.Interface.

func (DirEntries) Less 

func (ds DirEntries) Less(i, j int) bool

Less is part of sort.Interface.

func (DirEntries) Swap 

func (ds DirEntries) Swap(i, j int)

Swap is part of sort.Interface.

type DirEntry 

type DirEntry interface {
	// String returns a description of the Object
	String() string

	// Remote returns the remote path
	Remote() string

	// ModTime returns the modification date of the file
	// It should return a best guess if one isn't available
	ModTime() time.Time

	// Size returns the size of the file
	Size() int64
}

DirEntry provides read only information about the common subset of a Dir or Object. These are returned from directory listings - type assert them into the correct type.

type DirMover 

type DirMover interface {
	// DirMove moves src, srcRemote to this remote at dstRemote
	// using server side move operations.
	//
	// Will only be called if src.Fs().Name() == f.Name()
	//
	// If it isn't possible then return fs.ErrorCantDirMove
	//
	// If destination exists then return fs.ErrorDirExists
	DirMove(src Fs, srcRemote, dstRemote string) error
}

DirMover is an optional interface for Fs

type DirTree 

type DirTree map[string]DirEntries

DirTree is a map of directories to entries

func NewDirTree 

func NewDirTree(f Fs, path string, includeAll bool, maxLevel int) (DirTree, error)

NewDirTree returns a DirTree filled with the directory listing using the parameters supplied.

If includeAll is not set it will use the filters defined.

If maxLevel is < 0 then it will recurse indefinitely, else it will only do maxLevel levels.

This is implemented by WalkR if Config.UseRecursiveListing is true and f supports it and level > 1, or WalkN otherwise.

NB (f, path) to be replaced by fs.Dir at some point

func (DirTree) Dirs 

func (dt DirTree) Dirs() (dirNames []string)

Dirs returns the directories in sorted order

func (DirTree) Find 

func (dt DirTree) Find(filePath string) (parentPath string, entry DirEntry)

Find returns the DirEntry for filePath or nil if not found

func (DirTree) Prune 

func (dt DirTree) Prune(dirNames map[string]bool) error

Prune remove directories from a directory tree. dirNames contains all directories to remove as keys, with true as values. dirNames will be modified in the function.

func (DirTree) Sort 

func (dt DirTree) Sort()

Sort sorts all the Entries

func (DirTree) String 

func (dt DirTree) String() string

String emits a simple representation of the DirTree

type Directory 

type Directory interface {
	DirEntry

	// Items returns the count of items in this directory or this
	// directory and subdirectories if known, -1 for unknown
	Items() int64

	// ID returns the internal ID of this directory if known, or
	// "" otherwise
	ID() string
}

Directory is a filesystem like directory provided by an Fs

type DumpFlags 

type DumpFlags int

DumpFlags describes the Dump options in force 

const (
	DumpHeaders DumpFlags = 1 << iota
	DumpBodies
	DumpRequests
	DumpResponses
	DumpAuth
	DumpFilters
)

DumpFlags definitions

func (*DumpFlags) Set 

func (f *DumpFlags) Set(s string) error

Set a DumpFlags as a comma separated list of flags

func (DumpFlags) String 

func (f DumpFlags) String() string

String turns a DumpFlags into a string

func (*DumpFlags) Type 

func (f *DumpFlags) Type() string

Type of the value

type Fataler 

type Fataler interface {
	error
	Fatal() bool
}

Fataler is an optional interface for error as to whether the operation should cause the entire operation to finish immediately.

This should be returned from Update or Put methods as required

type Features 

type Features struct {
	// Feature flags, whether Fs
	CaseInsensitive         bool // has case insensitive files
	DuplicateFiles          bool // allows duplicate files
	ReadMimeType            bool // can read the mime type of objects
	WriteMimeType           bool // can set the mime type of objects
	CanHaveEmptyDirectories bool // can have empty directories
	BucketBased             bool // is bucket based (like s3, swift etc)

	// Purge all files in the root and the root directory
	//
	// Implement this if you have a way of deleting all the files
	// quicker than just running Remove() on the result of List()
	//
	// Return an error if it doesn't exist
	Purge func() error

	// Copy src to this remote using server side copy operations.
	//
	// This is stored with the remote path given
	//
	// It returns the destination Object and a possible error
	//
	// Will only be called if src.Fs().Name() == f.Name()
	//
	// If it isn't possible then return fs.ErrorCantCopy
	Copy func(src Object, remote string) (Object, error)

	// Move src to this remote using server side move operations.
	//
	// This is stored with the remote path given
	//
	// It returns the destination Object and a possible error
	//
	// Will only be called if src.Fs().Name() == f.Name()
	//
	// If it isn't possible then return fs.ErrorCantMove
	Move func(src Object, remote string) (Object, error)

	// DirMove moves src, srcRemote to this remote at dstRemote
	// using server side move operations.
	//
	// Will only be called if src.Fs().Name() == f.Name()
	//
	// If it isn't possible then return fs.ErrorCantDirMove
	//
	// If destination exists then return fs.ErrorDirExists
	DirMove func(src Fs, srcRemote, dstRemote string) error

	// DirChangeNotify calls the passed function with a path
	// of a directory that has had changes. If the implementation
	// uses polling, it should adhere to the given interval.
	DirChangeNotify func(func(string), time.Duration) chan bool

	// UnWrap returns the Fs that this Fs is wrapping
	UnWrap func() Fs

	// WrapFs returns the Fs that is wrapping this Fs
	WrapFs func() Fs

	// SetWrapper sets the Fs that is wrapping this Fs
	SetWrapper func(f Fs)

	// DirCacheFlush resets the directory cache - used in testing
	// as an optional interface
	DirCacheFlush func()

	// Put in to the remote path with the modTime given of the given size
	//
	// May create the object even if it returns an error - if so
	// will return the object and the error, otherwise will return
	// nil and the error
	//
	// May create duplicates or return errors if src already
	// exists.
	PutUnchecked func(in io.Reader, src ObjectInfo, options ...OpenOption) (Object, error)

	// PutStream uploads to the remote path with the modTime given of indeterminate size
	//
	// May create the object even if it returns an error - if so
	// will return the object and the error, otherwise will return
	// nil and the error
	PutStream func(in io.Reader, src ObjectInfo, options ...OpenOption) (Object, error)

	// MergeDirs merges the contents of all the directories passed
	// in into the first one and rmdirs the other directories.
	MergeDirs func([]Directory) error

	// CleanUp the trash in the Fs
	//
	// Implement this if you have a way of emptying the trash or
	// otherwise cleaning up old versions of files.
	CleanUp func() error

	// ListR lists the objects and directories of the Fs starting
	// from dir recursively into out.
	//
	// dir should be "" to start from the root, and should not
	// have trailing slashes.
	//
	// This should return ErrDirNotFound if the directory isn't
	// found.
	//
	// It should call callback for each tranche of entries read.
	// These need not be returned in any particular order.  If
	// callback returns an error then the listing will stop
	// immediately.
	//
	// Don't implement this unless you have a more efficient way
	// of listing recursively that doing a directory traversal.
	ListR ListRFn
}

Features describe the optional features of the Fs

func (*Features) Disable 

func (ft *Features) Disable(name string) *Features

Disable nil's out the named feature. If it isn't found then it will log a message.

func (*Features) DisableList 

func (ft *Features) DisableList(list []string) *Features

DisableList nil's out the comma separated list of named features. If it isn't found then it will log a message.

func (*Features) Fill 

func (ft *Features) Fill(f Fs) *Features

Fill fills in the function pointers in the Features struct from the optional interfaces. It returns the original updated Features struct passed in.

func (*Features) List 

func (ft *Features) List() (out []string)

List returns a slice of all the possible feature names

func (*Features) Mask 

func (ft *Features) Mask(f Fs) *Features

Mask the Features with the Fs passed in

Only optional features which are implemented in both the original Fs AND the one passed in will be advertised. Any features which aren't in both will be set to false/nil, except for UnWrap/Wrap which will be left untouched.

func (*Features) Wrap 

func (ft *Features) Wrap(f Fs) *Features

Wrap makes a Copy of the features passed in, overriding the UnWrap/Wrap method only if available in f.

func (*Features) WrapsFs 

func (ft *Features) WrapsFs(f Fs, w Fs) *Features

WrapsFs adds extra information between `f` which wraps `w`

type FilesMap 

type FilesMap map[string]struct{}

FilesMap describes the map of files to transfer

type Filter 

type Filter struct {
	DeleteExcluded bool
	MinSize        int64
	MaxSize        int64
	ModTimeFrom    time.Time
	ModTimeTo      time.Time

	ExcludeFile string
	// contains filtered or unexported fields
}

Filter describes any filtering in operation

func NewFilter 

func NewFilter() (f *Filter, err error)

NewFilter parses the command line options and creates a Filter object

func (*Filter) Add 

func (f *Filter) Add(Include bool, glob string) error

Add adds a filter rule with include or exclude status indicated

func (*Filter) AddFile 

func (f *Filter) AddFile(file string) error

AddFile adds a single file to the files from list

func (*Filter) AddRule 

func (f *Filter) AddRule(rule string) error

AddRule adds a filter rule with include/exclude indicated by the prefix

These are

  • glob
  • glob !

'+' includes the glob, '-' excludes it and '!' resets the filter list

Line comments may be introduced with '#' or ';'

func (*Filter) Clear 

func (f *Filter) Clear()

Clear clears all the filter rules

func (*Filter) DirContainsExcludeFile 

func (f *Filter) DirContainsExcludeFile(fs Fs, remote string) (bool, error)

DirContainsExcludeFile checks if exclude file is present in a directroy. If fs is nil, it works properly if ExcludeFile is an empty string (for testing).

func (*Filter) DumpFilters 

func (f *Filter) DumpFilters() string

DumpFilters dumps the filters in textual form, 1 per line

func (*Filter) Files 

func (f *Filter) Files() FilesMap

Files returns all the files from the `--files-from` list

It may be nil if the list is empty

func (*Filter) InActive 

func (f *Filter) InActive() bool

InActive returns false if any filters are active

func (*Filter) Include 

func (f *Filter) Include(remote string, size int64, modTime time.Time) bool

Include returns whether this object should be included into the sync or not

func (*Filter) IncludeDirectory 

func (f *Filter) IncludeDirectory(fs Fs) func(string) (bool, error)

IncludeDirectory returns a function which checks whether this directory should be included in the sync or not.

func (*Filter) IncludeObject 

func (f *Filter) IncludeObject(o Object) bool

IncludeObject returns whether this object should be included into the sync or not. This is a convenience function to avoid calling o.ModTime(), which is an expensive operation.

func (*Filter) ListContainsExcludeFile 

func (f *Filter) ListContainsExcludeFile(entries DirEntries) bool

ListContainsExcludeFile checks if exclude file is present in the list.

type Fs 

type Fs interface {
	Info

	// List the objects and directories in dir into entries.  The
	// entries can be returned in any order but should be for a
	// complete directory.
	//
	// dir should be "" to list the root, and should not have
	// trailing slashes.
	//
	// This should return ErrDirNotFound if the directory isn't
	// found.
	List(dir string) (entries DirEntries, err error)

	// NewObject finds the Object at remote.  If it can't be found
	// it returns the error ErrorObjectNotFound.
	NewObject(remote string) (Object, error)

	// Put in to the remote path with the modTime given of the given size
	//
	// May create the object even if it returns an error - if so
	// will return the object and the error, otherwise will return
	// nil and the error
	Put(in io.Reader, src ObjectInfo, options ...OpenOption) (Object, error)

	// Mkdir makes the directory (container, bucket)
	//
	// Shouldn't return an error if it already exists
	Mkdir(dir string) error

	// Rmdir removes the directory (container, bucket) if empty
	//
	// Return an error if it doesn't exist or isn't empty
	Rmdir(dir string) error
}

Fs is the interface a cloud storage system must provide

func NewFs 

func NewFs(path string) (Fs, error)

NewFs makes a new Fs object from the path

The path is of the form remote:path

Remotes are looked up in the config file. If the remote isn't found then NotFoundInConfigFile will be returned.

On Windows avoid single character remote names as they can be mixed up with drive letters.

type HTTPOption 

type HTTPOption struct {
	Key   string
	Value string
}

HTTPOption defines a general purpose HTTP option

func (*HTTPOption) Header 

func (o *HTTPOption) Header() (key string, value string)

Header formats the option as an http header

func (*HTTPOption) Mandatory 

func (o *HTTPOption) Mandatory() bool

Mandatory returns whether the option must be parsed or can be ignored

func (*HTTPOption) String 

func (o *HTTPOption) String() string

String formats the option into human readable form

type HashSet 

type HashSet int

A HashSet Indicates one or more hash types.

func NewHashSet 

func NewHashSet(t ...HashType) HashSet

NewHashSet will create a new hash set with the hash types supplied

func (*HashSet) Add 

func (h *HashSet) Add(t ...HashType) HashSet

Add one or more hash types to the set. Returns the modified hash set.

func (HashSet) Array 

func (h HashSet) Array() (ht []HashType)

Array returns an array of all hash types in the set

func (HashSet) Contains 

func (h HashSet) Contains(t HashType) bool

Contains returns true if the

func (HashSet) Count 

func (h HashSet) Count() int

Count returns the number of hash types in the set

func (HashSet) GetOne 

func (h HashSet) GetOne() HashType

GetOne will return a hash type. Currently the first is returned, but it could be improved to return the strongest.

func (HashSet) Overlap 

func (h HashSet) Overlap(t HashSet) HashSet

Overlap returns the overlapping hash types

func (HashSet) String 

func (h HashSet) String() string

String returns a string representation of the hash set. The function will panic if it contains an unknown type.

func (HashSet) SubsetOf 

func (h HashSet) SubsetOf(c HashSet) bool

SubsetOf will return true if all types of h is present in the set c

type HashType 

type HashType int

HashType indicates a standard hashing algorithm 

const (
	// HashMD5 indicates MD5 support
	HashMD5 HashType = 1 << iota

	// HashSHA1 indicates SHA-1 support
	HashSHA1

	// HashDropbox indicates Dropbox special hash
	// https://www.dropbox.com/developers/reference/content-hash
	HashDropbox

	// HashNone indicates no hashes are supported
	HashNone HashType = 0
)

func CheckHashes 

func CheckHashes(src ObjectInfo, dst Object) (equal bool, hash HashType, err error)

CheckHashes checks the two files to see if they have common known hash types and compares them

Returns

equal - which is equality of the hashes

hash - the HashType. This is HashNone if either of the hashes were unset or a compatible hash couldn't be found.

err - may return an error which will already have been logged

If an error is returned it will return equal as false

func (HashType) String 

func (h HashType) String() string

String returns a string representation of the hash type. The function will panic if the hash type is unknown.

type HashesOption 

type HashesOption struct {
	Hashes HashSet
}

HashesOption defines an option used to tell the local fs to limit the number of hashes it calculates.

func (*HashesOption) Header 

func (o *HashesOption) Header() (key string, value string)

Header formats the option as an http header

func (*HashesOption) Mandatory 

func (o *HashesOption) Mandatory() bool

Mandatory returns whether the option must be parsed or can be ignored

func (*HashesOption) String 

func (o *HashesOption) String() string

String formats the option into human readable form

type Info 

type Info interface {
	// Name of the remote (as passed into NewFs)
	Name() string

	// Root of the remote (as passed into NewFs)
	Root() string

	// String returns a description of the FS
	String() string

	// Precision of the ModTimes in this Fs
	Precision() time.Duration

	// Returns the supported hash types of the filesystem
	Hashes() HashSet

	// Features returns the optional features of this Fs
	Features() *Features
}

Info provides a read only interface to information about a filesystem.

type ListRCallback 

type ListRCallback func(entries DirEntries) error

ListRCallback defines a callback function for ListR to use

It is called for each tranche of entries read from the listing and if it returns an error, the listing stops.

type ListRFn 

type ListRFn func(dir string, callback ListRCallback) error

ListRFn is defines the call used to recursively list a directory

type ListRHelper 

type ListRHelper struct {
	// contains filtered or unexported fields
}

ListRHelper is used in the implementation of ListR to accumulate DirEntries

func NewListRHelper 

func NewListRHelper(callback ListRCallback) *ListRHelper

NewListRHelper should be called from ListR with the callback passed in

func (*ListRHelper) Add 

func (lh *ListRHelper) Add(entry DirEntry) error

Add an entry to the stored entries and send them if there are more than a certain amount

func (*ListRHelper) Flush 

func (lh *ListRHelper) Flush() error

Flush the stored entries (if any) sending them to the callback

type ListRer 

type ListRer interface {
	// ListR lists the objects and directories of the Fs starting
	// from dir recursively into out.
	//
	// dir should be "" to start from the root, and should not
	// have trailing slashes.
	//
	// This should return ErrDirNotFound if the directory isn't
	// found.
	//
	// It should call callback for each tranche of entries read.
	// These need not be returned in any particular order.  If
	// callback returns an error then the listing will stop
	// immediately.
	//
	// Don't implement this unless you have a more efficient way
	// of listing recursively that doing a directory traversal.
	ListR(dir string, callback ListRCallback) error
}

ListRer is an optional interfaces for Fs

type LogLevel 

type LogLevel byte

LogLevel describes rclone's logs. These are a subset of the syslog log levels. 

const (
	LogLevelEmergency LogLevel = iota
	LogLevelAlert
	LogLevelCritical
	LogLevelError // Error - can't be suppressed
	LogLevelWarning
	LogLevelNotice // Normal logging, -q suppresses
	LogLevelInfo   // Transfers, needs -v
	LogLevelDebug  // Debug level, needs -vv
)

Log levels. These are the syslog levels of which we only use a subset. 

LOG_EMERG      system is unusable
LOG_ALERT      action must be taken immediately
LOG_CRIT       critical conditions
LOG_ERR        error conditions
LOG_WARNING    warning conditions
LOG_NOTICE     normal, but significant, condition
LOG_INFO       informational message
LOG_DEBUG      debug-level message

func (*LogLevel) Set 

func (l *LogLevel) Set(s string) error

Set a LogLevel

func (LogLevel) String 

func (l LogLevel) String() string

String turns a LogLevel into a string

func (*LogLevel) Type 

func (l *LogLevel) Type() string

Type of the value

type MemoryObject 

type MemoryObject struct {
	// contains filtered or unexported fields
}

MemoryObject is an in memory object

func NewMemoryObject 

func NewMemoryObject(remote string, modTime time.Time, content []byte) *MemoryObject

NewMemoryObject returns an in memory Object with the modTime and content passed in

func (*MemoryObject) Content 

func (o *MemoryObject) Content() []byte

Content returns the underlying buffer

func (*MemoryObject) Fs 

func (o *MemoryObject) Fs() Info

Fs returns read only access to the Fs that this object is part of

func (*MemoryObject) Hash 

func (o *MemoryObject) Hash(h HashType) (string, error)

Hash returns the requested hash of the contents

func (*MemoryObject) ModTime 

func (o *MemoryObject) ModTime() time.Time

ModTime returns the modification date of the file

func (*MemoryObject) Open 

func (o *MemoryObject) Open(options ...OpenOption) (io.ReadCloser, error)

Open opens the file for read. Call Close() on the returned io.ReadCloser

func (*MemoryObject) Remote 

func (o *MemoryObject) Remote() string

Remote returns the remote path

func (*MemoryObject) Remove 

func (o *MemoryObject) Remove() error

Remove this object

func (*MemoryObject) SetModTime 

func (o *MemoryObject) SetModTime(modTime time.Time) error

SetModTime sets the metadata on the object to set the modification date

func (*MemoryObject) Size 

func (o *MemoryObject) Size() int64

Size returns the size of the file

func (*MemoryObject) Storable 

func (o *MemoryObject) Storable() bool

Storable says whether this object can be stored

func (*MemoryObject) String 

func (o *MemoryObject) String() string

String returns a description of the Object

func (*MemoryObject) Update 

func (o *MemoryObject) Update(in io.Reader, src ObjectInfo, options ...OpenOption) (err error)

Update in to the object with the modTime given of the given size

This re-uses the internal buffer if at all possible.

type MergeDirser 

type MergeDirser interface {
	// MergeDirs merges the contents of all the directories passed
	// in into the first one and rmdirs the other directories.
	MergeDirs([]Directory) error
}

MergeDirser is an option interface for Fs

type MimeTyper 

type MimeTyper interface {
	// MimeType returns the content type of the Object if
	// known, or "" if not
	MimeType() string
}

MimeTyper is an optional interface for Object

type Mover 

type Mover interface {
	// Move src to this remote using server side move operations.
	//
	// This is stored with the remote path given
	//
	// It returns the destination Object and a possible error
	//
	// Will only be called if src.Fs().Name() == f.Name()
	//
	// If it isn't possible then return fs.ErrorCantMove
	Move(src Object, remote string) (Object, error)
}

Mover is an optional interface for Fs

type MultiHasher 

type MultiHasher struct {
	// contains filtered or unexported fields
}

A MultiHasher will construct various hashes on all incoming writes.

func NewMultiHasher 

func NewMultiHasher() *MultiHasher

NewMultiHasher will return a hash writer that will write all supported hash types.

func NewMultiHasherTypes 

func NewMultiHasherTypes(set HashSet) (*MultiHasher, error)

NewMultiHasherTypes will return a hash writer that will write the requested hash types.

func (*MultiHasher) Size 

func (m *MultiHasher) Size() int64

Size returns the number of bytes written

func (*MultiHasher) Sums 

func (m *MultiHasher) Sums() map[HashType]string

Sums returns the sums of all accumulated hashes as hex encoded strings.

func (*MultiHasher) Write 

func (m *MultiHasher) Write(p []byte) (n int, err error)

type NoRetrier 

type NoRetrier interface {
	error
	NoRetry() bool
}

NoRetrier is an optional interface for error as to whether the operation should not be retried at a high level.

If only NoRetry errors are returned in a sync then the sync won't be retried.

This should be returned from Update or Put methods as required

type Object 

type Object interface {
	ObjectInfo

	// SetModTime sets the metadata on the object to set the modification date
	SetModTime(time.Time) error

	// Open opens the file for read.  Call Close() on the returned io.ReadCloser
	Open(options ...OpenOption) (io.ReadCloser, error)

	// Update in to the object with the modTime given of the given size
	Update(in io.Reader, src ObjectInfo, options ...OpenOption) error

	// Removes this object
	Remove() error
}

Object is a filesystem like object provided by an Fs

func Copy 

func Copy(f Fs, dst Object, remote string, src Object) (newDst Object, err error)

Copy src object to dst or f if nil. If dst is nil then it uses remote as the name of the new object.

It returns the destination object if possible. Note that this may be nil.

func Move 

func Move(fdst Fs, dst Object, remote string, src Object) (newDst Object, err error)

Move src object to dst or fdst if nil. If dst is nil then it uses remote as the name of the new object.

It returns the destination object if possible. Note that this may be nil.

func Rcat 

func Rcat(fdst Fs, dstFileName string, in io.ReadCloser, modTime time.Time) (dst Object, err error)

Rcat reads data from the Reader until EOF and uploads it to a file on remote

type ObjectInfo 

type ObjectInfo interface {
	DirEntry

	// Fs returns read only access to the Fs that this object is part of
	Fs() Info

	// Hash returns the selected checksum of the file
	// If no checksum is available it returns ""
	Hash(HashType) (string, error)

	// Storable says whether this object can be stored
	Storable() bool
}

ObjectInfo provides read only information about an object.

func NewStaticObjectInfo 

func NewStaticObjectInfo(remote string, modTime time.Time, size int64, storable bool, hashes map[HashType]string, fs Info) ObjectInfo

NewStaticObjectInfo returns a static ObjectInfo If hashes is nil and fs is not nil, the hash map will be replaced with empty hashes of the types supported by the fs.

type ObjectPair 

type ObjectPair struct {
	// contains filtered or unexported fields
}

ObjectPair is a pair of Objects used to describe a potential copy operation.

type ObjectPairChan 

type ObjectPairChan chan ObjectPair

ObjectPairChan is a channel of ObjectPair

type Objects 

type Objects []Object

Objects is a slice of Object~s

type ObjectsChan 

type ObjectsChan chan Object

ObjectsChan is a channel of Objects

type OpenOption 

type OpenOption interface {
	fmt.Stringer

	// Header returns the option as an HTTP header
	Header() (key string, value string)

	// Mandatory returns whether this option can be ignored or not
	Mandatory() bool
}

OpenOption is an interface describing options for Open

type Option 

type Option struct {
	Name       string
	Help       string
	Optional   bool
	IsPassword bool
	Examples   OptionExamples `json:",omitempty"`
}

Option is describes an option for the config wizard

type OptionExample 

type OptionExample struct {
	Value string
	Help  string
}

OptionExample describes an example for an Option

type OptionExamples 

type OptionExamples []OptionExample

OptionExamples is a slice of examples

func (OptionExamples) Len 

func (os OptionExamples) Len() int

Len is part of sort.Interface.

func (OptionExamples) Less 

func (os OptionExamples) Less(i, j int) bool

Less is part of sort.Interface.

func (OptionExamples) Sort 

func (os OptionExamples) Sort()

Sort sorts an OptionExamples

func (OptionExamples) Swap 

func (os OptionExamples) Swap(i, j int)

Swap is part of sort.Interface.

type Purger 

type Purger interface {
	// Purge all files in the root and the root directory
	//
	// Implement this if you have a way of deleting all the files
	// quicker than just running Remove() on the result of List()
	//
	// Return an error if it doesn't exist
	Purge() error
}

Purger is an optional interfaces for Fs

type PutStreamer 

type PutStreamer interface {
	// PutStream uploads to the remote path with the modTime given of indeterminate size
	//
	// May create the object even if it returns an error - if so
	// will return the object and the error, otherwise will return
	// nil and the error
	PutStream(in io.Reader, src ObjectInfo, options ...OpenOption) (Object, error)
}

PutStreamer is an optional interface for Fs

type PutUncheckeder 

type PutUncheckeder interface {
	// Put in to the remote path with the modTime given of the given size
	//
	// May create the object even if it returns an error - if so
	// will return the object and the error, otherwise will return
	// nil and the error
	//
	// May create duplicates or return errors if src already
	// exists.
	PutUnchecked(in io.Reader, src ObjectInfo, options ...OpenOption) (Object, error)
}

PutUncheckeder is an optional interface for Fs

type RangeOption 

type RangeOption struct {
	Start int64
	End   int64
}

RangeOption defines an HTTP Range option with start and end. If either start or end are < 0 then they will be omitted.

func (*RangeOption) Header 

func (o *RangeOption) Header() (key string, value string)

Header formats the option as an http header

func (*RangeOption) Mandatory 

func (o *RangeOption) Mandatory() bool

Mandatory returns whether the option must be parsed or can be ignored

func (*RangeOption) String 

func (o *RangeOption) String() string

String formats the option into human readable form

type RegInfo 

type RegInfo struct {
	// Name of this fs
	Name string
	// Description of this fs - defaults to Name
	Description string
	// Create a new file system.  If root refers to an existing
	// object, then it should return a Fs which which points to
	// the parent of that object and ErrorIsFile.
	NewFs func(name string, root string) (Fs, error) `json:"-"`
	// Function to call to help with config
	Config func(string) `json:"-"`
	// Options for the Fs configuration
	Options []Option
}

RegInfo provides information about a filesystem

func Find 

func Find(name string) (*RegInfo, error)

Find looks for an Info object for the name passed in

Services are looked up in the config file

func MustFind 

func MustFind(name string) *RegInfo

MustFind looks for an Info object for the type name passed in

Services are looked up in the config file

Exits with a fatal error if not found

func MustFindByName 

func MustFindByName(name string) *RegInfo

MustFindByName finds the RegInfo for the remote name passed in or exits with a fatal error.

func ParseRemote 

func ParseRemote(path string) (fsInfo *RegInfo, configName, fsPath string, err error)

ParseRemote deconstructs a path into configName, fsPath, looking up the fsName in the config file (returning NotFoundInConfigFile if not found)

type RepeatableReader 

type RepeatableReader struct {
	// contains filtered or unexported fields
}

A RepeatableReader implements the io.ReadSeeker it allow to seek cached data back and forth within the reader but will only read data from the internal Reader as necessary and will play nicely with the Account and io.LimitedReader to reflect current speed

func NewRepeatableLimitReader 

func NewRepeatableLimitReader(r io.Reader, size int) *RepeatableReader

NewRepeatableLimitReader create new repeatable reader from Reader r with an initial buffer of size wrapped in a io.LimitReader to read only size.

func NewRepeatableLimitReaderBuffer 

func NewRepeatableLimitReaderBuffer(r io.Reader, buf []byte, size int64) *RepeatableReader

NewRepeatableLimitReaderBuffer create new repeatable reader from Reader r and buf wrapped in a io.LimitReader to read only size.

func NewRepeatableReader 

func NewRepeatableReader(r io.Reader) *RepeatableReader

NewRepeatableReader create new repeatable reader from Reader r

func NewRepeatableReaderBuffer 

func NewRepeatableReaderBuffer(r io.Reader, buf []byte) *RepeatableReader

NewRepeatableReaderBuffer create new repeatable reader from Reader r using the buffer passed in.

func NewRepeatableReaderSized 

func NewRepeatableReaderSized(r io.Reader, size int) *RepeatableReader

NewRepeatableReaderSized create new repeatable reader from Reader r with an initial buffer of size.

func (*RepeatableReader) Read 

func (r *RepeatableReader) Read(b []byte) (n int, err error)

Read data from original Reader into bytes Data is either served from the underlying Reader or from cache if was already read

func (*RepeatableReader) Seek 

func (r *RepeatableReader) Seek(offset int64, whence int) (int64, error)

Seek implements the io.Seeker interface. If seek position is passed the cache buffer length the function will return the maximum offset that can be used and "fs.RepeatableReader.Seek: offset is unavailable" Error

type Retrier 

type Retrier interface {
	error
	Retry() bool
}

Retrier is an optional interface for error as to whether the operation should be retried at a high level.

This should be returned from Update or Put methods as required

type SeekOption 

type SeekOption struct {
	Offset int64
}

SeekOption defines an HTTP Range option with start only.

func (*SeekOption) Header 

func (o *SeekOption) Header() (key string, value string)

Header formats the option as an http header

func (*SeekOption) Mandatory 

func (o *SeekOption) Mandatory() bool

Mandatory returns whether the option must be parsed or can be ignored

func (*SeekOption) String 

func (o *SeekOption) String() string

String formats the option into human readable form

type SizeSuffix 

type SizeSuffix int64

SizeSuffix is parsed by flag with k/M/G suffixes

func (*SizeSuffix) Set 

func (x *SizeSuffix) Set(s string) error

Set a SizeSuffix

func (SizeSuffix) String 

func (x SizeSuffix) String() string

String turns SizeSuffix into a string

func (*SizeSuffix) Type 

func (x *SizeSuffix) Type() string

Type of the value

func (SizeSuffix) Unit 

func (x SizeSuffix) Unit(unit string) string

Unit turns SizeSuffix into a string with a unit

type StatsInfo 

type StatsInfo struct {
	// contains filtered or unexported fields
}

StatsInfo limits and accounts all transfers

func NewStats 

func NewStats() *StatsInfo

NewStats cretates an initialised StatsInfo

func (*StatsInfo) Bytes 

func (s *StatsInfo) Bytes(bytes int64)

Bytes updates the stats for bytes bytes

func (*StatsInfo) Checking 

func (s *StatsInfo) Checking(remote string)

Checking adds a check into the stats

func (*StatsInfo) DoneChecking 

func (s *StatsInfo) DoneChecking(remote string)

DoneChecking removes a check from the stats

func (*StatsInfo) DoneTransferring 

func (s *StatsInfo) DoneTransferring(remote string, ok bool)

DoneTransferring removes a transfer from the stats

if ok is true then it increments the transfers count

func (*StatsInfo) Error 

func (s *StatsInfo) Error(err error)

Error adds a single error into the stats and assigns lastError

func (*StatsInfo) Errored 

func (s *StatsInfo) Errored() bool

Errored returns whether there have been any errors

func (*StatsInfo) Errors 

func (s *StatsInfo) Errors(errors int64)

Errors updates the stats for errors

func (*StatsInfo) GetErrors 

func (s *StatsInfo) GetErrors() int64

GetErrors reads the number of errors

func (*StatsInfo) GetLastError 

func (s *StatsInfo) GetLastError() error

GetLastError returns the lastError

func (*StatsInfo) GetTransfers 

func (s *StatsInfo) GetTransfers() int64

GetTransfers reads the number of transfers

func (*StatsInfo) Log 

func (s *StatsInfo) Log()

Log outputs the StatsInfo to the log

func (*StatsInfo) ResetCounters 

func (s *StatsInfo) ResetCounters()

ResetCounters sets the counters (bytes, checks, errors, transfers) to 0

func (*StatsInfo) ResetErrors 

func (s *StatsInfo) ResetErrors()

ResetErrors sets the errors count to 0

func (*StatsInfo) String 

func (s *StatsInfo) String() string

String convert the StatsInfo to a string for printing

func (*StatsInfo) Transferring 

func (s *StatsInfo) Transferring(remote string)

Transferring adds a transfer into the stats

type Transport 

type Transport struct {
	*http.Transport
	// contains filtered or unexported fields
}

Transport is a our http Transport which wraps an http.Transport * Sets the User Agent * Does logging

func NewTransport 

func NewTransport(transport *http.Transport, dump DumpFlags) *Transport

NewTransport wraps the http.Transport passed in and logs all roundtrips including the body if logBody is set.

func (*Transport) RoundTrip 

func (t *Transport) RoundTrip(req *http.Request) (resp *http.Response, err error)

RoundTrip implements the RoundTripper interface.

func (*Transport) SetRequestFilter 

func (t *Transport) SetRequestFilter(f func(req *http.Request))

SetRequestFilter sets a filter to be used on each request

type UnWrapper 

type UnWrapper interface {
	// UnWrap returns the Fs that this Fs is wrapping
	UnWrap() Fs
}

UnWrapper is an optional interfaces for Fs

type WalkFunc 

type WalkFunc func(path string, entries DirEntries, err error) error

WalkFunc is the type of the function called for directory visited by Walk. The path argument contains remote path to the directory.

If there was a problem walking to directory named by path, the incoming error will describe the problem and the function can decide how to handle that error (and Walk will not descend into that directory). If an error is returned, processing stops. The sole exception is when the function returns the special value ErrorSkipDir. If the function returns ErrorSkipDir, Walk skips the directory's contents entirely.

type Wrapper 

type Wrapper interface {
	// Wrap returns the Fs that is wrapping this Fs
	WrapFs() Fs
	// SetWrapper sets the Fs that is wrapping this Fs
	SetWrapper(f Fs)
}

Wrapper is an optional interfaces for Fs

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.