fs

package
v1.57.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 12, 2022 License: MIT Imports: 31 Imported by: 4

Documentation

Overview

Package fs is a generic file system interface for rclone object storage systems

Index

Examples

Constants

View Source 
const (
	// ConfigToken is the key used to store the token under
	ConfigToken = "token"

	// ConfigKeyEphemeralPrefix marks config keys which shouldn't be stored in the config file
	ConfigKeyEphemeralPrefix = "config_"
)
View Source 
const (
	CountSuffixBase CountSuffix = 1
	Kilo                        = 1000 * CountSuffixBase
	Mega                        = 1000 * Kilo
	Giga                        = 1000 * Mega
	Tera                        = 1000 * Giga
	Peta                        = 1000 * Tera
	Exa                         = 1000 * Peta
)

Common multipliers for SizeSuffix

View Source 
const (
	// CountSuffixMax is the largest CountSuffix multiplier
	CountSuffixMax = Exa
	// CountSuffixMaxValue is the largest value that can be used to create CountSuffix
	CountSuffixMaxValue = math.MaxInt64
	// CountSuffixMinValue is the smallest value that can be used to create CountSuffix
	CountSuffixMinValue = math.MinInt64
)
View Source 
const (
	DaemonMarkVar   = "_RCLONE_DAEMON_"
	DaemonMarkChild = "_rclone_daemon_"
)

We use a special environment variable to let the child process know its role.

View Source 
const (
	// ModTimeNotSupported is a very large precision value to show
	// mod time isn't supported on this Fs
	ModTimeNotSupported = 100 * 365 * 24 * time.Hour
	// MaxLevel is a sentinel representing an infinite depth for listings
	MaxLevel = math.MaxInt32
)

Constants

View Source 
const (
	// SizeSuffixMax is the largest SizeSuffix multiplier
	SizeSuffixMax = Exbi
	// SizeSuffixMaxValue is the largest value that can be used to create SizeSuffix
	SizeSuffixMaxValue = math.MaxInt64
	// SizeSuffixMinValue is the smallest value that can be used to create SizeSuffix
	SizeSuffixMinValue = math.MinInt64
)
View Source 
const ConfigAll = "*all"

ConfigAll should be passed in as the initial state to run the entire config

View Source 
const DurationOff = Duration((1 << 63) - 1)

DurationOff is the default value for flags which can be turned off

Variables

View Source 
var (

	// Read a value from the config file
	//
	// This is a function pointer to decouple the config
	// implementation from the fs
	ConfigFileGet = func(section, key string) (string, bool) { return "", false }

	// Set a value into the config file and persist it
	//
	// This is a function pointer to decouple the config
	// implementation from the fs
	ConfigFileSet = func(section, key, value string) (err error) {
		return errors.New("no config file set handler")
	}

	// CountError counts an error.  If any errors have been
	// counted then rclone will exit with a non zero error code.
	//
	// This is a function pointer to decouple the config
	// implementation from the fs
	CountError = func(err error) error { return err }

	// ConfigProvider is the config key used for provider options
	ConfigProvider = "provider"

	// ConfigEdit is the config key used to show we wish to edit existing entries
	ConfigEdit = "config_fs_edit"
)

Global

View Source 
var (
	// ErrorNotFoundInConfigFile is returned by NewFs if not found in config file
	ErrorNotFoundInConfigFile        = errors.New("didn't find section in config file")
	ErrorCantPurge                   = errors.New("can't purge directory")
	ErrorCantCopy                    = errors.New("can't copy object - incompatible remotes")
	ErrorCantMove                    = errors.New("can't move object - incompatible remotes")
	ErrorCantDirMove                 = errors.New("can't move directory - incompatible remotes")
	ErrorCantUploadEmptyFiles        = errors.New("can't upload empty files to this remote")
	ErrorDirExists                   = errors.New("can't copy directory - destination already exists")
	ErrorCantSetModTime              = errors.New("can't set modified time")
	ErrorCantSetModTimeWithoutDelete = errors.New("can't set modified time without deleting existing object")
	ErrorDirNotFound                 = errors.New("directory not found")
	ErrorObjectNotFound              = errors.New("object not found")
	ErrorLevelNotSupported           = errors.New("level value not supported")
	ErrorListAborted                 = errors.New("list aborted")
	ErrorListBucketRequired          = errors.New("bucket or container name is needed in remote")
	ErrorIsFile                      = errors.New("is a file not a directory")
	ErrorIsDir                       = errors.New("is a directory not a file")
	ErrorNotAFile                    = errors.New("is not a regular file")
	ErrorNotDeleting                 = errors.New("not deleting files as there were IO errors")
	ErrorNotDeletingDirs             = errors.New("not deleting directories as there were IO errors")
	ErrorOverlapping                 = errors.New("can't sync or move files on overlapping remotes")
	ErrorDirectoryNotEmpty           = errors.New("directory not empty")
	ErrorImmutableModified           = errors.New("immutable file modified")
	ErrorPermissionDenied            = errors.New("permission denied")
	ErrorCantShareDirectories        = errors.New("this backend can't share directories with link")
	ErrorNotImplemented              = errors.New("optional feature not implemented")
	ErrorCommandNotFound             = errors.New("command not found")
	ErrorFileNameTooLong             = errors.New("file name too long")
)

Globals

View Source 
var ConfigOAuth func(ctx context.Context, name string, m configmap.Mapper, ri *RegInfo, in ConfigIn) (*ConfigOut, error)

ConfigOAuth should be called to do the OAuth

set in lib/oauthutil to avoid a circular import

View Source 
var DumpFlagsList string

DumpFlagsList is a list of dump flags used in the help

View Source 
var LogPrint = func(level LogLevel, text string) {
	text = fmt.Sprintf("%-6s: %s", level, text)
	if LogPrintPid {
		text = fmt.Sprintf("[%d] %s", os.Getpid(), text)
	}
	_ = log.Output(4, text)
}

LogPrint sends the text to the logger of level

View Source 
var LogPrintPid = false

LogPrintPid enables process pid in log

View Source 
var PassDaemonArgsAsEnviron bool

PassDaemonArgsAsEnviron tells how CLI arguments are passed to the daemon When false, arguments are passed as is, visible in the `ps` output. When true, arguments are converted into environment variables (more secure).

View Source 
var Registry []*RegInfo

Registry of filesystems

View Source 
var Version = "v1.59.0-DEV"

Version of rclone

Functions

func CheckClose 

func CheckClose(c io.Closer, err *error)

CheckClose is a utility function used to check the return from Close in a defer statement.

func CompareDirEntries added in v1.6.1 

func CompareDirEntries(a, b DirEntry) int

CompareDirEntries returns 1 if a > b, 0 if a == b and -1 if a < b If two dir entries have the same name, compare their types (directories are before objects)

func ConfigMap 

func ConfigMap(fsInfo *RegInfo, configName string, connectionStringConfig configmap.Simple) (config *configmap.Map)

ConfigMap creates a configmap.Map from the *RegInfo and the configName passed in. If connectionStringConfig has any entries (it may be nil), then it will be added to the lookup with the highest priority.

If fsInfo is nil then the returned configmap.Map should only be used for reading non backend specific parameters, such as "type".

func ConfigOAuthOnly added in v1.56.0 

func ConfigOAuthOnly(ctx context.Context) context.Context

ConfigOAuthOnly marks the ctx so that the Config will stop after finding an OAuth

func ConfigString added in v1.50.20 

func ConfigString(f Fs) string

ConfigString returns a canonical version of the config string used to configure the Fs as passed to fs.NewFs

func ConfigToEnv 

func ConfigToEnv(section, name string) string

ConfigToEnv converts a config section and name, e.g. ("my-remote", "ignore-size") into an environment name "RCLONE_CONFIG_MY-REMOTE_IGNORE_SIZE"

func CopyConfig added in v1.55.0 

func CopyConfig(dstCtx, srcCtx context.Context) context.Context

CopyConfig copies the global config (if any) from srcCtx into dstCtx returning the new context.

func Debugf 

func Debugf(o interface{}, text string, args ...interface{})

Debugf writes debugging output for this Object or Fs. Use this for debug only. The user must have to specify -vv to see this.

func DirEntryType 

func DirEntryType(d DirEntry) string

DirEntryType returns a string description of the DirEntry, either "object", "directory" or "unknown type XXX"

func Errorf 

func Errorf(o interface{}, text string, args ...interface{})

Errorf writes error log output for this Object or Fs. It should always be seen by the user.

func FileExists 

func FileExists(ctx context.Context, fs Fs, remote string) (bool, error)

FileExists returns true if a file remote exists. If remote is a directory, FileExists returns false.

func Fingerprint added in v1.50.20 

func Fingerprint(ctx context.Context, o ObjectInfo, fast bool) string

Fingerprint produces a unique-ish string for an object.

This is for detecting whether an object has changed since we last saw it, not for checking object identity between two different remotes - operations.Equal should be used for that.

If fast is set then Fingerprint will only include attributes where usually another operation is not required to fetch them. For example if fast is set then this won't include hashes on the local backend.

func FixRangeOption 

func FixRangeOption(options []OpenOption, size int64)

FixRangeOption looks through the slice of options and adjusts any RangeOption~s found that request a fetch from the end into an absolute fetch using the size passed in and makes sure the range does not exceed filesize. Some remotes (e.g. Onedrive, Box) don't support range requests which index from the end.

func GetModifyWindow 

func GetModifyWindow(ctx context.Context, fss ...Info) time.Duration

GetModifyWindow calculates the maximum modify window between the given Fses and the Config.ModifyWindow parameter.

func Infof 

func Infof(o interface{}, text string, args ...interface{})

Infof writes info on transfers for this Object or Fs. Use this level for logging transfers, deletions and things which should appear with the -v flag.

func IsDaemon added in v1.56.0 

func IsDaemon() bool

IsDaemon returns true if this process runs in background

func IsMountHelper added in v1.56.0 

func IsMountHelper() bool

IsMountHelper returns true if rclone was invoked as mount helper: as /sbin/mount.rlone (by /bin/mount) or /usr/bin/rclonefs (by fusermount or directly)

func LogDirName 

func LogDirName(f Fs, dir string) interface{}

LogDirName returns an object for the logger, logging a root directory which would normally be "" as the Fs

func LogLevelPrintf 

func LogLevelPrintf(level LogLevel, o interface{}, text string, args ...interface{})

LogLevelPrintf writes logs at the given level

func LogPrintf 

func LogPrintf(level LogLevel, o interface{}, text string, args ...interface{})

LogPrintf produces a log string from the arguments passed in

func Logf 

func Logf(o interface{}, text string, args ...interface{})

Logf writes log output for this Object or Fs. This should be considered to be Notice level logging. It is the default level. By default rclone should not log very much so only use this for important things the user should see. The user can filter these out with the -q flag.

func MatchProvider added in v1.56.0 

func MatchProvider(providerConfig, provider string) bool

MatchProvider returns true if provider matches the providerConfig string.

The providerConfig string can either be a list of providers to match, or if it starts with "!" it will be a list of providers not to match.

If either providerConfig or provider is blank then it will return true

func MimeType 

func MimeType(ctx context.Context, o ObjectInfo) (mimeType string)

MimeType returns the MimeType from the object, either by calling the MimeTyper interface or using MimeTypeFromName

func MimeTypeDirEntry 

func MimeTypeDirEntry(ctx context.Context, item DirEntry) string

MimeTypeDirEntry returns the MimeType of a DirEntry

It returns "inode/directory" for directories, or uses MimeType(Object)

func MimeTypeFromName 

func MimeTypeFromName(remote string) (mimeType string)

MimeTypeFromName returns a guess at the mime type from the name

func NewUsageValue 

func NewUsageValue(value int64) *int64

NewUsageValue makes a valid value

func ObjectOptionalInterfaces 

func ObjectOptionalInterfaces(o Object) (supported, unsupported []string)

ObjectOptionalInterfaces returns the names of supported and unsupported optional interfaces for an Object

func OpenOptionAddHTTPHeaders 

func OpenOptionAddHTTPHeaders(headers http.Header, options []OpenOption)

OpenOptionAddHTTPHeaders Sets each header found in options to the http.Header map provided the key was non empty.

func OpenOptionAddHeaders 

func OpenOptionAddHeaders(options []OpenOption, headers map[string]string)

OpenOptionAddHeaders adds each header found in options to the headers map provided the key was non empty.

func OpenOptionHeaders 

func OpenOptionHeaders(options []OpenOption) (headers map[string]string)

OpenOptionHeaders adds each header found in options to the headers map provided the key was non empty.

It returns a nil map if options was empty

func OptionToEnv 

func OptionToEnv(name string) string

OptionToEnv converts an option name, e.g. "ignore-size" into an environment name "RCLONE_IGNORE_SIZE"

func ParseDuration 

func ParseDuration(age string) (time.Duration, error)

ParseDuration parses a duration string. Accept ms|s|m|h|d|w|M|y suffixes. Defaults to second if not provided

func Register 

func Register(info *RegInfo)

Register a filesystem

Fs modules should use this in an init() function

func StatePop added in v1.56.0 

func StatePop(state string) (newState string, value string)

StatePop pops a state from the front of the config string It returns the new state and the value popped

func StatePush added in v1.56.0 

func StatePush(state string, values ...string) string

StatePush pushes a new values onto the front of the config string

func UnmarshalJSONFlag added in v1.55.0 

func UnmarshalJSONFlag(in []byte, x interface{ Set(string) error }, setInt func(int64) error) error

UnmarshalJSONFlag unmarshals a JSON input for a flag. If the input is a string then it calls the Set method on the flag otherwise it calls the setInt function with a parsed int64.

Types

type Abouter 

type Abouter interface {
	// About gets quota information from the Fs
	About(ctx context.Context) (*Usage, error)
}

Abouter is an optional interface for Fs

type BaseOption added in v1.6.1 

type BaseOption Option

BaseOption is an alias for Option used internally

type BwPair added in v1.55.0 

type BwPair struct {
	Tx SizeSuffix // upload bandwidth
	Rx SizeSuffix // download bandwidth
}

BwPair represents an upload and a download bandwidth

func (*BwPair) IsSet added in v1.55.0 

func (bp *BwPair) IsSet() bool

IsSet returns true if either of the bandwidth limits are set

func (*BwPair) Set added in v1.55.0 

func (bp *BwPair) Set(s string) (err error)

Set the bandwidth from a string which is either SizeSuffix or SizeSuffix:SizeSuffix (for tx:rx bandwidth)

func (*BwPair) String added in v1.55.0 

func (bp *BwPair) String() string

String returns a printable representation of a BwPair

type BwTimeSlot 

type BwTimeSlot struct {
	DayOfTheWeek int
	HHMM         int
	Bandwidth    BwPair
}

BwTimeSlot represents a bandwidth configuration at a point in time.

type BwTimetable 

type BwTimetable []BwTimeSlot

BwTimetable contains all configured time slots.

func (BwTimetable) LimitAt 

func (x BwTimetable) LimitAt(tt time.Time) BwTimeSlot

LimitAt returns a BwTimeSlot for the time requested.

func (BwTimetable) MarshalJSON added in v1.55.0 

func (x BwTimetable) MarshalJSON() ([]byte, error)

MarshalJSON marshals as a string value

func (*BwTimetable) Set 

func (x *BwTimetable) Set(s string) error

Set the bandwidth timetable.

func (BwTimetable) String 

func (x BwTimetable) String() string

String returns a printable representation of BwTimetable.

func (BwTimetable) Type 

func (x BwTimetable) Type() string

Type of the value

func (*BwTimetable) UnmarshalJSON added in v1.55.0 

func (x *BwTimetable) UnmarshalJSON(in []byte) error

UnmarshalJSON unmarshals a string value

type ChangeNotifier 

type ChangeNotifier interface {
	// ChangeNotify calls the passed function with a path
	// that has had changes. If the implementation
	// uses polling, it should adhere to the given interval.
	// At least one value will be written to the channel,
	// specifying the initial value and updated values might
	// follow. A 0 Duration should pause the polling.
	// The ChangeNotify implementation must empty the channel
	// regularly. When the channel gets closed, the implementation
	// should stop polling and release resources.
	ChangeNotify(context.Context, func(string, EntryType), <-chan time.Duration)
}

ChangeNotifier is an optional interface for Fs

type CleanUpper 

type CleanUpper interface {
	// CleanUp the trash in the Fs
	//
	// Implement this if you have a way of emptying the trash or
	// otherwise cleaning up old versions of files.
	CleanUp(ctx context.Context) error
}

CleanUpper is an optional interfaces for Fs

type CommaSepList 

type CommaSepList []string

CommaSepList is a comma separated config value It uses the encoding/csv rules for quoting and escaping

Example 
for _, s := range []string{
	`remotea:test/dir,remoteb:`,
	`"remotea:test/space dir",remoteb:`,
	`"remotea:test/quote""dir",remoteb:`,
} {
	var l CommaSepList
	must(l.Set(s))
	fmt.Printf("%#v\n", l)
}

Output:

fs.CommaSepList{"remotea:test/dir", "remoteb:"}
fs.CommaSepList{"remotea:test/space dir", "remoteb:"}
fs.CommaSepList{"remotea:test/quote\"dir", "remoteb:"}

func (*CommaSepList) Scan 

func (l *CommaSepList) Scan(s fmt.ScanState, ch rune) error

Scan implements the fmt.Scanner interface

func (*CommaSepList) Set 

func (l *CommaSepList) Set(s string) error

Set the List entries

func (CommaSepList) String 

func (l CommaSepList) String() string

func (CommaSepList) Type 

func (CommaSepList) Type() string

Type of the value

type CommandHelp added in v1.50.20 

type CommandHelp struct {
	Name  string            // Name of the command, e.g. "link"
	Short string            // Single line description
	Long  string            // Long multi-line description
	Opts  map[string]string // maps option name to a single line help
}

CommandHelp describes a single backend Command

These are automatically inserted in the docs

type Commander added in v1.50.20 

type Commander interface {
	// Command the backend to run a named command
	//
	// The command run is name
	// args may be used to read arguments from
	// opts may be used to read optional arguments from
	//
	// The result should be capable of being JSON encoded
	// If it is a string or a []string it will be shown to the user
	// otherwise it will be JSON encoded and shown to the user like that
	Command(ctx context.Context, name string, arg []string, opt map[string]string) (interface{}, error)
}

Commander is an interface to wrap the Command function

type ConfigIn added in v1.56.0 

type ConfigIn struct {
	State  string // State to run
	Result string // Result from previous Option
}

ConfigIn is passed to the Config function for an Fs

The interactive config system for backends is state based. This is so that different frontends to the config can be attached, eg over the API or web page.

Each call to the config system supplies ConfigIn which tells the system what to do. Each will return a ConfigOut which gives a question to ask the user and a state to return to. There is one special question which allows the backends to do OAuth.

The ConfigIn contains a State which the backend should act upon and a Result from the previous question to the user.

If ConfigOut is nil or ConfigOut.State == "" then the process is deemed to have finished. If there is no Option in ConfigOut then the next state will be called immediately. This is wrapped in ConfigGoto and ConfigResult.

Backends should keep no state in memory - if they need to persist things between calls it should be persisted in the config file. Things can also be persisted in the state using the StatePush and StatePop utilities here.

The utilities here are convenience methods for different kinds of questions and responses.

Where the questions ask for a name then this should start with "config_" to show it is an ephemeral config input rather than the actual value stored in the config file. Names beginning with "config_fs_" are reserved for internal use.

State names starting with "*" are reserved for internal use.

Note that in the bin directory there is a python program called "config.py" which shows how this interface should be used.

type ConfigInfo 

type ConfigInfo struct {
	LogLevel               LogLevel
	StatsLogLevel          LogLevel
	UseJSONLog             bool
	DryRun                 bool
	Interactive            bool
	CheckSum               bool
	SizeOnly               bool
	IgnoreTimes            bool
	IgnoreExisting         bool
	IgnoreErrors           bool
	ModifyWindow           time.Duration
	Checkers               int
	Transfers              int
	ConnectTimeout         time.Duration // Connect timeout
	Timeout                time.Duration // Data channel timeout
	ExpectContinueTimeout  time.Duration
	Dump                   DumpFlags
	InsecureSkipVerify     bool // Skip server certificate verification
	DeleteMode             DeleteMode
	MaxDelete              int64
	TrackRenames           bool   // Track file renames.
	TrackRenamesStrategy   string // Comma separated list of strategies used to track renames
	LowLevelRetries        int
	UpdateOlder            bool // Skip files that are newer on the destination
	NoGzip                 bool // Disable compression
	MaxDepth               int
	IgnoreSize             bool
	IgnoreChecksum         bool
	IgnoreCaseSync         bool
	NoTraverse             bool
	CheckFirst             bool
	NoCheckDest            bool
	NoUnicodeNormalization bool
	NoUpdateModTime        bool
	DataRateUnit           string
	CompareDest            []string
	CopyDest               []string
	BackupDir              string
	Suffix                 string
	SuffixKeepExtension    bool
	UseListR               bool
	BufferSize             SizeSuffix
	BwLimit                BwTimetable
	BwLimitFile            BwTimetable
	TPSLimit               float64
	TPSLimitBurst          int
	BindAddr               net.IP
	DisableFeatures        []string
	UserAgent              string
	Immutable              bool
	AutoConfirm            bool
	StreamingUploadCutoff  SizeSuffix
	StatsFileNameLength    int
	AskPassword            bool
	PasswordCommand        SpaceSepList
	UseServerModTime       bool
	MaxTransfer            SizeSuffix
	MaxDuration            time.Duration
	CutoffMode             CutoffMode
	MaxBacklog             int
	MaxStatsGroups         int
	StatsOneLine           bool
	StatsOneLineDate       bool   // If we want a date prefix at all
	StatsOneLineDateFormat string // If we want to customize the prefix
	ErrorOnNoTransfer      bool   // Set appropriate exit code if no files transferred
	Progress               bool
	ProgressTerminalTitle  bool
	Cookie                 bool
	UseMmap                bool
	CaCert                 string // Client Side CA
	ClientCert             string // Client Side Cert
	ClientKey              string // Client Side Key
	MultiThreadCutoff      SizeSuffix
	MultiThreadStreams     int
	MultiThreadSet         bool   // whether MultiThreadStreams was set (set in fs/config/configflags)
	OrderBy                string // instructions on how to order the transfer
	UploadHeaders          []*HTTPOption
	DownloadHeaders        []*HTTPOption
	Headers                []*HTTPOption
	RefreshTimes           bool
	NoConsole              bool
	TrafficClass           uint8
	FsCacheExpireDuration  time.Duration
	FsCacheExpireInterval  time.Duration
	DisableHTTP2           bool
	HumanReadable          bool
	KvLockTime             time.Duration // maximum time to keep key-value database locked by process
}

ConfigInfo is filesystem config options

func AddConfig added in v1.55.0 

func AddConfig(ctx context.Context) (context.Context, *ConfigInfo)

AddConfig returns a mutable config structure based on a shallow copy of that found in ctx and returns a new context with that added to it.

func GetConfig added in v1.55.0 

func GetConfig(ctx context.Context) *ConfigInfo

GetConfig returns the global or context sensitive context

func NewConfig 

func NewConfig() *ConfigInfo

NewConfig creates a new config with everything set to the default value. These are the ultimate defaults and are overridden by the config module.

func (*ConfigInfo) TimeoutOrInfinite added in v1.55.0 

func (c *ConfigInfo) TimeoutOrInfinite() time.Duration

TimeoutOrInfinite returns ci.Timeout if > 0 or infinite otherwise

type ConfigOut added in v1.56.0 

type ConfigOut struct {
	State  string      // State to jump to after this
	Option *Option     // Option to query user about
	OAuth  interface{} `json:"-"` // Do OAuth if set
	Error  string      // error to be displayed to the user
	Result string      // if Option/OAuth not set then this is passed to the next state
}

ConfigOut is returned from Config function for an Fs

State is the state for the next call to Config OAuth is a special value set by oauthutil.ConfigOAuth Error is displayed to the user before asking a question Result is passed to the next call to Config if Option/OAuth isn't set

func BackendConfig added in v1.56.0 

func BackendConfig(ctx context.Context, name string, m configmap.Mapper, ri *RegInfo, choices configmap.Getter, in ConfigIn) (out *ConfigOut, err error)

BackendConfig calls the config for the backend in ri

It wraps any OAuth transactions as necessary so only straight forward config questions are emitted

func ConfigChoose added in v1.56.0 

func ConfigChoose(state string, name string, help string, n int, getItem func(i int) (itemValue string, itemHelp string)) (*ConfigOut, error)

ConfigChoose returns a ConfigOut structure which has a list of items to choose from.

state should be the next state required name is the config name for this item help should be the help shown to the user n should be the number of items in the list getItem should return the items (value, help)

It chooses the first item to be the default. If there are no items then it will return an error. If there is only one item it will short cut to the next state

func ConfigChooseFixed added in v1.56.0 

func ConfigChooseFixed(state string, name string, help string, items []OptionExample) (*ConfigOut, error)

ConfigChooseFixed returns a ConfigOut structure which has a list of items to choose from.

state should be the next state required name is the config name for this item help should be the help shown to the user items should be the items in the list

It chooses the first item to be the default. If there are no items then it will return an error. If there is only one item it will short cut to the next state

func ConfigConfirm added in v1.56.0 

func ConfigConfirm(state string, Default bool, name string, help string) (*ConfigOut, error)

ConfigConfirm returns a ConfigOut structure which asks a Yes/No question

state should be the next state required Default should be the default state name is the config name for this item help should be the help shown to the user

func ConfigError added in v1.56.0 

func ConfigError(state string, Error string) (*ConfigOut, error)

ConfigError shows the error to the user and goes to the state passed in

state should be the next state required Error should be the error shown to the user

func ConfigGoto added in v1.56.0 

func ConfigGoto(state string) (*ConfigOut, error)

ConfigGoto goes to the next state with empty Result

state should be the next state required

func ConfigInput added in v1.56.0 

func ConfigInput(state string, name string, help string) (*ConfigOut, error)

ConfigInput asks the user for a non-empty string

state should be the next state required name is the config name for this item help should be the help shown to the user

func ConfigInputOptional added in v1.56.0 

func ConfigInputOptional(state string, name string, help string) (*ConfigOut, error)

ConfigInputOptional asks the user for a string which may be empty

state should be the next state required name is the config name for this item help should be the help shown to the user

func ConfigPassword added in v1.56.0 

func ConfigPassword(state string, name string, help string) (*ConfigOut, error)

ConfigPassword asks the user for a password

state should be the next state required name is the config name for this item help should be the help shown to the user

func ConfigResult added in v1.56.0 

func ConfigResult(state, result string) (*ConfigOut, error)

ConfigResult goes to the next state with result given

state should be the next state required result should be the result for the next state

type Copier 

type Copier interface {
	// Copy src to this remote using server-side copy operations.
	//
	// This is stored with the remote path given
	//
	// It returns the destination Object and a possible error
	//
	// Will only be called if src.Fs().Name() == f.Name()
	//
	// If it isn't possible then return fs.ErrorCantCopy
	Copy(ctx context.Context, src Object, remote string) (Object, error)
}

Copier is an optional interface for Fs

type CountSuffix added in v1.56.0 

type CountSuffix int64

CountSuffix is an int64 with a friendly way of printing setting

func (*CountSuffix) Scan added in v1.56.0 

func (x *CountSuffix) Scan(s fmt.ScanState, ch rune) error

Scan implements the fmt.Scanner interface

func (*CountSuffix) Set added in v1.56.0 

func (x *CountSuffix) Set(s string) error

Set a CountSuffix

func (CountSuffix) String added in v1.56.0 

func (x CountSuffix) String() string

String turns CountSuffix into a string

func (*CountSuffix) Type added in v1.56.0 

func (x *CountSuffix) Type() string

Type of the value

func (CountSuffix) Unit added in v1.56.0 

func (x CountSuffix) Unit(unit string) string

Unit turns CountSuffix into a string with a unit

func (*CountSuffix) UnmarshalJSON added in v1.56.0 

func (x *CountSuffix) UnmarshalJSON(in []byte) error

UnmarshalJSON makes sure the value can be parsed as a string or integer in JSON

type CountSuffixList added in v1.56.0 

type CountSuffixList []CountSuffix

CountSuffixList is a slice CountSuffix values

func (CountSuffixList) Len added in v1.56.0 

func (l CountSuffixList) Len() int

func (CountSuffixList) Less added in v1.56.0 

func (l CountSuffixList) Less(i, j int) bool

func (CountSuffixList) Sort added in v1.56.0 

func (l CountSuffixList) Sort()

Sort sorts the list

func (CountSuffixList) Swap added in v1.56.0 

func (l CountSuffixList) Swap(i, j int)

type CutoffMode added in v1.50.12 

type CutoffMode byte

CutoffMode describes the possible delete modes in the config 

const (
	CutoffModeHard CutoffMode = iota
	CutoffModeSoft
	CutoffModeCautious
	CutoffModeDefault = CutoffModeHard
)

MaxTransferMode constants

func (*CutoffMode) Set added in v1.50.12 

func (m *CutoffMode) Set(s string) error

Set a LogLevel

func (CutoffMode) String added in v1.50.12 

func (m CutoffMode) String() string

String turns a LogLevel into a string

func (*CutoffMode) Type added in v1.50.12 

func (m *CutoffMode) Type() string

Type of the value

func (*CutoffMode) UnmarshalJSON added in v1.55.0 

func (m *CutoffMode) UnmarshalJSON(in []byte) error

UnmarshalJSON makes sure the value can be parsed as a string or integer in JSON

type DeleteMode 

type DeleteMode byte

DeleteMode describes the possible delete modes in the config 

const (
	DeleteModeOff DeleteMode = iota
	DeleteModeBefore
	DeleteModeDuring
	DeleteModeAfter
	DeleteModeOnly
	DeleteModeDefault = DeleteModeAfter
)

DeleteMode constants

type Dir 

type Dir struct {
	// contains filtered or unexported fields
}

Dir describes an unspecialized directory for directory/container/bucket lists

func NewDir 

func NewDir(remote string, modTime time.Time) *Dir

NewDir creates an unspecialized Directory object

func NewDirCopy 

func NewDirCopy(ctx context.Context, d Directory) *Dir

NewDirCopy creates an unspecialized copy of the Directory object passed in

func (*Dir) ID 

func (d *Dir) ID() string

ID gets the optional ID

func (*Dir) Items 

func (d *Dir) Items() int64

Items returns the count of items in this directory or this directory and subdirectories if known, -1 for unknown

func (*Dir) ModTime 

func (d *Dir) ModTime(ctx context.Context) time.Time

ModTime returns the modification date of the file It should return a best guess if one isn't available

func (*Dir) ParentID added in v1.55.0 

func (d *Dir) ParentID() string

ParentID returns the IDs of the Dir parent if known

func (*Dir) Remote 

func (d *Dir) Remote() string

Remote returns the remote path

func (*Dir) SetID 

func (d *Dir) SetID(id string) *Dir

SetID sets the optional ID

func (*Dir) SetItems 

func (d *Dir) SetItems(items int64) *Dir

SetItems sets the number of items in the directory

func (*Dir) SetParentID added in v1.55.0 

func (d *Dir) SetParentID(parent string) *Dir

SetParentID sets the optional parent ID of the Dir

func (*Dir) SetRemote 

func (d *Dir) SetRemote(remote string) *Dir

SetRemote sets the remote

func (*Dir) SetSize 

func (d *Dir) SetSize(size int64) *Dir

SetSize sets the size of the directory

func (*Dir) Size 

func (d *Dir) Size() int64

Size returns the size of the file

func (*Dir) String 

func (d *Dir) String() string

String returns the name

type DirCacheFlusher 

type DirCacheFlusher interface {
	// DirCacheFlush resets the directory cache - used in testing
	// as an optional interface
	DirCacheFlush()
}

DirCacheFlusher is an optional interface for Fs

type DirEntries 

type DirEntries []DirEntry

DirEntries is a slice of Object or *Dir

func (DirEntries) ForDir 

func (ds DirEntries) ForDir(fn func(dir Directory))

ForDir runs the function supplied on every Directory in the entries

func (DirEntries) ForDirError 

func (ds DirEntries) ForDirError(fn func(dir Directory) error) error

ForDirError runs the function supplied on every Directory in the entries

func (DirEntries) ForObject 

func (ds DirEntries) ForObject(fn func(o Object))

ForObject runs the function supplied on every object in the entries

func (DirEntries) ForObjectError 

func (ds DirEntries) ForObjectError(fn func(o Object) error) error

ForObjectError runs the function supplied on every object in the entries

func (DirEntries) Len 

func (ds DirEntries) Len() int

Len is part of sort.Interface.

func (DirEntries) Less 

func (ds DirEntries) Less(i, j int) bool

Less is part of sort.Interface.

func (DirEntries) Swap 

func (ds DirEntries) Swap(i, j int)

Swap is part of sort.Interface.

type DirEntry 

type DirEntry interface {
	// String returns a description of the Object
	String() string

	// Remote returns the remote path
	Remote() string

	// ModTime returns the modification date of the file
	// It should return a best guess if one isn't available
	ModTime(context.Context) time.Time

	// Size returns the size of the file
	Size() int64
}

DirEntry provides read only information about the common subset of a Dir or Object. These are returned from directory listings - type assert them into the correct type.

type DirMover 

type DirMover interface {
	// DirMove moves src, srcRemote to this remote at dstRemote
	// using server-side move operations.
	//
	// Will only be called if src.Fs().Name() == f.Name()
	//
	// If it isn't possible then return fs.ErrorCantDirMove
	//
	// If destination exists then return fs.ErrorDirExists
	DirMove(ctx context.Context, src Fs, srcRemote, dstRemote string) error
}

DirMover is an optional interface for Fs

type Directory 

type Directory interface {
	DirEntry

	// Items returns the count of items in this directory or this
	// directory and subdirectories if known, -1 for unknown
	Items() int64

	// ID returns the internal ID of this directory if known, or
	// "" otherwise
	ID() string
}

Directory is a filesystem like directory provided by an Fs

type Disconnecter added in v1.6.1 

type Disconnecter interface {
	// Disconnect the current user
	Disconnect(ctx context.Context) error
}

Disconnecter is an optional interface for Fs

type DumpFlags 

type DumpFlags int

DumpFlags describes the Dump options in force 

const (
	DumpHeaders DumpFlags = 1 << iota
	DumpBodies
	DumpRequests
	DumpResponses
	DumpAuth
	DumpFilters
	DumpGoRoutines
	DumpOpenFiles
)

DumpFlags definitions

func (*DumpFlags) Set 

func (f *DumpFlags) Set(s string) error

Set a DumpFlags as a comma separated list of flags

func (DumpFlags) String 

func (f DumpFlags) String() string

String turns a DumpFlags into a string

func (*DumpFlags) Type 

func (f *DumpFlags) Type() string

Type of the value

func (*DumpFlags) UnmarshalJSON added in v1.55.0 

func (f *DumpFlags) UnmarshalJSON(in []byte) error

UnmarshalJSON makes sure the value can be parsed as a string or integer in JSON

type Duration 

type Duration time.Duration

Duration is a time.Duration with some more parsing options

func (Duration) IsSet 

func (d Duration) IsSet() bool

IsSet returns if the duration is != DurationOff

func (Duration) ReadableString added in v1.6.1 

func (d Duration) ReadableString() string

ReadableString parses d into a human-readable duration. Based on https://github.com/hako/durafmt

func (*Duration) Scan 

func (d *Duration) Scan(s fmt.ScanState, ch rune) error

Scan implements the fmt.Scanner interface

func (*Duration) Set 

func (d *Duration) Set(s string) error

Set a Duration

func (Duration) String 

func (d Duration) String() string

Turn Duration into a string

func (Duration) Type 

func (d Duration) Type() string

Type of the value

func (*Duration) UnmarshalJSON added in v1.55.0 

func (d *Duration) UnmarshalJSON(in []byte) error

UnmarshalJSON makes sure the value can be parsed as a string or integer in JSON

type EntryType 

type EntryType int

EntryType can be associated with remote paths to identify their type 

const (
	// EntryDirectory should be used to classify remote paths in directories
	EntryDirectory EntryType = iota // 0
	// EntryObject should be used to classify remote paths in objects
	EntryObject // 1
)

Constants

type Features 

type Features struct {
	// Feature flags, whether Fs
	CaseInsensitive         bool // has case insensitive files
	DuplicateFiles          bool // allows duplicate files
	ReadMimeType            bool // can read the mime type of objects
	WriteMimeType           bool // can set the mime type of objects
	CanHaveEmptyDirectories bool // can have empty directories
	BucketBased             bool // is bucket based (like s3, swift, etc.)
	BucketBasedRootOK       bool // is bucket based and can use from root
	SetTier                 bool // allows set tier functionality on objects
	GetTier                 bool // allows to retrieve storage tier of objects
	ServerSideAcrossConfigs bool // can server-side copy between different remotes of the same type
	IsLocal                 bool // is the local backend
	SlowModTime             bool // if calling ModTime() generally takes an extra transaction
	SlowHash                bool // if calling Hash() generally takes an extra transaction

	// Purge all files in the directory specified
	//
	// Implement this if you have a way of deleting all the files
	// quicker than just running Remove() on the result of List()
	//
	// Return an error if it doesn't exist
	Purge func(ctx context.Context, dir string) error

	// Copy src to this remote using server-side copy operations.
	//
	// This is stored with the remote path given
	//
	// It returns the destination Object and a possible error
	//
	// Will only be called if src.Fs().Name() == f.Name()
	//
	// If it isn't possible then return fs.ErrorCantCopy
	Copy func(ctx context.Context, src Object, remote string) (Object, error)

	// Move src to this remote using server-side move operations.
	//
	// This is stored with the remote path given
	//
	// It returns the destination Object and a possible error
	//
	// Will only be called if src.Fs().Name() == f.Name()
	//
	// If it isn't possible then return fs.ErrorCantMove
	Move func(ctx context.Context, src Object, remote string) (Object, error)

	// DirMove moves src, srcRemote to this remote at dstRemote
	// using server-side move operations.
	//
	// Will only be called if src.Fs().Name() == f.Name()
	//
	// If it isn't possible then return fs.ErrorCantDirMove
	//
	// If destination exists then return fs.ErrorDirExists
	DirMove func(ctx context.Context, src Fs, srcRemote, dstRemote string) error

	// ChangeNotify calls the passed function with a path
	// that has had changes. If the implementation
	// uses polling, it should adhere to the given interval.
	ChangeNotify func(context.Context, func(string, EntryType), <-chan time.Duration)

	// UnWrap returns the Fs that this Fs is wrapping
	UnWrap func() Fs

	// WrapFs returns the Fs that is wrapping this Fs
	WrapFs func() Fs

	// SetWrapper sets the Fs that is wrapping this Fs
	SetWrapper func(f Fs)

	// DirCacheFlush resets the directory cache - used in testing
	// as an optional interface
	DirCacheFlush func()

	// PublicLink generates a public link to the remote path (usually readable by anyone)
	PublicLink func(ctx context.Context, remote string, expire Duration, unlink bool) (string, error)

	// Put in to the remote path with the modTime given of the given size
	//
	// May create the object even if it returns an error - if so
	// will return the object and the error, otherwise will return
	// nil and the error
	//
	// May create duplicates or return errors if src already
	// exists.
	PutUnchecked func(ctx context.Context, in io.Reader, src ObjectInfo, options ...OpenOption) (Object, error)

	// PutStream uploads to the remote path with the modTime given of indeterminate size
	//
	// May create the object even if it returns an error - if so
	// will return the object and the error, otherwise will return
	// nil and the error
	PutStream func(ctx context.Context, in io.Reader, src ObjectInfo, options ...OpenOption) (Object, error)

	// MergeDirs merges the contents of all the directories passed
	// in into the first one and rmdirs the other directories.
	MergeDirs func(ctx context.Context, dirs []Directory) error

	// CleanUp the trash in the Fs
	//
	// Implement this if you have a way of emptying the trash or
	// otherwise cleaning up old versions of files.
	CleanUp func(ctx context.Context) error

	// ListR lists the objects and directories of the Fs starting
	// from dir recursively into out.
	//
	// dir should be "" to start from the root, and should not
	// have trailing slashes.
	//
	// This should return ErrDirNotFound if the directory isn't
	// found.
	//
	// It should call callback for each tranche of entries read.
	// These need not be returned in any particular order.  If
	// callback returns an error then the listing will stop
	// immediately.
	//
	// Don't implement this unless you have a more efficient way
	// of listing recursively that doing a directory traversal.
	ListR ListRFn

	// About gets quota information from the Fs
	About func(ctx context.Context) (*Usage, error)

	// OpenWriterAt opens with a handle for random access writes
	//
	// Pass in the remote desired and the size if known.
	//
	// It truncates any existing object
	OpenWriterAt func(ctx context.Context, remote string, size int64) (WriterAtCloser, error)

	// UserInfo returns info about the connected user
	UserInfo func(ctx context.Context) (map[string]string, error)

	// Disconnect the current user
	Disconnect func(ctx context.Context) error

	// Command the backend to run a named command
	//
	// The command run is name
	// args may be used to read arguments from
	// opts may be used to read optional arguments from
	//
	// The result should be capable of being JSON encoded
	// If it is a string or a []string it will be shown to the user
	// otherwise it will be JSON encoded and shown to the user like that
	Command func(ctx context.Context, name string, arg []string, opt map[string]string) (interface{}, error)

	// Shutdown the backend, closing any background tasks and any
	// cached connections.
	Shutdown func(ctx context.Context) error
}

Features describe the optional features of the Fs

func (*Features) Disable 

func (ft *Features) Disable(name string) *Features

Disable nil's out the named feature. If it isn't found then it will log a message.

func (*Features) DisableList 

func (ft *Features) DisableList(list []string) *Features

DisableList nil's out the comma separated list of named features. If it isn't found then it will log a message.

func (*Features) Enabled added in v1.6.1 

func (ft *Features) Enabled() (features map[string]bool)

Enabled returns a map of features with keys showing whether they are enabled or not

func (*Features) Fill 

func (ft *Features) Fill(ctx context.Context, f Fs) *Features

Fill fills in the function pointers in the Features struct from the optional interfaces. It returns the original updated Features struct passed in.

func (*Features) List 

func (ft *Features) List() (out []string)

List returns a slice of all the possible feature names

func (*Features) Mask 

func (ft *Features) Mask(ctx context.Context, f Fs) *Features

Mask the Features with the Fs passed in

Only optional features which are implemented in both the original Fs AND the one passed in will be advertised. Any features which aren't in both will be set to false/nil, except for UnWrap/Wrap which will be left untouched.

func (*Features) Wrap 

func (ft *Features) Wrap(f Fs) *Features

Wrap makes a Copy of the features passed in, overriding the UnWrap/Wrap method only if available in f.

func (*Features) WrapsFs 

func (ft *Features) WrapsFs(f Fs, w Fs) *Features

WrapsFs adds extra information between `f` which wraps `w`

type Fs 

type Fs interface {
	Info

	// List the objects and directories in dir into entries.  The
	// entries can be returned in any order but should be for a
	// complete directory.
	//
	// dir should be "" to list the root, and should not have
	// trailing slashes.
	//
	// This should return ErrDirNotFound if the directory isn't
	// found.
	List(ctx context.Context, dir string) (entries DirEntries, err error)

	// NewObject finds the Object at remote.  If it can't be found
	// it returns the error ErrorObjectNotFound.
	//
	// If remote points to a directory then it should return
	// ErrorIsDir if possible without doing any extra work,
	// otherwise ErrorObjectNotFound.
	NewObject(ctx context.Context, remote string) (Object, error)

	// Put in to the remote path with the modTime given of the given size
	//
	// When called from outside an Fs by rclone, src.Size() will always be >= 0.
	// But for unknown-sized objects (indicated by src.Size() == -1), Put should either
	// return an error or upload it properly (rather than e.g. calling panic).
	//
	// May create the object even if it returns an error - if so
	// will return the object and the error, otherwise will return
	// nil and the error
	Put(ctx context.Context, in io.Reader, src ObjectInfo, options ...OpenOption) (Object, error)

	// Mkdir makes the directory (container, bucket)
	//
	// Shouldn't return an error if it already exists
	Mkdir(ctx context.Context, dir string) error

	// Rmdir removes the directory (container, bucket) if empty
	//
	// Return an error if it doesn't exist or isn't empty
	Rmdir(ctx context.Context, dir string) error
}

Fs is the interface a cloud storage system must provide

func NewFs 

func NewFs(ctx context.Context, path string) (Fs, error)

NewFs makes a new Fs object from the path

The path is of the form remote:path

Remotes are looked up in the config file. If the remote isn't found then NotFoundInConfigFile will be returned.

On Windows avoid single character remote names as they can be mixed up with drive letters.

func TemporaryLocalFs 

func TemporaryLocalFs(ctx context.Context) (Fs, error)

TemporaryLocalFs creates a local FS in the OS's temporary directory.

No cleanup is performed, the caller must call Purge on the Fs themselves.

func UnWrapFs added in v1.6.1 

func UnWrapFs(f Fs) Fs

UnWrapFs unwraps f as much as possible and returns the base Fs

type FullObject added in v1.50.4 

type FullObject interface {
	Object
	MimeTyper
	IDer
	ObjectUnWrapper
	GetTierer
	SetTierer
}

FullObject contains all the optional interfaces for Object

Use for checking making wrapping Objects implement everything

type FullObjectInfo added in v1.50.4 

type FullObjectInfo interface {
	ObjectInfo
	MimeTyper
	IDer
	ObjectUnWrapper
	GetTierer
}

FullObjectInfo contains all the read-only optional interfaces

Use for checking making wrapping ObjectInfos implement everything

type GetTierer 

type GetTierer interface {
	// GetTier returns storage tier or class of the Object
	GetTier() string
}

GetTierer is an optional interface for Object

type HTTPOption 

type HTTPOption struct {
	Key   string
	Value string
}

HTTPOption defines a general purpose HTTP option

func (*HTTPOption) Header 

func (o *HTTPOption) Header() (key string, value string)

Header formats the option as an http header

func (*HTTPOption) Mandatory 

func (o *HTTPOption) Mandatory() bool

Mandatory returns whether the option must be parsed or can be ignored

func (*HTTPOption) String 

func (o *HTTPOption) String() string

String formats the option into human-readable form

type HashesOption 

type HashesOption struct {
	Hashes hash.Set
}

HashesOption defines an option used to tell the local fs to limit the number of hashes it calculates.

func (*HashesOption) Header 

func (o *HashesOption) Header() (key string, value string)

Header formats the option as an http header

func (*HashesOption) Mandatory 

func (o *HashesOption) Mandatory() bool

Mandatory returns whether the option must be parsed or can be ignored

func (*HashesOption) String 

func (o *HashesOption) String() string

String formats the option into human-readable form

type IDer 

type IDer interface {
	// ID returns the ID of the Object if known, or "" if not
	ID() string
}

IDer is an optional interface for Object

type Info 

type Info interface {
	// Name of the remote (as passed into NewFs)
	Name() string

	// Root of the remote (as passed into NewFs)
	Root() string

	// String returns a description of the FS
	String() string

	// Precision of the ModTimes in this Fs
	Precision() time.Duration

	// Returns the supported hash types of the filesystem
	Hashes() hash.Set

	// Features returns the optional features of this Fs
	Features() *Features
}

Info provides a read only interface to information about a filesystem.

type ListRCallback 

type ListRCallback func(entries DirEntries) error

ListRCallback defines a callback function for ListR to use

It is called for each tranche of entries read from the listing and if it returns an error, the listing stops.

type ListRFn 

type ListRFn func(ctx context.Context, dir string, callback ListRCallback) error

ListRFn is defines the call used to recursively list a directory

type ListRer 

type ListRer interface {
	// ListR lists the objects and directories of the Fs starting
	// from dir recursively into out.
	//
	// dir should be "" to start from the root, and should not
	// have trailing slashes.
	//
	// This should return ErrDirNotFound if the directory isn't
	// found.
	//
	// It should call callback for each tranche of entries read.
	// These need not be returned in any particular order.  If
	// callback returns an error then the listing will stop
	// immediately.
	//
	// Don't implement this unless you have a more efficient way
	// of listing recursively that doing a directory traversal.
	ListR(ctx context.Context, dir string, callback ListRCallback) error
}

ListRer is an optional interfaces for Fs

type LogLevel 

type LogLevel byte

LogLevel describes rclone's logs. These are a subset of the syslog log levels. 

const (
	LogLevelEmergency LogLevel = iota
	LogLevelAlert
	LogLevelCritical
	LogLevelError // Error - can't be suppressed
	LogLevelWarning
	LogLevelNotice // Normal logging, -q suppresses
	LogLevelInfo   // Transfers, needs -v
	LogLevelDebug  // Debug level, needs -vv
)

Log levels. These are the syslog levels of which we only use a subset. 

LOG_EMERG      system is unusable
LOG_ALERT      action must be taken immediately
LOG_CRIT       critical conditions
LOG_ERR        error conditions
LOG_WARNING    warning conditions
LOG_NOTICE     normal, but significant, condition
LOG_INFO       informational message
LOG_DEBUG      debug-level message

func (*LogLevel) Set 

func (l *LogLevel) Set(s string) error

Set a LogLevel

func (LogLevel) String 

func (l LogLevel) String() string

String turns a LogLevel into a string

func (*LogLevel) Type 

func (l *LogLevel) Type() string

Type of the value

func (*LogLevel) UnmarshalJSON added in v1.55.0 

func (l *LogLevel) UnmarshalJSON(in []byte) error

UnmarshalJSON makes sure the value can be parsed as a string or integer in JSON

type LogValueItem added in v1.50.20 

type LogValueItem struct {
	// contains filtered or unexported fields
}

LogValueItem describes keyed item for a JSON log entry

func LogValue added in v1.50.20 

func LogValue(key string, value interface{}) LogValueItem

LogValue should be used as an argument to any logging calls to augment the JSON output with more structured information.

key is the dictionary parameter used to store value.

func LogValueHide added in v1.55.0 

func LogValueHide(key string, value interface{}) LogValueItem

LogValueHide should be used as an argument to any logging calls to augment the JSON output with more structured information.

key is the dictionary parameter used to store value.

String() will return a blank string - this is useful to put items in which don't print into the log.

func (LogValueItem) String added in v1.50.20 

func (j LogValueItem) String() string

String returns the representation of value. If render is fals this is an empty string so LogValueItem entries won't show in the textual representation of logs.

type MergeDirser 

type MergeDirser interface {
	// MergeDirs merges the contents of all the directories passed
	// in into the first one and rmdirs the other directories.
	MergeDirs(ctx context.Context, dirs []Directory) error
}

MergeDirser is an option interface for Fs

type MimeTyper 

type MimeTyper interface {
	// MimeType returns the content type of the Object if
	// known, or "" if not
	MimeType(ctx context.Context) string
}

MimeTyper is an optional interface for Object

type Mover 

type Mover interface {
	// Move src to this remote using server-side move operations.
	//
	// This is stored with the remote path given
	//
	// It returns the destination Object and a possible error
	//
	// Will only be called if src.Fs().Name() == f.Name()
	//
	// If it isn't possible then return fs.ErrorCantMove
	Move(ctx context.Context, src Object, remote string) (Object, error)
}

Mover is an optional interface for Fs

type NullOption added in v1.50.20 

type NullOption struct {
}

NullOption defines an Option which does nothing

func (NullOption) Header added in v1.50.20 

func (o NullOption) Header() (key string, value string)

Header formats the option as an http header

func (NullOption) Mandatory added in v1.50.20 

func (o NullOption) Mandatory() bool

Mandatory returns whether the option must be parsed or can be ignored

func (NullOption) String added in v1.50.20 

func (o NullOption) String() string

String formats the option into human-readable form

type Object 

type Object interface {
	ObjectInfo

	// SetModTime sets the metadata on the object to set the modification date
	SetModTime(ctx context.Context, t time.Time) error

	// Open opens the file for read.  Call Close() on the returned io.ReadCloser
	Open(ctx context.Context, options ...OpenOption) (io.ReadCloser, error)

	// Update in to the object with the modTime given of the given size
	//
	// When called from outside an Fs by rclone, src.Size() will always be >= 0.
	// But for unknown-sized objects (indicated by src.Size() == -1), Upload should either
	// return an error or update the object properly (rather than e.g. calling panic).
	Update(ctx context.Context, in io.Reader, src ObjectInfo, options ...OpenOption) error

	// Removes this object
	Remove(ctx context.Context) error
}

Object is a filesystem like object provided by an Fs

func UnWrapObject added in v1.6.1 

func UnWrapObject(o Object) Object

UnWrapObject unwraps o as much as possible and returns the base object

func UnWrapObjectInfo added in v1.55.0 

func UnWrapObjectInfo(oi ObjectInfo) Object

UnWrapObjectInfo returns the underlying Object unwrapped as much as possible or nil.

type ObjectInfo 

type ObjectInfo interface {
	DirEntry

	// Fs returns read only access to the Fs that this object is part of
	Fs() Info

	// Hash returns the selected checksum of the file
	// If no checksum is available it returns ""
	Hash(ctx context.Context, ty hash.Type) (string, error)

	// Storable says whether this object can be stored
	Storable() bool
}

ObjectInfo provides read only information about an object.

type ObjectPair 

type ObjectPair struct {
	Src, Dst Object
}

ObjectPair is a pair of Objects used to describe a potential copy operation.

type ObjectUnWrapper 

type ObjectUnWrapper interface {
	// UnWrap returns the Object that this Object is wrapping or
	// nil if it isn't wrapping anything
	UnWrap() Object
}

ObjectUnWrapper is an optional interface for Object

type Objects 

type Objects []Object

Objects is a slice of Object~s

type ObjectsChan 

type ObjectsChan chan Object

ObjectsChan is a channel of Objects

type OpenOption 

type OpenOption interface {
	fmt.Stringer

	// Header returns the option as an HTTP header
	Header() (key string, value string)

	// Mandatory returns whether this option can be ignored or not
	Mandatory() bool
}

OpenOption is an interface describing options for Open

type OpenWriterAter added in v1.6.1 

type OpenWriterAter interface {
	// OpenWriterAt opens with a handle for random access writes
	//
	// Pass in the remote desired and the size if known.
	//
	// It truncates any existing object
	OpenWriterAt(ctx context.Context, remote string, size int64) (WriterAtCloser, error)
}

OpenWriterAter is an optional interface for Fs

type Option 

type Option struct {
	Name       string           // name of the option in snake_case
	Help       string           // help, start with a single sentence on a single line that will be extracted for command line help
	Provider   string           // set to filter on provider
	Default    interface{}      // default value, nil => "", if set (and not to nil or "") then Required does nothing
	Value      interface{}      // value to be set by flags
	Examples   OptionExamples   `json:",omitempty"` // predefined values that can be selected from list (multiple-choice option)
	ShortOpt   string           // the short option for this if required
	Hide       OptionVisibility // set this to hide the config from the configurator or the command line
	Required   bool             // this option is required, meaning value cannot be empty unless there is a default
	IsPassword bool             // set if the option is a password
	NoPrefix   bool             // set if the option for this should not use the backend prefix
	Advanced   bool             // set if this is an advanced config option
	Exclusive  bool             // set if the answer can only be one of the examples (empty string allowed unless Required or Default is set)
}

Option is describes an option for the config wizard

This also describes command line options and environment variables

To create a multiple-choice option, specify the possible values in the Examples property. Whether the option's value is required to be one of these depends on other properties:

  • Default is to allow any value, either from specified examples, or any other value. To restrict exclusively to the specified examples, also set Exclusive=true.
  • If empty string should not be allowed then set Required=true, and do not set Default.

func (*Option) Copy added in v1.56.0 

func (o *Option) Copy() *Option

Copy makes a shallow copy of the option

func (*Option) EnvVarName 

func (o *Option) EnvVarName(prefix string) string

EnvVarName for the option

func (*Option) FlagName 

func (o *Option) FlagName(prefix string) string

FlagName for the option

func (*Option) GetValue 

func (o *Option) GetValue() interface{}

GetValue gets the current current value which is the default if not set

func (*Option) MarshalJSON added in v1.6.1 

func (o *Option) MarshalJSON() ([]byte, error)

MarshalJSON turns an Option into JSON

It adds some generated fields for ease of use - DefaultStr - a string rendering of Default - ValueStr - a string rendering of Value - Type - the type of the option

func (*Option) Set 

func (o *Option) Set(s string) (err error)

Set an Option from a string

func (*Option) String 

func (o *Option) String() string

String turns Option into a string

func (*Option) Type 

func (o *Option) Type() string

Type of the value

type OptionExample 

type OptionExample struct {
	Value    string
	Help     string
	Provider string
}

OptionExample describes an example for an Option

type OptionExamples 

type OptionExamples []OptionExample

OptionExamples is a slice of examples

func (OptionExamples) Len 

func (os OptionExamples) Len() int

Len is part of sort.Interface.

func (OptionExamples) Less 

func (os OptionExamples) Less(i, j int) bool

Less is part of sort.Interface.

func (OptionExamples) Sort 

func (os OptionExamples) Sort()

Sort sorts an OptionExamples

func (OptionExamples) Swap 

func (os OptionExamples) Swap(i, j int)

Swap is part of sort.Interface.

type OptionVisibility 

type OptionVisibility byte

OptionVisibility controls whether the options are visible in the configurator or the command line. 

const (
	OptionHideCommandLine OptionVisibility = 1 << iota
	OptionHideConfigurator
	OptionHideBoth = OptionHideCommandLine | OptionHideConfigurator
)

Constants Option.Hide

type Options 

type Options []Option

Options is a slice of configuration Option for a backend

func (Options) Get added in v1.6.1 

func (os Options) Get(name string) *Option

Get the Option corresponding to name or return nil if not found

func (Options) HasAdvanced added in v1.56.0 

func (os Options) HasAdvanced() bool

HasAdvanced discovers if any options have an Advanced setting

func (Options) NonDefault added in v1.56.0 

func (os Options) NonDefault(m configmap.Getter) configmap.Simple

NonDefault discovers which config values aren't at their default

func (Options) Overridden added in v1.56.0 

func (os Options) Overridden(m *configmap.Map) configmap.Simple

Overridden discovers which config items have been overridden in the configmap passed in, either by the config string, command line flags or environment variables

type Pacer 

type Pacer struct {
	*pacer.Pacer
}

Pacer is a simple wrapper around a pacer.Pacer with logging.

func NewPacer 

func NewPacer(ctx context.Context, c pacer.Calculator) *Pacer

NewPacer creates a Pacer for the given Fs and Calculator.

func (*Pacer) ModifyCalculator 

func (p *Pacer) ModifyCalculator(f func(pacer.Calculator))

ModifyCalculator calls the given function with the currently configured Calculator and the Pacer lock held.

func (*Pacer) SetCalculator 

func (p *Pacer) SetCalculator(c pacer.Calculator)

SetCalculator sets the pacing algorithm. Don't modify the Calculator object afterwards, use the ModifyCalculator method when needed.

It will choose the default algorithm if nil is passed in.

type ParentIDer added in v1.55.0 

type ParentIDer interface {
	// ParentID returns the ID of the parent directory if known or nil if not
	ParentID() string
}

ParentIDer is an optional interface for Object

type PublicLinker 

type PublicLinker interface {
	// PublicLink generates a public link to the remote path (usually readable by anyone)
	PublicLink(ctx context.Context, remote string, expire Duration, unlink bool) (string, error)
}

PublicLinker is an optional interface for Fs

type Purger 

type Purger interface {
	// Purge all files in the directory specified
	//
	// Implement this if you have a way of deleting all the files
	// quicker than just running Remove() on the result of List()
	//
	// Return an error if it doesn't exist
	Purge(ctx context.Context, dir string) error
}

Purger is an optional interfaces for Fs

type PutStreamer 

type PutStreamer interface {
	// PutStream uploads to the remote path with the modTime given of indeterminate size
	//
	// May create the object even if it returns an error - if so
	// will return the object and the error, otherwise will return
	// nil and the error
	PutStream(ctx context.Context, in io.Reader, src ObjectInfo, options ...OpenOption) (Object, error)
}

PutStreamer is an optional interface for Fs

type PutUncheckeder 

type PutUncheckeder interface {
	// Put in to the remote path with the modTime given of the given size
	//
	// May create the object even if it returns an error - if so
	// will return the object and the error, otherwise will return
	// nil and the error
	//
	// May create duplicates or return errors if src already
	// exists.
	PutUnchecked(ctx context.Context, in io.Reader, src ObjectInfo, options ...OpenOption) (Object, error)
}

PutUncheckeder is an optional interface for Fs

type RangeOption 

type RangeOption struct {
	Start int64
	End   int64
}

RangeOption defines an HTTP Range option with start and end. If either start or end are < 0 then they will be omitted.

End may be bigger than the Size of the object in which case it will be capped to the size of the object.

Note that the End is inclusive, so to fetch 100 bytes you would use RangeOption{Start: 0, End: 99}

If Start is specified but End is not then it will fetch from Start to the end of the file.

If End is specified, but Start is not then it will fetch the last End bytes.

Examples: 

RangeOption{Start: 0, End: 99} - fetch the first 100 bytes
RangeOption{Start: 100, End: 199} - fetch the second 100 bytes
RangeOption{Start: 100, End: -1} - fetch bytes from offset 100 to the end
RangeOption{Start: -1, End: 100} - fetch the last 100 bytes

A RangeOption implements a single byte-range-spec from https://tools.ietf.org/html/rfc7233#section-2.1

func ParseRangeOption 

func ParseRangeOption(s string) (po *RangeOption, err error)

ParseRangeOption parses a RangeOption from a Range: header. It only accepts single ranges.

func (*RangeOption) Decode 

func (o *RangeOption) Decode(size int64) (offset, limit int64)

Decode interprets the RangeOption into an offset and a limit

The offset is the start of the stream and the limit is how many bytes should be read from it. If the limit is -1 then the stream should be read to the end.

func (*RangeOption) Header 

func (o *RangeOption) Header() (key string, value string)

Header formats the option as an http header

func (*RangeOption) Mandatory 

func (o *RangeOption) Mandatory() bool

Mandatory returns whether the option must be parsed or can be ignored

func (*RangeOption) String 

func (o *RangeOption) String() string

String formats the option into human-readable form

type RangeSeeker 

type RangeSeeker interface {
	// RangeSeek behaves like a call to Seek(offset int64, whence
	// int) with the output wrapped in an io.LimitedReader
	// limiting the total length to limit.
	//
	// RangeSeek with a limit of < 0 is equivalent to a regular Seek.
	RangeSeek(ctx context.Context, offset int64, whence int, length int64) (int64, error)
}

RangeSeeker is the interface that wraps the RangeSeek method.

Some of the returns from Object.Open() may optionally implement this method for efficiency purposes.

type RegInfo 

type RegInfo struct {
	// Name of this fs
	Name string
	// Description of this fs - defaults to Name
	Description string
	// Prefix for command line flags for this fs - defaults to Name if not set
	Prefix string
	// Create a new file system.  If root refers to an existing
	// object, then it should return an Fs which which points to
	// the parent of that object and ErrorIsFile.
	NewFs func(ctx context.Context, name string, root string, config configmap.Mapper) (Fs, error) `json:"-"`
	// Function to call to help with config - see docs for ConfigIn for more info
	Config func(ctx context.Context, name string, m configmap.Mapper, configIn ConfigIn) (*ConfigOut, error) `json:"-"`
	// Options for the Fs configuration
	Options Options
	// The command help, if any
	CommandHelp []CommandHelp
	// Aliases - other names this backend is known by
	Aliases []string
	// Hide - if set don't show in the configurator
	Hide bool
}

RegInfo provides information about a filesystem

func ConfigFs 

func ConfigFs(path string) (fsInfo *RegInfo, configName, fsPath string, config *configmap.Map, err error)

ConfigFs makes the config for calling NewFs with.

It parses the path which is of the form remote:path

Remotes are looked up in the config file. If the remote isn't found then NotFoundInConfigFile will be returned.

func Find 

func Find(name string) (*RegInfo, error)

Find looks for a RegInfo object for the name passed in. The name can be either the Name or the Prefix.

Services are looked up in the config file

func MustFind 

func MustFind(name string) *RegInfo

MustFind looks for an Info object for the type name passed in

Services are looked up in the config file

Exits with a fatal error if not found

func ParseRemote 

func ParseRemote(path string) (fsInfo *RegInfo, configName, fsPath string, connectionStringConfig configmap.Simple, err error)

ParseRemote deconstructs a path into configName, fsPath, looking up the fsName in the config file (returning NotFoundInConfigFile if not found)

func (*RegInfo) FileName 

func (ri *RegInfo) FileName() string

FileName returns the on disk file name for this backend

type SeekOption 

type SeekOption struct {
	Offset int64
}

SeekOption defines an HTTP Range option with start only.

func (*SeekOption) Header 

func (o *SeekOption) Header() (key string, value string)

Header formats the option as an http header

func (*SeekOption) Mandatory 

func (o *SeekOption) Mandatory() bool

Mandatory returns whether the option must be parsed or can be ignored

func (*SeekOption) String 

func (o *SeekOption) String() string

String formats the option into human-readable form

type SetTierer 

type SetTierer interface {
	// SetTier performs changing storage tier of the Object if
	// multiple storage classes supported
	SetTier(tier string) error
}

SetTierer is an optional interface for Object

type Shutdowner added in v1.55.0 

type Shutdowner interface {
	// Shutdown the backend, closing any background tasks and any
	// cached connections.
	Shutdown(ctx context.Context) error
}

Shutdowner is an interface to wrap the Shutdown function

type SizeSuffix 

type SizeSuffix int64

SizeSuffix is an int64 with a friendly way of printing setting 

const (
	SizeSuffixBase SizeSuffix = 1 << (iota * 10)
	Kibi
	Mebi
	Gibi
	Tebi
	Pebi
	Exbi
)

Common multipliers for SizeSuffix

func (SizeSuffix) BitRateUnit added in v1.56.0 

func (x SizeSuffix) BitRateUnit() string

BitRateUnit turns SizeSuffix into a string with bit rate unit

func (SizeSuffix) BitUnit added in v1.56.0 

func (x SizeSuffix) BitUnit() string

BitUnit turns SizeSuffix into a string with bit unit

func (SizeSuffix) ByteRateUnit added in v1.56.0 

func (x SizeSuffix) ByteRateUnit() string

ByteRateUnit turns SizeSuffix into a string with byte rate unit

func (SizeSuffix) ByteUnit added in v1.56.0 

func (x SizeSuffix) ByteUnit() string

ByteUnit turns SizeSuffix into a string with byte unit

func (*SizeSuffix) Scan 

func (x *SizeSuffix) Scan(s fmt.ScanState, ch rune) error

Scan implements the fmt.Scanner interface

func (*SizeSuffix) Set 

func (x *SizeSuffix) Set(s string) error

Set a SizeSuffix

func (SizeSuffix) String 

func (x SizeSuffix) String() string

String turns SizeSuffix into a string

func (*SizeSuffix) Type 

func (x *SizeSuffix) Type() string

Type of the value

func (*SizeSuffix) UnmarshalJSON added in v1.55.0 

func (x *SizeSuffix) UnmarshalJSON(in []byte) error

UnmarshalJSON makes sure the value can be parsed as a string or integer in JSON

type SizeSuffixList 

type SizeSuffixList []SizeSuffix

SizeSuffixList is a slice SizeSuffix values

func (SizeSuffixList) Len 

func (l SizeSuffixList) Len() int

func (SizeSuffixList) Less 

func (l SizeSuffixList) Less(i, j int) bool

func (SizeSuffixList) Sort 

func (l SizeSuffixList) Sort()

Sort sorts the list

func (SizeSuffixList) Swap 

func (l SizeSuffixList) Swap(i, j int)

type SpaceSepList 

type SpaceSepList []string

SpaceSepList is a space separated config value It uses the encoding/csv rules for quoting and escaping

Example 
for _, s := range []string{
	`remotea:test/dir remoteb:`,
	`"remotea:test/space dir" remoteb:`,
	`"remotea:test/quote""dir" remoteb:`,
} {
	var l SpaceSepList
	must(l.Set(s))
	fmt.Printf("%#v\n", l)
}

Output:

fs.SpaceSepList{"remotea:test/dir", "remoteb:"}
fs.SpaceSepList{"remotea:test/space dir", "remoteb:"}
fs.SpaceSepList{"remotea:test/quote\"dir", "remoteb:"}

func (*SpaceSepList) Scan 

func (l *SpaceSepList) Scan(s fmt.ScanState, ch rune) error

Scan implements the fmt.Scanner interface

func (*SpaceSepList) Set 

func (l *SpaceSepList) Set(s string) error

Set the List entries

func (SpaceSepList) String 

func (l SpaceSepList) String() string

func (SpaceSepList) Type 

func (SpaceSepList) Type() string

Type of the value

type Tristate added in v1.57.1 

type Tristate struct {
	Value bool
	Valid bool
}

Tristate is a boolean that can has the states, true, false and unset/invalid/nil

func (*Tristate) Scan added in v1.57.1 

func (t *Tristate) Scan(s fmt.ScanState, ch rune) error

Scan implements the fmt.Scanner interface

func (*Tristate) Set added in v1.57.1 

func (t *Tristate) Set(s string) error

Set the List entries

func (Tristate) String added in v1.57.1 

func (t Tristate) String() string

String renders the tristate as true/false/unset

func (Tristate) Type added in v1.57.1 

func (Tristate) Type() string

Type of the value

func (*Tristate) UnmarshalJSON added in v1.57.1 

func (t *Tristate) UnmarshalJSON(in []byte) error

UnmarshalJSON parses it as a bool or nil for unset

type UnWrapper 

type UnWrapper interface {
	// UnWrap returns the Fs that this Fs is wrapping
	UnWrap() Fs
}

UnWrapper is an optional interfaces for Fs

type Usage 

type Usage struct {
	Total   *int64 `json:"total,omitempty"`   // quota of bytes that can be used
	Used    *int64 `json:"used,omitempty"`    // bytes in use
	Trashed *int64 `json:"trashed,omitempty"` // bytes in trash
	Other   *int64 `json:"other,omitempty"`   // other usage e.g. gmail in drive
	Free    *int64 `json:"free,omitempty"`    // bytes which can be uploaded before reaching the quota
	Objects *int64 `json:"objects,omitempty"` // objects in the storage system
}

Usage is returned by the About call

If a value is nil then it isn't supported by that backend

type UserInfoer added in v1.6.1 

type UserInfoer interface {
	// UserInfo returns info about the connected user
	UserInfo(ctx context.Context) (map[string]string, error)
}

UserInfoer is an optional interface for Fs

type Wrapper 

type Wrapper interface {
	// Wrap returns the Fs that is wrapping this Fs
	WrapFs() Fs
	// SetWrapper sets the Fs that is wrapping this Fs
	SetWrapper(f Fs)
}

Wrapper is an optional interfaces for Fs

type WriterAtCloser added in v1.6.1 

type WriterAtCloser interface {
	io.WriterAt
	io.Closer
}

WriterAtCloser wraps io.WriterAt and io.Closer

Source Files

View all Source files

Directories

Path Synopsis
Package accounting providers an accounting and limiting reader
 Package accounting providers an accounting and limiting reader
Package asyncreader provides an asynchronous reader which reads independently of write
 Package asyncreader provides an asynchronous reader which reads independently of write
Package cache implements the Fs cache
 Package cache implements the Fs cache
Package config reads, writes and edits the config file and deals with command line flags
 Package config reads, writes and edits the config file and deals with command line flags
configfile
Package configfile implements a config file loader and saver
 Package configfile implements a config file loader and saver
configflags
Package configflags defines the flags used by rclone.
 Package configflags defines the flags used by rclone.
configmap
Package configmap provides an abstraction for reading and writing config
 Package configmap provides an abstraction for reading and writing config
configstruct
Package configstruct parses unstructured maps into structures
 Package configstruct parses unstructured maps into structures
flags
Package flags contains enhanced versions of spf13/pflag flag routines which will read from the environment also.
 Package flags contains enhanced versions of spf13/pflag flag routines which will read from the environment also.
obscure
Package obscure contains the Obscure and Reveal commands
 Package obscure contains the Obscure and Reveal commands
Package dirtree contains the DirTree type which is used for building filesystem heirachies in memory.
 Package dirtree contains the DirTree type which is used for building filesystem heirachies in memory.
Package filter controls the filtering of files
 Package filter controls the filtering of files
filterflags
Package filterflags implements command line flags to set up a filter
 Package filterflags implements command line flags to set up a filter
Package fserrors provides errors and error handling
 Package fserrors provides errors and error handling
Package fshttp contains the common http parts of the config, Transport and Client
 Package fshttp contains the common http parts of the config, Transport and Client
Package fspath contains routines for fspath manipulation
 Package fspath contains routines for fspath manipulation
Package list contains list functions
 Package list contains list functions
log
Package log provides logging for rclone
 Package log provides logging for rclone
logflags
Package logflags implements command line flags to set up the log
 Package logflags implements command line flags to set up the log
Package march traverses two directories in lock step
 Package march traverses two directories in lock step
Package object defines some useful Objects
 Package object defines some useful Objects
Package operations does generic operations on filesystems and objects
 Package operations does generic operations on filesystems and objects
rc
Package rc implements a remote control server and registry for rclone To register your internal calls, call rc.Add(path, function).
 Package rc implements a remote control server and registry for rclone To register your internal calls, call rc.Add(path, function).
jobs
js
rcflags
Package rcflags implements command line flags to set up the remote control
 Package rcflags implements command line flags to set up the remote control
rcserver
Package rcserver implements the HTTP endpoint to serve the remote control
 Package rcserver implements the HTTP endpoint to serve the remote control
webgui
Package sync is the implementation of sync/copy/move
 Package sync is the implementation of sync/copy/move
Package walk walks directories
 Package walk walks directories

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.