fs

package
v0.0.0-...-e7704eb Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 2, 2019 License: MIT Imports: 28 Imported by: 0

Documentation

Overview

Package fs is a generic file system interface for rclone object storage systems

Index

Constants

View Source 
const (
	// UserAgent for Fs which can set it
	UserAgent = "rclone/" + Version
	// ModTimeNotSupported is a very large precision value to show
	// mod time isn't supported on this Fs
	ModTimeNotSupported = 100 * 365 * 24 * time.Hour
)

Constants

View Source 
const Version = "v1.26"

Version of rclone

Variables

View Source 
var (
	// ConfigFile is the config file data structure
	ConfigFile *goconfig.ConfigFile
	// HomeDir is the home directory of the user
	HomeDir = configHome()
	// ConfigPath points to the config file
	ConfigPath = path.Join(HomeDir, configFileName)
	// Config is the global config
	Config = &ConfigInfo{}
)

Global

View Source 
var (

	// ErrorNotFoundInConfigFile is returned by NewFs if not found in config file
	ErrorNotFoundInConfigFile = fmt.Errorf("Didn't find section in config file")
	ErrorCantPurge            = fmt.Errorf("Can't purge directory")
	ErrorCantCopy             = fmt.Errorf("Can't copy object - incompatible remotes")
	ErrorCantMove             = fmt.Errorf("Can't copy object - incompatible remotes")
	ErrorCantDirMove          = fmt.Errorf("Can't copy directory - incompatible remotes")
	ErrorDirExists            = fmt.Errorf("Can't copy directory - destination already exists")
)

Globals

View Source 
var (
	Stats = NewStats()
)

Globals

Functions

func CalculateModifyWindow 

func CalculateModifyWindow(fs ...Fs)

CalculateModifyWindow works out modify window for Fses passed in - sets Config.ModifyWindow

This is the largest modify window of all the fses in use, and the user configured value

func Check 

func Check(fdst, fsrc Fs) error

Check the files in fsrc and fdst according to Size and MD5SUM

func CheckClose 

func CheckClose(c io.Closer, err *error)

CheckClose is a utility function used to check the return from Close in a defer statement.

func CheckMd5sums 

func CheckMd5sums(src, dst Object) (equal bool, unset bool, err error)

CheckMd5sums checks the two files to see if the MD5sums are the same

Returns two bools, the first of which is equality and the second of which is true if either of the MD5SUMs were unset.

May return an error which will already have been logged

If an error is returned it will return equal as false

func Choose 

func Choose(what string, defaults, help []string, newOk bool) string

Choose one of the defaults or type a new string if newOk is set

func ChooseOption 

func ChooseOption(o *Option) string

ChooseOption asks the user to choose an option

func ChooseRemote 

func ChooseRemote() string

ChooseRemote chooses a remote name

func Command 

func Command(commands []string) byte

Command - choose one

func Confirm 

func Confirm() bool

Confirm asks the user for Yes or No and returns true or false

func Copy 

func Copy(f Fs, dst, src Object)

Copy src object to dst or f if nil

If dst is nil then the object must not exist already. If you do call Copy() with dst nil on a pre-existing file then some filing systems (eg Drive) may duplicate the file.

func CopyDir 

func CopyDir(fdst, fsrc Fs) error

CopyDir copies fsrc into fdst

func Count 

func Count(f Fs) (objects int64, size int64, err error)

Count counts the objects and their sizes in the Fs

Obeys includes and excludes

func Debug 

func Debug(o interface{}, text string, args ...interface{})

Debug writes debuging output for this Object or Fs

func DeleteFiles 

func DeleteFiles(toBeDeleted ObjectsChan)

DeleteFiles removes all the files passed in the channel

func DeleteRemote 

func DeleteRemote(name string)

DeleteRemote gets the user to delete a remote

func EditConfig 

func EditConfig()

EditConfig edits the config file interactively

func EditRemote 

func EditRemote(name string)

EditRemote gets the user to edit a remote

func Equal 

func Equal(src, dst Object) bool

Equal checks to see if the src and dst objects are equal by looking at size, mtime and MD5SUM

If the src and dst size are different then it is considered to be not equal. If --size-only is in effect then this is the only check that is done.

If the size is the same and the mtime is the same then it is considered to be equal. This check is skipped if using --checksum.

If the size is the same and mtime is different, unreadable or --checksum is set and the MD5SUM is the same then the file is considered to be equal. In this case the mtime on the dst is updated if --checksum is not set.

Otherwise the file is considered to be not equal including if there were errors reading info.

func ErrorLog 

func ErrorLog(o interface{}, text string, args ...interface{})

ErrorLog writes error log output for this Object or Fs. It unconditionally logs a message regardless of Config.Quiet or Config.Verbose.

func List 

func List(f Fs, w io.Writer) error

List the Fs to the supplied writer

Shows size and path - obeys includes and excludes

Lists in parallel which may get them out of order

func ListDir 

func ListDir(f Fs, w io.Writer) error

ListDir lists the directories/buckets/containers in the Fs to the supplied writer

func ListFn 

func ListFn(f Fs, fn func(Object)) error

ListFn lists the Fs to the supplied function

Lists in parallel which may get them out of order

func ListLong 

func ListLong(f Fs, w io.Writer) error

ListLong lists the Fs to the supplied writer

Shows size, mod time and path - obeys includes and excludes

Lists in parallel which may get them out of order

func LoadConfig 

func LoadConfig()

LoadConfig loads the config file

func Log 

func Log(o interface{}, text string, args ...interface{})

Log writes log output for this Object or Fs

func Md5sum 

func Md5sum(f Fs, w io.Writer) error

Md5sum list the Fs to the supplied writer

Produces the same output as the md5sum command - obeys includes and excludes

Lists in parallel which may get them out of order

func Md5sumsEqual 

func Md5sumsEqual(src, dst string) bool

Md5sumsEqual checks to see if src == dst, but ignores empty strings

func MimeType 

func MimeType(o Object) string

MimeType returns a guess at the mime type from the extension

func Mkdir 

func Mkdir(f Fs) error

Mkdir makes a destination directory or container

func MoveDir 

func MoveDir(fdst, fsrc Fs) error

MoveDir moves fsrc into fdst

func NewRemote 

func NewRemote(name string)

NewRemote make a new remote from its name

func Obscure 

func Obscure(x string) string

Obscure a config value

func OkRemote 

func OkRemote(name string) bool

OkRemote prints the contents of the remote and ask if it is OK

func OutputLog 

func OutputLog(o interface{}, text string, args ...interface{})

OutputLog logs for an object

func PairChecker 

func PairChecker(in ObjectPairChan, out ObjectPairChan, wg *sync.WaitGroup)

PairChecker reads Objects~s on in send to out if they need transferring.

FIXME potentially doing lots of MD5SUMS at once

func PairCopier 

func PairCopier(in ObjectPairChan, fdst Fs, wg *sync.WaitGroup)

PairCopier reads Objects on in and copies them.

func PairMover 

func PairMover(in ObjectPairChan, fdst Fs, wg *sync.WaitGroup)

PairMover reads Objects on in and moves them if possible, or copies them if not

func ParseDuration 

func ParseDuration(age string) (time.Duration, error)

ParseDuration parses a duration string. Accept ms|s|m|h|d|w|M|y suffixes. Defaults to second if not provided

func Purge 

func Purge(f Fs) error

Purge removes a container and all of its contents

FIXME doesn't delete local directories

func ReadLine 

func ReadLine() string

ReadLine reads some input

func Register 

func Register(info *Info)

Register a filesystem

Fs modules should use this in an init() function

func RemoteConfig 

func RemoteConfig(name string)

RemoteConfig runs the config helper for the remote if needed

func RetryError 

func RetryError(err error) error

RetryError makes an error which indicates it would like to be retried

func RetryErrorf 

func RetryErrorf(format string, a ...interface{}) error

RetryErrorf makes an error which indicates it would like to be retried

func Reveal 

func Reveal(y string) string

Reveal a config value

func Rmdir 

func Rmdir(f Fs) error

Rmdir removes a container but not if not empty

func Same 

func Same(fdst, fsrc Fs) bool

Same returns true if fdst and fsrc point to the same underlying Fs

func SaveConfig 

func SaveConfig()

SaveConfig saves configuration file.

func ShouldRetry 

func ShouldRetry(err error) bool

ShouldRetry looks at an error and tries to work out if retrying the operation that caused it would be a good idea. It returns true if the error implements Timeout() or Temporary() and it returns true.

func ShouldRetryHTTP 

func ShouldRetryHTTP(resp *http.Response, retryErrorCodes []int) bool

ShouldRetryHTTP returns a boolean as to whether this resp deserves. It checks to see if the HTTP response code is in the slice retryErrorCodes.

func ShowRemote 

func ShowRemote(name string)

ShowRemote shows the contents of the remote

func ShowRemotes 

func ShowRemotes()

ShowRemotes shows an overview of the config file

func Sync 

func Sync(fdst, fsrc Fs) error

Sync fsrc into fdst

Types

type Account 

type Account struct {
	// contains filtered or unexported fields
}

Account limits and accounts for one transfer

func NewAccount 

func NewAccount(in io.ReadCloser, obj Object) *Account

NewAccount makes a Account reader for an object

func (*Account) Close 

func (file *Account) Close() error

Close the object

func (*Account) ETA 

func (file *Account) ETA() (eta time.Duration, ok bool)

ETA returns the ETA of the current operation, rounded to full seconds. If the ETA cannot be determined 'ok' returns false.

func (*Account) Progress 

func (file *Account) Progress() (bytes, size int64)

Progress returns bytes read as well as the size. Size can be <= 0 if the size is unknown.

func (*Account) Read 

func (file *Account) Read(p []byte) (n int, err error)

Read bytes from the object - see io.Reader

func (*Account) Speed 

func (file *Account) Speed() (bps, current float64)

Speed returns the speed of the current file transfer in bytes per second, as well a an exponentially weighted moving average If no read has completed yet, 0 is returned for both values.

func (*Account) String 

func (file *Account) String() string

String produces stats for this file

type ConfigInfo 

type ConfigInfo struct {
	Verbose            bool
	Quiet              bool
	DryRun             bool
	CheckSum           bool
	SizeOnly           bool
	ModifyWindow       time.Duration
	Checkers           int
	Transfers          int
	ConnectTimeout     time.Duration // Connect timeout
	Timeout            time.Duration // Data channel timeout
	DumpHeaders        bool
	DumpBodies         bool
	Filter             *Filter
	InsecureSkipVerify bool // Skip server certificate verification
}

ConfigInfo is filesystem config options

func (*ConfigInfo) Client 

func (ci *ConfigInfo) Client() *http.Client

Client returns an http.Client with the correct timeouts

func (*ConfigInfo) Transport 

func (ci *ConfigInfo) Transport() http.RoundTripper

Transport returns an http.RoundTripper with the correct timeouts

type Copier 

type Copier interface {
	// Copy src to this remote using server side copy operations.
	//
	// This is stored with the remote path given
	//
	// It returns the destination Object and a possible error
	//
	// Will only be called if src.Fs().Name() == f.Name()
	//
	// If it isn't possible then return fs.ErrorCantCopy
	Copy(src Object, remote string) (Object, error)
}

Copier is an optional interface for Fs

type Dir 

type Dir struct {
	Name  string    // name of the directory
	When  time.Time // modification or creation time - IsZero for unknown
	Bytes int64     // size of directory and contents -1 for unknown
	Count int64     // number of objects -1 for unknown
}

Dir describes a directory for directory/container/bucket lists

type DirChan 

type DirChan chan *Dir

DirChan is a channel of Dir objects

type DirMover 

type DirMover interface {
	// DirMove moves src to this remote using server side move
	// operations.
	//
	// Will only be called if src.Fs().Name() == f.Name()
	//
	// If it isn't possible then return fs.ErrorCantDirMove
	//
	// If destination exists then return fs.ErrorDirExists
	DirMove(src Fs) error
}

DirMover is an optional interface for Fs

type Filter 

type Filter struct {
	DeleteExcluded bool
	MinSize        int64
	MaxSize        int64
	ModTimeFrom    time.Time
	ModTimeTo      time.Time
	// contains filtered or unexported fields
}

Filter describes any filtering in operation

func NewFilter 

func NewFilter() (f *Filter, err error)

NewFilter parses the command line options and creates a Filter object

func (*Filter) Add 

func (f *Filter) Add(Include bool, glob string) error

Add adds a filter rule with include or exclude status indicated

func (*Filter) AddFile 

func (f *Filter) AddFile(file string) error

AddFile adds a single file to the files from list

func (*Filter) AddRule 

func (f *Filter) AddRule(rule string) error

AddRule adds a filter rule with include/exclude indicated by the prefix

These are

  • glob
  • glob !

'+' includes the glob, '-' excludes it and '!' resets the filter list

Line comments may be introduced with '#' or ';'

func (*Filter) Clear 

func (f *Filter) Clear()

Clear clears all the filter rules

func (*Filter) DumpFilters 

func (f *Filter) DumpFilters() string

DumpFilters dumps the filters in textual form, 1 per line

func (*Filter) Include 

func (f *Filter) Include(remote string, size int64, modTime time.Time) bool

Include returns whether this object should be included into the sync or not

func (*Filter) IncludeObject 

func (f *Filter) IncludeObject(o Object) bool

IncludeObject returns whether this object should be included into the sync or not. This is a convenience function to avoid calling o.ModTime(), which is an expensive operation.

type Fs 

type Fs interface {
	// Name of the remote (as passed into NewFs)
	Name() string

	// Root of the remote (as passed into NewFs)
	Root() string

	// String returns a description of the FS
	String() string

	// List the Fs into a channel
	List() ObjectsChan

	// ListDir lists the Fs directories/buckets/containers into a channel
	ListDir() DirChan

	// NewFsObject finds the Object at remote.  Returns nil if can't be found
	NewFsObject(remote string) Object

	// Put in to the remote path with the modTime given of the given size
	//
	// May create the object even if it returns an error - if so
	// will return the object and the error, otherwise will return
	// nil and the error
	Put(in io.Reader, remote string, modTime time.Time, size int64) (Object, error)

	// Mkdir makes the directory (container, bucket)
	//
	// Shouldn't return an error if it already exists
	Mkdir() error

	// Rmdir removes the directory (container, bucket) if empty
	//
	// Return an error if it doesn't exist or isn't empty
	Rmdir() error

	// Precision of the ModTimes in this Fs
	Precision() time.Duration
}

Fs is the interface a cloud storage system must provide

func NewFs 

func NewFs(path string) (Fs, error)

NewFs makes a new Fs object from the path

The path is of the form remote:path

Remotes are looked up in the config file. If the remote isn't found then NotFoundInConfigFile will be returned.

On Windows avoid single character remote names as they can be mixed up with drive letters.

func NewLimited 

func NewLimited(fs Fs, objects ...Object) Fs

NewLimited maks a limited Fs limited to the objects passed in

type Info 

type Info struct {
	// Name of this fs
	Name string
	// Create a new file system.  If root refers to an existing
	// object, then it should return a Fs which only returns that
	// object.
	NewFs func(name string, root string) (Fs, error)
	// Function to call to help with config
	Config func(string)
	// Options for the Fs configuration
	Options []Option
}

Info information about a filesystem

func Find 

func Find(name string) (*Info, error)

Find looks for an Info object for the name passed in

Services are looked up in the config file

type Limited 

type Limited struct {
	// contains filtered or unexported fields
}

Limited defines a Fs which can only return the Objects passed in from the Fs passed in

func (*Limited) Copy 

func (f *Limited) Copy(src Object, remote string) (Object, error)

Copy src to this remote using server side copy operations.

This is stored with the remote path given

It returns the destination Object and a possible error

Will only be called if src.Fs().Name() == f.Name()

If it isn't possible then return fs.ErrorCantCopy

func (*Limited) List 

func (f *Limited) List() ObjectsChan

List the Fs into a channel

func (*Limited) ListDir 

func (f *Limited) ListDir() DirChan

ListDir lists the Fs directories/buckets/containers into a channel

func (*Limited) Mkdir 

func (f *Limited) Mkdir() error

Mkdir make the directory (container, bucket)

func (*Limited) Move 

func (f *Limited) Move(src Object, remote string) (Object, error)

Move src to this remote using server side move operations.

This is stored with the remote path given

It returns the destination Object and a possible error

Will only be called if src.Fs().Name() == f.Name()

If it isn't possible then return fs.ErrorCantMove

func (*Limited) Name 

func (f *Limited) Name() string

Name is name of the remote (as passed into NewFs)

func (*Limited) NewFsObject 

func (f *Limited) NewFsObject(remote string) Object

NewFsObject finds the Object at remote. Returns nil if can't be found

func (*Limited) Precision 

func (f *Limited) Precision() time.Duration

Precision of the ModTimes in this Fs

func (*Limited) Put 

func (f *Limited) Put(in io.Reader, remote string, modTime time.Time, size int64) (Object, error)

Put in to the remote path with the modTime given of the given size

May create the object even if it returns an error - if so will return the object and the error, otherwise will return nil and the error

func (*Limited) Rmdir 

func (f *Limited) Rmdir() error

Rmdir removes the directory (container, bucket) if empty

func (*Limited) Root 

func (f *Limited) Root() string

Root is the root of the remote (as passed into NewFs)

func (*Limited) String 

func (f *Limited) String() string

String returns a description of the FS

func (*Limited) UnWrap 

func (f *Limited) UnWrap() Fs

UnWrap returns the Fs that this Fs is wrapping

type LoggedTransport 

type LoggedTransport struct {
	// contains filtered or unexported fields
}

LoggedTransport is an http transport which logs the traffic

func NewLoggedTransport 

func NewLoggedTransport(transport http.RoundTripper, logBody bool) *LoggedTransport

NewLoggedTransport wraps the transport passed in and logs all roundtrips including the body if logBody is set.

func (*LoggedTransport) CancelRequest 

func (t *LoggedTransport) CancelRequest(req *http.Request)

CancelRequest cancels an in-flight request by closing its connection. CancelRequest should only be called after RoundTrip has returned.

func (*LoggedTransport) RoundTrip 

func (t *LoggedTransport) RoundTrip(req *http.Request) (resp *http.Response, err error)

RoundTrip implements the RoundTripper interface.

type Mover 

type Mover interface {
	// Move src to this remote using server side move operations.
	//
	// This is stored with the remote path given
	//
	// It returns the destination Object and a possible error
	//
	// Will only be called if src.Fs().Name() == f.Name()
	//
	// If it isn't possible then return fs.ErrorCantMove
	Move(src Object, remote string) (Object, error)
}

Mover is an optional interface for Fs

type Object 

type Object interface {
	// String returns a description of the Object
	String() string

	// Fs returns the Fs that this object is part of
	Fs() Fs

	// Remote returns the remote path
	Remote() string

	// Md5sum returns the md5 checksum of the file
	// If no Md5sum is available it returns ""
	Md5sum() (string, error)

	// ModTime returns the modification date of the file
	// It should return a best guess if one isn't available
	ModTime() time.Time

	// SetModTime sets the metadata on the object to set the modification date
	SetModTime(time.Time)

	// Size returns the size of the file
	Size() int64

	// Open opens the file for read.  Call Close() on the returned io.ReadCloser
	Open() (io.ReadCloser, error)

	// Update in to the object with the modTime given of the given size
	Update(in io.Reader, modTime time.Time, size int64) error

	// Storable says whether this object can be stored
	Storable() bool

	// Removes this object
	Remove() error
}

Object is a filesystem like object provided by an Fs

type ObjectPair 

type ObjectPair struct {
	// contains filtered or unexported fields
}

ObjectPair is a pair of Objects used to describe a potential copy operation.

type ObjectPairChan 

type ObjectPairChan chan ObjectPair

ObjectPairChan is a channel of ObjectPair

type Objects 

type Objects []Object

Objects is a slice of Object~s

type ObjectsChan 

type ObjectsChan chan Object

ObjectsChan is a channel of Objects

type Option 

type Option struct {
	Name     string
	Help     string
	Optional bool
	Examples []OptionExample
}

Option is describes an option for the config wizard

type OptionExample 

type OptionExample struct {
	Value string
	Help  string
}

OptionExample describes an example for an Option

type Purger 

type Purger interface {
	// Purge all files in the root and the root directory
	//
	// Implement this if you have a way of deleting all the files
	// quicker than just running Remove() on the result of List()
	//
	// Return an error if it doesn't exist
	Purge() error
}

Purger is an optional interfaces for Fs

type Retry 

type Retry interface {
	error
	Retry() bool
}

Retry is an optional interface for error as to whether the operation should be retried at a high level.

This should be returned from Update or Put methods as required

type SizeSuffix 

type SizeSuffix int64

SizeSuffix is parsed by flag with k/M/G suffixes

func (*SizeSuffix) Set 

func (x *SizeSuffix) Set(s string) error

Set a SizeSuffix

func (SizeSuffix) String 

func (x SizeSuffix) String() string

Turn SizeSuffix into a string

func (*SizeSuffix) Type 

func (x *SizeSuffix) Type() string

Type of the value

type StatsInfo 

type StatsInfo struct {
	// contains filtered or unexported fields
}

StatsInfo limits and accounts all transfers

func NewStats 

func NewStats() *StatsInfo

NewStats cretates an initialised StatsInfo

func (*StatsInfo) Bytes 

func (s *StatsInfo) Bytes(bytes int64)

Bytes updates the stats for bytes bytes

func (*StatsInfo) Checking 

func (s *StatsInfo) Checking(o Object)

Checking adds a check into the stats

func (*StatsInfo) DoneChecking 

func (s *StatsInfo) DoneChecking(o Object)

DoneChecking removes a check from the stats

func (*StatsInfo) DoneTransferring 

func (s *StatsInfo) DoneTransferring(o Object)

DoneTransferring removes a transfer from the stats

func (*StatsInfo) Error 

func (s *StatsInfo) Error()

Error adds a single error into the stats

func (*StatsInfo) Errored 

func (s *StatsInfo) Errored() bool

Errored returns whether there have been any errors

func (*StatsInfo) Errors 

func (s *StatsInfo) Errors(errors int64)

Errors updates the stats for errors

func (*StatsInfo) GetErrors 

func (s *StatsInfo) GetErrors() int64

GetErrors reads the number of errors

func (*StatsInfo) GetTransfers 

func (s *StatsInfo) GetTransfers() int64

GetTransfers reads the number of transfers

func (*StatsInfo) Log 

func (s *StatsInfo) Log()

Log outputs the StatsInfo to the log

func (*StatsInfo) ResetCounters 

func (s *StatsInfo) ResetCounters()

ResetCounters sets the counters (bytes, checks, errors, transfers) to 0

func (*StatsInfo) ResetErrors 

func (s *StatsInfo) ResetErrors()

ResetErrors sets the errors count to 0

func (*StatsInfo) String 

func (s *StatsInfo) String() string

String convert the StatsInfo to a string for printing

func (*StatsInfo) Transferring 

func (s *StatsInfo) Transferring(o Object)

Transferring adds a transfer into the stats

type UnWrapper 

type UnWrapper interface {
	// UnWrap returns the Fs that this Fs is wrapping
	UnWrap() Fs
}

UnWrapper is an optional interfaces for Fs

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.