operations

package
v1.67.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 14, 2024 License: MIT Imports: 49 Imported by: 208

Documentation

Overview

Package operations does generic operations on filesystems and objects

Index

Constants

This section is empty.

Variables

View Source 
var StdoutMutex sync.Mutex

StdoutMutex mutex for synchronized output on stdout

View Source 
var SyncPrintf = func(format string, a ...interface{}) {
	StdoutMutex.Lock()
	defer StdoutMutex.Unlock()
	fmt.Printf(format, a...)
}

SyncPrintf is a global var holding the Printf function so that it can be overridden.

This writes to stdout holding the StdoutMutex. If you are going to override it and write to os.Stdout then you should hold the StdoutMutex too.

Functions

func ApplyTransforms added in v1.66.0 

func ApplyTransforms(ctx context.Context, s string) string

ApplyTransforms handles --no-unicode-normalization and --ignore-case-sync for CheckSum so that it matches behavior of Check (where it's handled by March)

func BackupDir 

func BackupDir(ctx context.Context, fdst fs.Fs, fsrc fs.Fs, srcFileName string) (backupDir fs.Fs, err error)

BackupDir returns the correctly configured --backup-dir

func CanServerSideMove 

func CanServerSideMove(fdst fs.Fs) bool

CanServerSideMove returns true if fdst support server-side moves or server-side copies

Some remotes simulate rename by server-side copy and delete, so include remotes that implements either Mover or Copier.

func Cat 

func Cat(ctx context.Context, f fs.Fs, w io.Writer, offset, count int64, sep []byte) error

Cat any files to the io.Writer

if offset == 0 it will be ignored if offset > 0 then the file will be seeked to that offset if offset < 0 then the file will be seeked that far from the end

if count < 0 then it will be ignored if count >= 0 then only that many characters will be output

func Check 

func Check(ctx context.Context, opt *CheckOpt) error

Check the files in fsrc and fdst according to Size and hash

func CheckDownload 

func CheckDownload(ctx context.Context, opt *CheckOpt) error

CheckDownload checks the files in fsrc and fdst according to Size and the actual contents of the files.

func CheckEqualReaders 

func CheckEqualReaders(in1, in2 io.Reader) (differ bool, err error)

CheckEqualReaders checks to see if in1 and in2 have the same content when read.

it returns true if differences were found

func CheckFn 

func CheckFn(ctx context.Context, opt *CheckOpt) error

CheckFn checks the files in fsrc and fdst according to Size and hash using checkFunction on each file to check the hashes.

checkFunction sees if dst and src are identical

it returns true if differences were found it also returns whether it couldn't be hashed

func CheckHashes 

func CheckHashes(ctx context.Context, src fs.ObjectInfo, dst fs.Object) (equal bool, ht hash.Type, err error)

CheckHashes checks the two files to see if they have common known hash types and compares them

Returns.

equal - which is equality of the hashes

hash - the HashType. This is HashNone if either of the hashes were unset or a compatible hash couldn't be found.

err - may return an error which will already have been logged

If an error is returned it will return equal as false

func CheckIdenticalDownload added in v1.53.0 

func CheckIdenticalDownload(ctx context.Context, dst, src fs.Object) (differ bool, err error)

CheckIdenticalDownload checks to see if dst and src are identical by reading all their bytes if necessary.

it returns true if differences were found

func CheckSum added in v1.56.0 

func CheckSum(ctx context.Context, fsrc, fsum fs.Fs, sumFile string, hashType hash.Type, opt *CheckOpt, download bool) error

CheckSum checks filesystem hashes against a SUM file

func CleanUp 

func CleanUp(ctx context.Context, f fs.Fs) error

CleanUp removes the trash for the Fs

func CommonHash added in v1.52.0 

func CommonHash(ctx context.Context, fa, fb fs.Info) (hash.Type, *fs.HashesOption)

CommonHash returns a single hash.Type and a HashOption with that type which is in common between the two fs.Fs.

func CompareOrCopyDest 

func CompareOrCopyDest(ctx context.Context, fdst fs.Fs, dst, src fs.Object, CompareOrCopyDest []fs.Fs, backupDir fs.Fs) (NoNeedTransfer bool, err error)

CompareOrCopyDest checks --compare-dest and --copy-dest to see if src does not need to be copied

Returns True if src does not need to be copied

func ConfigMaxDepth 

func ConfigMaxDepth(ctx context.Context, recursive bool) int

ConfigMaxDepth returns the depth to use for a recursive or non recursive listing.

func Copy 

func Copy(ctx context.Context, f fs.Fs, dst fs.Object, remote string, src fs.Object) (newDst fs.Object, err error)

Copy src object to dst or f if nil. If dst is nil then it uses remote as the name of the new object.

It returns the destination object if possible. Note that this may be nil.

func CopyDirMetadata added in v1.66.0 

func CopyDirMetadata(ctx context.Context, f fs.Fs, dst fs.Directory, dir string, src fs.Directory) (newDst fs.Directory, err error)

CopyDirMetadata copies the src directory to dst or f if nil. If dst is nil then it uses dir as the name of the new directory.

It returns the destination directory if possible. Note that this may be nil.

func CopyFile 

func CopyFile(ctx context.Context, fdst fs.Fs, fsrc fs.Fs, dstFileName string, srcFileName string) (err error)

CopyFile moves a single file possibly to a new name

func CopyURL 

func CopyURL(ctx context.Context, fdst fs.Fs, dstFileName string, url string, autoFilename, dstFileNameFromHeader bool, noClobber bool) (dst fs.Object, err error)

CopyURL copies the data from the url to (fdst, dstFileName)

func CopyURLToWriter added in v1.51.0 

func CopyURLToWriter(ctx context.Context, url string, out io.Writer) (err error)

CopyURLToWriter copies the data from the url to the io.Writer supplied

func Count 

func Count(ctx context.Context, f fs.Fs) (objects int64, size int64, sizelessObjects int64, err error)

Count counts the objects and their sizes in the Fs

Obeys includes and excludes

func CountString added in v1.57.0 

func CountString(count int64, humanReadable bool) string

CountString make string representation of count for output

Optional human-readable format including a decimal suffix

func CountStringField added in v1.57.0 

func CountStringField(count int64, humanReadable bool, rawWidth int) string

CountStringField make string representation of count for output in fixed width field

Similar to SizeStringField, but human readable with decimal prefix and field width 8 since there is no 'i' in the decimal prefix symbols (e.g. "999.999E")

func Deduplicate 

func Deduplicate(ctx context.Context, f fs.Fs, mode DeduplicateMode, byHash bool) error

Deduplicate interactively finds duplicate files and offers to delete all but one or rename them to be different. Only useful with Google Drive which can have duplicate file names.

func Delete 

func Delete(ctx context.Context, f fs.Fs) error

Delete removes all the contents of a container. Unlike Purge, it obeys includes and excludes.

func DeleteFile 

func DeleteFile(ctx context.Context, dst fs.Object) (err error)

DeleteFile deletes a single file respecting --dry-run and accumulating stats and errors.

If useBackupDir is set and --backup-dir is in effect then it moves the file to there instead of deleting

func DeleteFileWithBackupDir 

func DeleteFileWithBackupDir(ctx context.Context, dst fs.Object, backupDir fs.Fs) (err error)

DeleteFileWithBackupDir deletes a single file respecting --dry-run and accumulating stats and errors.

If backupDir is set then it moves the file to there instead of deleting

func DeleteFiles 

func DeleteFiles(ctx context.Context, toBeDeleted fs.ObjectsChan) error

DeleteFiles removes all the files passed in the channel

func DeleteFilesWithBackupDir 

func DeleteFilesWithBackupDir(ctx context.Context, toBeDeleted fs.ObjectsChan, backupDir fs.Fs) error

DeleteFilesWithBackupDir removes all the files passed in the channel

If backupDir is set the files will be placed into that directory instead of being deleted.

func DirMove 

func DirMove(ctx context.Context, f fs.Fs, srcRemote, dstRemote string) (err error)

DirMove renames srcRemote to dstRemote

It does this by loading the directory tree into memory (using ListR if available) and doing renames in parallel.

func DirMoveCaseInsensitive added in v1.66.0 

func DirMoveCaseInsensitive(ctx context.Context, f fs.Fs, srcRemote, dstRemote string) (err error)

DirMoveCaseInsensitive does DirMove in two steps (to temp name, then real name) which is necessary for some case-insensitive backends

func DirsEqual added in v1.66.0 

func DirsEqual(ctx context.Context, src, dst fs.Directory, opt DirsEqualOpt) (equal bool)

DirsEqual is like Equal but for dirs instead of objects. It returns true if two dirs should be considered "equal" for the purposes of syncCopyMove (in other words, true == "skip updating modtime/metadata for this dir".) Unlike Equal, it does not consider size or checksum, as these do not apply to directories.

func Equal 

func Equal(ctx context.Context, src fs.ObjectInfo, dst fs.Object) bool

Equal checks to see if the src and dst objects are equal by looking at size, mtime and hash

If the src and dst size are different then it is considered to be not equal. If --size-only is in effect then this is the only check that is done. If --ignore-size is in effect then this check is skipped and the files are considered the same size.

If the size is the same and the mtime is the same then it is considered to be equal. This check is skipped if using --checksum.

If the size is the same and mtime is different, unreadable or --checksum is set and the hash is the same then the file is considered to be equal. In this case the mtime on the dst is updated if --checksum is not set.

Otherwise the file is considered to be not equal including if there were errors reading info.

func FormatForLSFPrecision added in v1.66.0 

func FormatForLSFPrecision(precision time.Duration) string

FormatForLSFPrecision Returns a time format for the given precision

func GetCompareDest 

func GetCompareDest(ctx context.Context) (CompareDest []fs.Fs, err error)

GetCompareDest sets up --compare-dest

func GetCopyDest 

func GetCopyDest(ctx context.Context, fdst fs.Fs) (CopyDest []fs.Fs, err error)

GetCopyDest sets up --copy-dest

func HashLister 

func HashLister(ctx context.Context, ht hash.Type, outputBase64 bool, downloadFlag bool, f fs.Fs, w io.Writer) error

HashLister does an md5sum equivalent for the hash type passed in Updated to handle both standard hex encoding and base64 Updated to perform multiple hashes concurrently

func HashSum added in v1.66.0 

func HashSum(ctx context.Context, ht hash.Type, base64Encoded bool, downloadFlag bool, o fs.Object) (string, error)

HashSum returns the human-readable hash for ht passed in. This may be UNSUPPORTED or ERROR. If it isn't returning a valid hash it will return an error.

func HashSumStream added in v1.58.0 

func HashSumStream(ht hash.Type, outputBase64 bool, in io.ReadCloser, w io.Writer) error

HashSumStream outputs a line compatible with md5sum to w based on the input stream in and the hash type ht passed in. If outputBase64 is set then the hash will be base64 instead of hexadecimal.

func List 

func List(ctx context.Context, f fs.Fs, w io.Writer) error

List the Fs to the supplied writer

Shows size and path - obeys includes and excludes.

Lists in parallel which may get them out of order

func ListDir 

func ListDir(ctx context.Context, f fs.Fs, w io.Writer) error

ListDir lists the directories/buckets/containers in the Fs to the supplied writer

func ListFn 

func ListFn(ctx context.Context, f fs.Fs, fn func(fs.Object)) error

ListFn lists the Fs to the supplied function

Lists in parallel which may get them out of order

func ListJSON 

func ListJSON(ctx context.Context, fsrc fs.Fs, remote string, opt *ListJSONOpt, callback func(*ListJSONItem) error) error

ListJSON lists fsrc using the options in opt calling callback for each item

func ListLong 

func ListLong(ctx context.Context, f fs.Fs, w io.Writer) error

ListLong lists the Fs to the supplied writer

Shows size, mod time and path - obeys includes and excludes.

Lists in parallel which may get them out of order

func Mkdir 

func Mkdir(ctx context.Context, f fs.Fs, dir string) error

Mkdir makes a destination directory or container

func MkdirMetadata added in v1.66.0 

func MkdirMetadata(ctx context.Context, f fs.Fs, dir string, metadata fs.Metadata) (newDst fs.Directory, err error)

MkdirMetadata makes a destination directory or container with metadata

If the destination Fs doesn't support this it will fall back to Mkdir and in this case newDst will be nil.

func MkdirModTime added in v1.66.0 

func MkdirModTime(ctx context.Context, f fs.Fs, dir string, modTime time.Time) (newDst fs.Directory, err error)

MkdirModTime makes a destination directory or container with modtime

It will try to make the directory with MkdirMetadata and if that succeeds it will return a non-nil newDst. In all other cases newDst will be nil.

If the directory was created with MkDir then it will attempt to use Fs.DirSetModTime to update the directory modtime if available.

func Move 

func Move(ctx context.Context, fdst fs.Fs, dst fs.Object, remote string, src fs.Object) (newDst fs.Object, err error)

Move src object to dst or fdst if nil. If dst is nil then it uses remote as the name of the new object.

Note that you must check the destination does not exist before calling this and pass it as dst. If you pass dst=nil and the destination does exist then this may create duplicates or return errors.

It returns the destination object if possible. Note that this may be nil.

This is accounted as a check.

func MoveBackupDir 

func MoveBackupDir(ctx context.Context, backupDir fs.Fs, dst fs.Object) (err error)

MoveBackupDir moves a file to the backup dir

func MoveCaseInsensitive added in v1.67.0 

func MoveCaseInsensitive(ctx context.Context, fdst fs.Fs, fsrc fs.Fs, dstFileName string, srcFileName string, cp bool, srcObj fs.Object) (newDst fs.Object, err error)

MoveCaseInsensitive handles changing case of a file on a case insensitive remote. This will move the file to a temporary name then move it back to the intended destination. This is required to avoid issues with certain remotes and avoid file deletion. returns nil, nil if !needsMoveCaseInsensitive. this does not account a transfer -- the caller should do that if desired.

func MoveFile 

func MoveFile(ctx context.Context, fdst fs.Fs, fsrc fs.Fs, dstFileName string, srcFileName string) (err error)

MoveFile moves a single file possibly to a new name

This is treated as a transfer.

func MoveTransfer added in v1.65.1 

func MoveTransfer(ctx context.Context, fdst fs.Fs, dst fs.Object, remote string, src fs.Object) (newDst fs.Object, err error)

MoveTransfer moves src object to dst or fdst if nil. If dst is nil then it uses remote as the name of the new object.

This is identical to Move but is accounted as a transfer.

func NeedTransfer 

func NeedTransfer(ctx context.Context, dst, src fs.Object) bool

NeedTransfer checks to see if src needs to be copied to dst using the current config.

Returns a flag which indicates whether the file needs to be transferred or not.

func OverlappingFilterCheck added in v1.59.0 

func OverlappingFilterCheck(ctx context.Context, fdst fs.Fs, fsrc fs.Fs) bool

OverlappingFilterCheck returns true if fdst and fsrc point to the same underlying Fs and they overlap without fdst being excluded by any filter rule. 

func PublicLink(ctx context.Context, f fs.Fs, remote string, expire fs.Duration, unlink bool) (string, error)

PublicLink adds a "readable by anyone with link" permission on the given file or folder.

func Purge 

func Purge(ctx context.Context, f fs.Fs, dir string) (err error)

Purge removes a directory and all of its contents

func Rcat 

func Rcat(ctx context.Context, fdst fs.Fs, dstFileName string, in io.ReadCloser, modTime time.Time, meta fs.Metadata) (dst fs.Object, err error)

Rcat reads data from the Reader until EOF and uploads it to a file on remote

in is closed at the end of the transfer

func RcatSize 

func RcatSize(ctx context.Context, fdst fs.Fs, dstFileName string, in io.ReadCloser, size int64, modTime time.Time, meta fs.Metadata) (dst fs.Object, err error)

RcatSize reads data from the Reader until EOF and uploads it to a file on remote. Pass in size >=0 if known, <0 if not known

func ReadFile added in v1.67.0 

func ReadFile(ctx context.Context, o fs.Object) (b []byte, err error)

ReadFile reads the object into memory and accounts it

func Retry added in v1.53.0 

func Retry(ctx context.Context, o interface{}, maxTries int, fn func() error) (err error)

Retry runs fn up to maxTries times if it returns a retriable error

func Rmdir 

func Rmdir(ctx context.Context, f fs.Fs, dir string) error

Rmdir removes a container but not if not empty

func Rmdirs 

func Rmdirs(ctx context.Context, f fs.Fs, dir string, leaveRoot bool) error

Rmdirs removes any empty directories (or directories only containing empty directories) under f, including f.

Rmdirs obeys the filters

func Same 

func Same(fdst, fsrc fs.Info) bool

Same returns true if fdst and fsrc point to the same underlying Fs

func SameConfig 

func SameConfig(fdst, fsrc fs.Info) bool

SameConfig returns true if fdst and fsrc are using the same config file entry

func SameConfigArr added in v1.55.0 

func SameConfigArr(fdst fs.Info, fsrcs []fs.Fs) bool

SameConfigArr returns true if any of []fsrcs has same config file entry with fdst

func SameDir 

func SameDir(fdst, fsrc fs.Info) bool

SameDir returns true if fdst and fsrc point to the same underlying Fs and they are the same directory.

func SameObject 

func SameObject(src, dst fs.Object) bool

SameObject returns true if src and dst could be pointing to the same object.

func SameRemoteType 

func SameRemoteType(fdst, fsrc fs.Info) bool

SameRemoteType returns true if fdst and fsrc are the same type

func SetDirModTime added in v1.66.0 

func SetDirModTime(ctx context.Context, f fs.Fs, dst fs.Directory, dir string, modTime time.Time) (newDst fs.Directory, err error)

SetDirModTime sets the modtime on dst or dir

If dst is nil then it uses dir as the name of the directory.

It returns the destination directory if possible. Note that this may be nil.

It does not create the directory.

func SetTier 

func SetTier(ctx context.Context, fsrc fs.Fs, tier string) error

SetTier changes tier of object in remote

func SetTierFile added in v1.64.0 

func SetTierFile(ctx context.Context, o fs.Object, tier string) error

SetTierFile changes tier of a single file in remote

func SizeString added in v1.57.0 

func SizeString(size int64, humanReadable bool) string

SizeString make string representation of size for output

Optional human-readable format including a binary suffix

func SizeStringField added in v1.57.0 

func SizeStringField(size int64, humanReadable bool, rawWidth int) string

SizeStringField make string representation of size for output in fixed width field

Optional human-readable format including a binary suffix Argument rawWidth is used to format field with of raw value. When humanReadable option the width is hard coded to 9, since SizeSuffix strings have precision 3 and longest value will be "999.999Ei". This way the width can be optimized depending to the humanReadable option. To always use a longer width the return value can always be fed into another format string with a specific field with.

func SkipDestructive added in v1.53.0 

func SkipDestructive(ctx context.Context, subject interface{}, action string) (skip bool)

SkipDestructive should be called whenever rclone is about to do an destructive operation.

It will check the --dry-run flag and it will ask the user if the --interactive flag is set.

subject should be the object or directory in use

action should be a descriptive word or short phrase

Together they should make sense in this sentence: "Rclone is about to action subject".

func SuffixName 

func SuffixName(ctx context.Context, remote string) string

SuffixName adds the current --suffix to the remote, obeying --suffix-keep-extension if set

func SyncFprintf added in v1.66.0 

func SyncFprintf(w io.Writer, format string, a ...interface{})

SyncFprintf - Synchronized fmt.Fprintf

Ignores errors from Fprintf.

Prints to stdout if w is nil

func ToNormal added in v1.66.0 

func ToNormal(s string, normUnicode, normCase bool) string

ToNormal normalizes case and unicode form and returns the transformed string. It is similar to ApplyTransforms but does not use a context. If normUnicode == true, s will be transformed to NFC. If normCase == true, s will be transformed to lowercase. If both are true, both transformations will be performed.

func TouchDir added in v1.57.0 

func TouchDir(ctx context.Context, f fs.Fs, remote string, t time.Time, recursive bool) error

TouchDir touches every file in directory with time t

func TruncateString added in v1.65.0 

func TruncateString(s string, n int) string

TruncateString s to n bytes.

If s is valid UTF-8 then this may truncate to fewer than n bytes to make the returned string also valid UTF-8.

func TryRmdir 

func TryRmdir(ctx context.Context, f fs.Fs, dir string) error

TryRmdir removes a container but not if not empty. It doesn't count errors but may return one.

func WithEqualFn added in v1.66.0 

func WithEqualFn(ctx context.Context, equalFn EqualFn) context.Context

WithEqualFn stores equalFn in ctx and returns a copy of ctx in which equalFnKey = equalFn

func WithLogger added in v1.66.0 

func WithLogger(ctx context.Context, logger LoggerFn) context.Context

WithLogger stores logger in ctx and returns a copy of ctx in which loggerKey = logger

func WithLoggerOpt added in v1.66.0 

func WithLoggerOpt(ctx context.Context, loggerOpt LoggerOpt) context.Context

WithLoggerOpt stores loggerOpt in ctx and returns a copy of ctx in which loggerOptKey = loggerOpt

func WithSyncLogger added in v1.66.0 

func WithSyncLogger(ctx context.Context, opt LoggerOpt) context.Context

WithSyncLogger starts a new logger with the options passed in and saves it to ctx for retrieval later

Types

type AccountFn added in v1.65.0 

type AccountFn func(n int) error

AccountFn is a function which will be called after every read from the ReOpen.

It may return an error which will be passed back to the user.

type CheckOpt added in v1.53.0 

type CheckOpt struct {
	Fdst, Fsrc   fs.Fs     // fses to check
	Check        checkFn   // function to use for checking
	OneWay       bool      // one way only?
	Combined     io.Writer // a file with file names with leading sigils
	MissingOnSrc io.Writer // files only in the destination
	MissingOnDst io.Writer // files only in the source
	Match        io.Writer // matching files
	Differ       io.Writer // differing files
	Error        io.Writer // files with errors of some kind
}

CheckOpt contains options for the Check functions

type DeduplicateMode 

type DeduplicateMode int

DeduplicateMode is how the dedupe command chooses what to do 

const (
	DeduplicateInteractive DeduplicateMode = iota // interactively ask the user
	DeduplicateSkip                               // skip all conflicts
	DeduplicateFirst                              // choose the first object
	DeduplicateNewest                             // choose the newest object
	DeduplicateOldest                             // choose the oldest object
	DeduplicateRename                             // rename the objects
	DeduplicateLargest                            // choose the largest object
	DeduplicateSmallest                           // choose the smallest object
	DeduplicateList                               // list duplicates only
)

Deduplicate modes

func (*DeduplicateMode) Set 

func (x *DeduplicateMode) Set(s string) error

Set a DeduplicateMode from a string

func (DeduplicateMode) String 

func (x DeduplicateMode) String() string

func (*DeduplicateMode) Type 

func (x *DeduplicateMode) Type() string

Type of the value

type DirsEqualOpt added in v1.66.0 

type DirsEqualOpt struct {
	ModifyWindow   time.Duration // Max time diff to be considered the same
	SetDirModtime  bool          // whether to consider dir modtime
	SetDirMetadata bool          // whether to consider dir metadata
}

DirsEqualOpt represents options for DirsEqual function()

type EqualFn added in v1.66.0 

type EqualFn func(ctx context.Context, src fs.ObjectInfo, dst fs.Object) bool

EqualFn allows replacing Equal() with a custom function during NeedTransfer()

type FsInfo 

type FsInfo struct {
	// Name of the remote (as passed into NewFs)
	Name string

	// Root of the remote (as passed into NewFs)
	Root string

	// String returns a description of the FS
	String string

	// Precision of the ModTimes in this Fs in Nanoseconds
	Precision time.Duration

	// Returns the supported hash types of the filesystem
	Hashes []string

	// Features returns the optional features of this Fs
	Features map[string]bool

	// MetadataInfo returns info about the metadata for this backend
	MetadataInfo *fs.MetadataInfo
}

FsInfo provides information about a remote

func GetFsInfo 

func GetFsInfo(f fs.Fs) *FsInfo

GetFsInfo gets the information (FsInfo) about a given Fs

type HashSums added in v1.56.0 

type HashSums map[string]string

HashSums represents a parsed SUM file

func ParseSumFile added in v1.56.0 

func ParseSumFile(ctx context.Context, sumFile fs.Object) (HashSums, error)

ParseSumFile parses a hash SUM file and returns hashes as a map

type ListFormat 

type ListFormat struct {
	// contains filtered or unexported fields
}

ListFormat defines files information print format

func (*ListFormat) AddEncrypted 

func (l *ListFormat) AddEncrypted()

AddEncrypted adds the encrypted path to file to output

func (*ListFormat) AddHash 

func (l *ListFormat) AddHash(ht hash.Type)

AddHash adds the hash of the type given to the output

func (*ListFormat) AddID 

func (l *ListFormat) AddID()

AddID adds file's ID to the output if known

func (*ListFormat) AddMetadata added in v1.59.0 

func (l *ListFormat) AddMetadata()

AddMetadata adds file's Metadata to the output if known

func (*ListFormat) AddMimeType 

func (l *ListFormat) AddMimeType()

AddMimeType adds file's MimeType to the output if known

func (*ListFormat) AddModTime 

func (l *ListFormat) AddModTime(timeFormat string)

AddModTime adds file's Mod Time to output

func (*ListFormat) AddOrigID 

func (l *ListFormat) AddOrigID()

AddOrigID adds file's Original ID to the output if known

func (*ListFormat) AddPath 

func (l *ListFormat) AddPath()

AddPath adds path to file to output

func (*ListFormat) AddSize 

func (l *ListFormat) AddSize()

AddSize adds file's size to output

func (*ListFormat) AddTier 

func (l *ListFormat) AddTier()

AddTier adds file's Tier to the output if known

func (*ListFormat) AppendOutput 

func (l *ListFormat) AppendOutput(functionToAppend func(item *ListJSONItem) string)

AppendOutput adds string generated by specific function to printed output

func (*ListFormat) Format 

func (l *ListFormat) Format(entry *ListJSONItem) (result string)

Format prints information about the DirEntry in the format defined

func (*ListFormat) SetAbsolute 

func (l *ListFormat) SetAbsolute(absolute bool)

SetAbsolute prints a leading slash in front of path names

func (*ListFormat) SetCSV 

func (l *ListFormat) SetCSV(useCSV bool)

SetCSV defines if the output should be csv

Note that you should call SetSeparator before this if you want a custom separator

func (*ListFormat) SetDirSlash 

func (l *ListFormat) SetDirSlash(dirSlash bool)

SetDirSlash defines if slash should be printed

func (*ListFormat) SetOutput 

func (l *ListFormat) SetOutput(output []func(entry *ListJSONItem) string)

SetOutput sets functions used to create files information

func (*ListFormat) SetSeparator 

func (l *ListFormat) SetSeparator(separator string)

SetSeparator changes separator in struct

type ListJSONItem 

type ListJSONItem struct {
	Path          string
	Name          string
	EncryptedPath string `json:",omitempty"`
	Encrypted     string `json:",omitempty"`
	Size          int64
	MimeType      string    `json:",omitempty"`
	ModTime       Timestamp //`json:",omitempty"`
	IsDir         bool
	Hashes        map[string]string `json:",omitempty"`
	ID            string            `json:",omitempty"`
	OrigID        string            `json:",omitempty"`
	Tier          string            `json:",omitempty"`
	IsBucket      bool              `json:",omitempty"`
	Metadata      fs.Metadata       `json:",omitempty"`
}

ListJSONItem in the struct which gets marshalled for each line

func StatJSON added in v1.57.0 

func StatJSON(ctx context.Context, fsrc fs.Fs, remote string, opt *ListJSONOpt) (item *ListJSONItem, err error)

StatJSON returns a single JSON stat entry for the fsrc, remote path

The item returned may be nil if it is not found or excluded with DirsOnly/FilesOnly

type ListJSONOpt 

type ListJSONOpt struct {
	Recurse       bool     `json:"recurse"`
	NoModTime     bool     `json:"noModTime"`
	NoMimeType    bool     `json:"noMimeType"`
	ShowEncrypted bool     `json:"showEncrypted"`
	ShowOrigIDs   bool     `json:"showOrigIDs"`
	ShowHash      bool     `json:"showHash"`
	DirsOnly      bool     `json:"dirsOnly"`
	FilesOnly     bool     `json:"filesOnly"`
	Metadata      bool     `json:"metadata"`
	HashTypes     []string `json:"hashTypes"` // hash types to show if ShowHash is set, e.g. "MD5", "SHA-1"
}

ListJSONOpt describes the options for ListJSON

type LoggerFn added in v1.66.0 

type LoggerFn func(ctx context.Context, sigil Sigil, src, dst fs.DirEntry, err error)

LoggerFn uses fs.DirEntry instead of fs.Object so it can include Dirs For LoggerFn example, see bisync.WriteResults() or sync.SyncLoggerFn() Usage example: s.logger(ctx, operations.Differ, src, dst, nil)

func GetLogger added in v1.66.0 

func GetLogger(ctx context.Context) (LoggerFn, bool)

GetLogger attempts to retrieve LoggerFn from context, returns it if found, otherwise returns no-op function

type LoggerOpt added in v1.66.0 

type LoggerOpt struct {
	// Fdst, Fsrc   fs.Fs         // fses to check
	// Check        checkFn       // function to use for checking
	// OneWay       bool          // one way only?
	LoggerFn      LoggerFn      // function to use for logging
	Combined      io.Writer     // a file with file names with leading sigils
	MissingOnSrc  io.Writer     // files only in the destination
	MissingOnDst  io.Writer     // files only in the source
	Match         io.Writer     // matching files
	Differ        io.Writer     // differing files
	Error         io.Writer     // files with errors of some kind
	DestAfter     io.Writer     // files that exist on the destination post-sync
	JSON          *bytes.Buffer // used by bisync to read/write struct as JSON
	DeleteModeOff bool          //affects whether Logger expects MissingOnSrc to be deleted

	// lsf options for destAfter
	ListFormat ListFormat
	JSONOpt    ListJSONOpt
	LJ         *listJSON
	Format     string
	TimeFormat string
	Separator  string
	DirSlash   bool
	// Recurse   bool
	HashType  hash.Type
	FilesOnly bool
	DirsOnly  bool
	Csv       bool
	Absolute  bool
}

LoggerOpt contains options for the Sync Logger functions TODO: refactor Check in here too?

func GetLoggerOpt added in v1.66.0 

func GetLoggerOpt(ctx context.Context) LoggerOpt

GetLoggerOpt attempts to retrieve LoggerOpt from context, returns it if found, otherwise returns NewLoggerOpt()

func NewLoggerOpt added in v1.66.0 

func NewLoggerOpt() LoggerOpt

NewLoggerOpt returns a new LoggerOpt struct with defaults

func (*LoggerOpt) JSONEntry added in v1.66.0 

func (opt *LoggerOpt) JSONEntry(ctx context.Context, entry fs.DirEntry) (*ListJSONItem, error)

JSONEntry returns a *ListJSONItem for destAfter

func (*LoggerOpt) NewListJSON added in v1.66.0 

func (opt *LoggerOpt) NewListJSON(ctx context.Context, fdst fs.Fs, remote string)

NewListJSON makes a new *listJSON for destAfter

func (*LoggerOpt) PrintDestAfter added in v1.66.0 

func (opt *LoggerOpt) PrintDestAfter(ctx context.Context, sigil Sigil, src, dst fs.DirEntry, err error)

PrintDestAfter writes a *ListJSONItem to opt.DestAfter

func (*LoggerOpt) SetListFormat added in v1.66.0 

func (opt *LoggerOpt) SetListFormat(ctx context.Context, cmdFlags *pflag.FlagSet)

SetListFormat sets opt.ListFormat for destAfter TODO: possibly refactor duplicate code from cmd/lsf, where this is mostly copied from

type ReOpen added in v1.52.0 

type ReOpen struct {
	// contains filtered or unexported fields
}

ReOpen is a wrapper for an object reader which reopens the stream on error

func NewReOpen added in v1.52.0 

func NewReOpen(ctx context.Context, src fs.Object, maxTries int, options ...fs.OpenOption) (rc *ReOpen, err error)

NewReOpen makes a handle which will reopen itself and seek to where it was on errors up to maxTries times.

If an fs.HashesOption is set this will be applied when reading from the start.

If an fs.RangeOption is set then this will applied when reading from the start, and updated on retries.

func Open added in v1.63.0 

func Open(ctx context.Context, src fs.Object, options ...fs.OpenOption) (rc *ReOpen, err error)

Open makes a handle which will reopen itself and seek to where it was on errors.

If an fs.HashesOption is set this will be applied when reading from the start.

If an fs.RangeOption is set then this will applied when reading from the start, and updated on retries.

It will obey LowLevelRetries in the ctx as the maximum number of tries.

Use this instead of calling the Open method on fs.Objects

func (*ReOpen) Close added in v1.52.0 

func (h *ReOpen) Close() error

Close the stream

func (*ReOpen) DelayAccounting added in v1.65.0 

func (h *ReOpen) DelayAccounting(i int)

DelayAccounting makes sure the accounting function only gets called on the i-th or later read of the data from this point (counting from 1).

This is useful so that we don't account initial reads of the data e.g. when calculating hashes.

Set this to 0 to account everything.

func (*ReOpen) Read added in v1.52.0 

func (h *ReOpen) Read(p []byte) (n int, err error)

Read bytes retrying as necessary

func (*ReOpen) Seek added in v1.65.0 

func (h *ReOpen) Seek(offset int64, whence int) (int64, error)

Seek sets the offset for the next Read or Write to offset, interpreted according to whence: SeekStart means relative to the start of the file, SeekCurrent means relative to the current offset, and SeekEnd means relative to the end (for example, offset = -2 specifies the penultimate byte of the file). Seek returns the new offset relative to the start of the file or an error, if any.

Seeking to an offset before the start of the file is an error. Seeking to any positive offset may be allowed, but if the new offset exceeds the size of the underlying object the behavior of subsequent I/O operations is implementation-dependent.

func (*ReOpen) SetAccounting added in v1.65.0 

func (h *ReOpen) SetAccounting(account AccountFn) *ReOpen

SetAccounting should be provided with a function which will be called after every read from the RW.

It may return an error which will be passed back to the user.

type Sigil added in v1.66.0 

type Sigil rune

Sigil represents the rune (-+=*!?) used by Logger to categorize files by their match/differ/missing status. 

const (
	MissingOnSrc  Sigil = '-'
	MissingOnDst  Sigil = '+'
	Match         Sigil = '='
	Differ        Sigil = '*'
	TransferError Sigil = '!'
	Other         Sigil = '?' // reserved but not currently used
)

Sigil constants

func (Sigil) String added in v1.66.0 

func (sigil Sigil) String() string

String converts sigil to more human-readable string

func (Sigil) Writer added in v1.66.0 

func (sigil Sigil) Writer(opt LoggerOpt) io.Writer

Writer directs traffic from sigil -> LoggerOpt.Writer

type Timestamp 

type Timestamp struct {
	When   time.Time
	Format string
}

Timestamp a time in the provided format

func (Timestamp) MarshalJSON 

func (t Timestamp) MarshalJSON() (out []byte, err error)

MarshalJSON turns a Timestamp into JSON

type Winner added in v1.66.0 

type Winner struct {
	Obj  fs.DirEntry // the object that should exist on dst post-sync, if any
	Side string      // whether the winning object was from the src or dst
	Err  error       // whether there's an error preventing us from predicting winner correctly (not whether there was a sync error more generally)
}

Winner predicts which side (src or dst) should end up winning out on the dst.

func WinningSide added in v1.66.0 

func WinningSide(ctx context.Context, sigil Sigil, src, dst fs.DirEntry, err error) Winner

WinningSide can be called in a LoggerFn to predict what the dest will look like post-sync

This attempts to account for every case in which dst (intentionally) does not match src after a sync.

Known issues / cases we can't confidently predict yet: 

--max-duration / CutoffModeHard
--compare-dest / --copy-dest (because equal() is called multiple times for the same file)
server-side moves of an entire dir at once (because we never get the individual file objects in the dir)
High-level retries, because there would be dupes (use --retries 1 to disable)
Possibly some error scenarios

Source Files

View all Source files

Directories

Path Synopsis
Package operationsflags defines the flags used by rclone operations.
 Package operationsflags defines the flags used by rclone operations.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.