accounting

package
v1.68.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 15, 2024 License: MIT Imports: 21 Imported by: 81

Documentation

Overview

Package accounting providers an accounting and limiting reader

Index

Constants

This section is empty.

Variables

View Source
var ErrorMaxTransferLimitReached = errors.New("max transfer limit reached as set by --max-transfer")

ErrorMaxTransferLimitReached defines error when transfer limit is reached. Used for checking on exit and matching to correct exit code.

View Source
var ErrorMaxTransferLimitReachedFatal = fserrors.FatalError(ErrorMaxTransferLimitReached)

ErrorMaxTransferLimitReachedFatal is returned from Read when the max transfer limit is reached.

View Source
var ErrorMaxTransferLimitReachedGraceful = fserrors.NoRetryError(ErrorMaxTransferLimitReached)

ErrorMaxTransferLimitReachedGraceful is returned from operations.Copy when the max transfer limit is reached and a graceful stop is required.

View Source
var MaxCompletedTransfers = 100

MaxCompletedTransfers specifies maximum number of completed transfers in startedTransfers list

View Source
var TokenBucket tokenBucket

TokenBucket holds the global token bucket limiter

Functions

func LimitTPS added in v1.54.0

func LimitTPS(ctx context.Context)

LimitTPS limits the number of transactions per second if enabled. It should be called once per transaction.

func Start added in v1.55.0

func Start(ctx context.Context)

Start sets up the accounting, in particular the bandwidth limiting

func StartLimitTPS added in v1.54.0

func StartLimitTPS(ctx context.Context)

StartLimitTPS starts the token bucket for transactions per second limiting if necessary

func StatsGroupFromContext

func StatsGroupFromContext(ctx context.Context) (string, bool)

StatsGroupFromContext returns group from the context if it's available. Returns false if group is empty.

func WithStatsGroup

func WithStatsGroup(parent context.Context, group string) context.Context

WithStatsGroup returns copy of the parent context with assigned group.

Types

type Account

type Account struct {
	// contains filtered or unexported fields
}

Account limits and accounts for one transfer

func UnWrapAccounting added in v1.64.0

func UnWrapAccounting(in io.Reader) (unwrapped io.Reader, acc *Account)

UnWrapAccounting unwraps a reader returning unwrapped and acc a pointer to the accounting.

The caller is expected to manage the accounting at this point.

func (*Account) Abandon added in v1.53.0

func (acc *Account) Abandon()

Abandon stops the async buffer doing any more buffering

func (*Account) AccountRead

func (acc *Account) AccountRead(n int) (err error)

AccountRead account having read n bytes

func (*Account) Close

func (acc *Account) Close() error

Close the object

func (*Account) Done added in v1.49.4

func (acc *Account) Done()

Done with accounting - must be called to free accounting goroutine

func (*Account) DryRun added in v1.54.0

func (acc *Account) DryRun(n int64)

DryRun accounts for statistics without running the operation

func (*Account) GetAsyncReader

func (acc *Account) GetAsyncReader() *asyncreader.AsyncReader

GetAsyncReader returns the current AsyncReader or nil if Account is unbuffered

func (*Account) GetReader

func (acc *Account) GetReader() io.ReadCloser

GetReader returns the underlying io.ReadCloser under any Buffer

func (*Account) HasBuffer added in v1.53.0

func (acc *Account) HasBuffer() bool

HasBuffer - returns true if this Account has an AsyncReader with a buffer

func (*Account) OldStream

func (acc *Account) OldStream() io.Reader

OldStream returns the top io.Reader

func (*Account) Read

func (acc *Account) Read(p []byte) (n int, err error)

Read bytes from the object - see io.Reader

func (*Account) ServerSideCopyEnd added in v1.50.0

func (acc *Account) ServerSideCopyEnd(n int64)

ServerSideCopyEnd accounts for a read of n bytes in a sever side copy

func (*Account) ServerSideMoveEnd added in v1.64.0

func (acc *Account) ServerSideMoveEnd(n int64)

ServerSideMoveEnd accounts for a read of n bytes in a sever side move

func (*Account) ServerSideTransferEnd added in v1.64.0

func (acc *Account) ServerSideTransferEnd(n int64)

ServerSideTransferEnd accounts for a read of n bytes in a sever side transfer to be treated as a normal transfer.

func (*Account) ServerSideTransferStart added in v1.64.0

func (acc *Account) ServerSideTransferStart()

ServerSideTransferStart should be called at the start of a server-side transfer

This pretends a transfer has started

func (*Account) SetStream

func (acc *Account) SetStream(in io.Reader)

SetStream updates the top io.Reader

func (*Account) StopBuffering

func (acc *Account) StopBuffering()

StopBuffering stops the async buffer doing any more buffering

func (*Account) String

func (acc *Account) String() string

String produces stats for this file

func (*Account) UpdateReader

func (acc *Account) UpdateReader(ctx context.Context, in io.ReadCloser)

UpdateReader updates the underlying io.ReadCloser stopping the async buffer (if any) and re-adding it

func (*Account) WithBuffer

func (acc *Account) WithBuffer() *Account

WithBuffer - If the file is above a certain size it adds an Async reader

func (*Account) WrapStream

func (acc *Account) WrapStream(in io.Reader) io.Reader

WrapStream wraps an io Reader so it will be accounted in the same way as account

func (*Account) WriteTo added in v1.52.0

func (acc *Account) WriteTo(w io.Writer) (n int64, err error)

WriteTo writes data to w until there's no more data to write or when an error occurs. The return value n is the number of bytes written. Any error encountered during the write is also returned.

type Accounter

type Accounter interface {
	io.Reader
	OldStream() io.Reader
	SetStream(io.Reader)
	WrapStream(io.Reader) io.Reader
}

Accounter accounts a stream allowing the accounting to be removed and re-added

type RcloneCollector added in v1.52.0

type RcloneCollector struct {
	// contains filtered or unexported fields
}

RcloneCollector is a Prometheus collector for Rclone

func NewRcloneCollector added in v1.52.0

func NewRcloneCollector(ctx context.Context) *RcloneCollector

NewRcloneCollector make a new RcloneCollector

func (*RcloneCollector) Collect added in v1.52.0

func (c *RcloneCollector) Collect(ch chan<- prometheus.Metric)

Collect is part of the Collector interface: https://godoc.org/github.com/prometheus/client_golang/prometheus#Collector

func (*RcloneCollector) Describe added in v1.52.0

func (c *RcloneCollector) Describe(ch chan<- *prometheus.Desc)

Describe is part of the Collector interface: https://godoc.org/github.com/prometheus/client_golang/prometheus#Collector

type StatsInfo

type StatsInfo struct {
	// contains filtered or unexported fields
}

StatsInfo accounts all transfers N.B.: if this struct is modified, please remember to also update sum() function in stats_groups to correctly count the updated fields

func GlobalStats

func GlobalStats() *StatsInfo

GlobalStats returns special stats used for global accounting.

func NewStats

func NewStats(ctx context.Context) *StatsInfo

NewStats creates an initialised StatsInfo

func NewStatsGroup

func NewStatsGroup(ctx context.Context, group string) *StatsInfo

NewStatsGroup creates new stats under named group.

func Stats

func Stats(ctx context.Context) *StatsInfo

Stats gets stats by extracting group from context.

func StatsGroup

func StatsGroup(ctx context.Context, group string) *StatsInfo

StatsGroup gets stats by group name.

func (*StatsInfo) AddServerSideCopy added in v1.64.0

func (s *StatsInfo) AddServerSideCopy(n int64)

AddServerSideCopy counts a server side copy

func (*StatsInfo) AddServerSideMove added in v1.64.0

func (s *StatsInfo) AddServerSideMove(n int64)

AddServerSideMove counts a server side move

func (*StatsInfo) AddTransfer

func (s *StatsInfo) AddTransfer(transfer *Transfer)

AddTransfer adds reference to the started transfer.

func (*StatsInfo) Bytes

func (s *StatsInfo) Bytes(bytes int64)

Bytes updates the stats for bytes bytes

func (*StatsInfo) BytesNoNetwork added in v1.65.1

func (s *StatsInfo) BytesNoNetwork(bytes int64)

BytesNoNetwork updates the stats for bytes bytes but doesn't include the transfer stats

func (*StatsInfo) DeleteFile added in v1.62.0

func (s *StatsInfo) DeleteFile(ctx context.Context, size int64) error

DeleteFile updates the stats for deleting a file

It may return fatal errors if the threshold for --max-delete or --max-delete-size have been reached.

func (*StatsInfo) DeletedDirs added in v1.54.0

func (s *StatsInfo) DeletedDirs(deletedDirs int64) int64

DeletedDirs updates the stats for deletedDirs

func (*StatsInfo) DoneChecking

func (s *StatsInfo) DoneChecking(remote string)

DoneChecking removes a check from the stats

func (*StatsInfo) DoneTransferring

func (s *StatsInfo) DoneTransferring(remote string, ok bool)

DoneTransferring removes a transfer from the stats

if ok is true and it was in the transfermap (to avoid incrementing in case of nested calls, #6213) then it increments the transfers count

func (*StatsInfo) Error

func (s *StatsInfo) Error(err error) error

Error adds a single error into the stats, assigns lastError and eventually sets fatalError or retryError

func (*StatsInfo) Errored

func (s *StatsInfo) Errored() bool

Errored returns whether there have been any errors

func (*StatsInfo) Errors

func (s *StatsInfo) Errors(errors int64)

Errors updates the stats for errors

func (*StatsInfo) FatalError

func (s *StatsInfo) FatalError()

FatalError sets the fatalError flag

func (*StatsInfo) GetBytes

func (s *StatsInfo) GetBytes() int64

GetBytes returns the number of bytes transferred so far

func (*StatsInfo) GetBytesWithPending added in v1.52.0

func (s *StatsInfo) GetBytesWithPending() int64

GetBytesWithPending returns the number of bytes transferred and remaining transfers

func (*StatsInfo) GetChecks

func (s *StatsInfo) GetChecks() int64

GetChecks returns the number of checks

func (*StatsInfo) GetDeletes added in v1.62.0

func (s *StatsInfo) GetDeletes() int64

GetDeletes returns the number of deletes

func (*StatsInfo) GetErrors

func (s *StatsInfo) GetErrors() int64

GetErrors reads the number of errors

func (*StatsInfo) GetLastError

func (s *StatsInfo) GetLastError() error

GetLastError returns the lastError

func (*StatsInfo) GetTransfers

func (s *StatsInfo) GetTransfers() int64

GetTransfers reads the number of transfers

func (*StatsInfo) HadFatalError

func (s *StatsInfo) HadFatalError() bool

HadFatalError returns whether there has been at least one FatalError

func (*StatsInfo) HadRetryError

func (s *StatsInfo) HadRetryError() bool

HadRetryError returns whether there has been at least one non-NoRetryError

func (*StatsInfo) Log

func (s *StatsInfo) Log()

Log outputs the StatsInfo to the log

func (*StatsInfo) NewCheckingTransfer

func (s *StatsInfo) NewCheckingTransfer(obj fs.DirEntry, what string) *Transfer

NewCheckingTransfer adds a checking transfer to the stats, from the object.

func (*StatsInfo) NewTransfer

func (s *StatsInfo) NewTransfer(obj fs.DirEntry, dstFs fs.Fs) *Transfer

NewTransfer adds a transfer to the stats from the object.

The obj is uses as the srcFs, the dstFs must be supplied

func (*StatsInfo) NewTransferRemoteSize

func (s *StatsInfo) NewTransferRemoteSize(remote string, size int64, srcFs, dstFs fs.Fs) *Transfer

NewTransferRemoteSize adds a transfer to the stats based on remote and size.

func (*StatsInfo) PruneTransfers added in v1.50.0

func (s *StatsInfo) PruneTransfers()

PruneTransfers makes sure there aren't too many old transfers by removing single finished transfer.

func (*StatsInfo) RemoteStats

func (s *StatsInfo) RemoteStats() (out rc.Params, err error)

RemoteStats returns stats for rc

func (*StatsInfo) RemoveTransfer added in v1.50.0

func (s *StatsInfo) RemoveTransfer(transfer *Transfer)

RemoveTransfer removes a reference to the started transfer.

func (*StatsInfo) Renames added in v1.52.0

func (s *StatsInfo) Renames(renames int64) int64

Renames updates the stats for renames

func (*StatsInfo) ResetCounters

func (s *StatsInfo) ResetCounters()

ResetCounters sets the counters (bytes, checks, errors, transfers, deletes, renames) to 0 and resets lastError, fatalError and retryError

func (*StatsInfo) ResetErrors

func (s *StatsInfo) ResetErrors()

ResetErrors sets the errors count to 0 and resets lastError, fatalError and retryError

func (*StatsInfo) RetryAfter

func (s *StatsInfo) RetryAfter() time.Time

RetryAfter returns the time to retry after if it is set. It will be Zero if it isn't set.

func (*StatsInfo) RetryError

func (s *StatsInfo) RetryError()

RetryError sets the retryError flag

func (*StatsInfo) SetCheckQueue

func (s *StatsInfo) SetCheckQueue(n int, size int64)

SetCheckQueue sets the number of queued checks

func (*StatsInfo) SetRenameQueue

func (s *StatsInfo) SetRenameQueue(n int, size int64)

SetRenameQueue sets the number of queued transfers

func (*StatsInfo) SetTransferQueue

func (s *StatsInfo) SetTransferQueue(n int, size int64)

SetTransferQueue sets the number of queued transfers

func (*StatsInfo) String

func (s *StatsInfo) String() string

String convert the StatsInfo to a string for printing

func (*StatsInfo) Transferred

func (s *StatsInfo) Transferred() []TransferSnapshot

Transferred returns list of all completed transfers including checked and failed ones.

type TokenBucketSlot added in v1.54.0

type TokenBucketSlot int

TokenBucketSlot is the type to select which token bucket to use

const (
	TokenBucketSlotAccounting TokenBucketSlot = iota
	TokenBucketSlotTransportRx
	TokenBucketSlotTransportTx
	TokenBucketSlots
)

Slots for the token bucket

type Transfer

type Transfer struct {
	// contains filtered or unexported fields
}

Transfer keeps track of initiated transfers and provides access to accounting functions. Transfer needs to be closed on completion.

func (*Transfer) Account

func (tr *Transfer) Account(ctx context.Context, in io.ReadCloser) *Account

Account returns reader that knows how to keep track of transfer progress.

func (*Transfer) Done

func (tr *Transfer) Done(ctx context.Context, err error)

Done ends the transfer. Must be called after transfer is finished to run proper cleanups.

func (*Transfer) IsDone

func (tr *Transfer) IsDone() bool

IsDone returns true if transfer is completed.

func (*Transfer) Reset added in v1.50.0

func (tr *Transfer) Reset(ctx context.Context)

Reset allows to switch the Account to another transfer method.

func (*Transfer) Snapshot

func (tr *Transfer) Snapshot() TransferSnapshot

Snapshot produces stats for this account at point in time.

func (*Transfer) TimeRange

func (tr *Transfer) TimeRange() (time.Time, time.Time)

TimeRange returns the time transfer started and ended at. If not completed it will return zero time for end time.

type TransferSnapshot

type TransferSnapshot struct {
	Name        string    `json:"name"`
	Size        int64     `json:"size"`
	Bytes       int64     `json:"bytes"`
	Checked     bool      `json:"checked"`
	StartedAt   time.Time `json:"started_at"`
	CompletedAt time.Time `json:"completed_at,omitempty"`
	Error       error     `json:"-"`
	Group       string    `json:"group"`
	SrcFs       string    `json:"srcFs,omitempty"`
	DstFs       string    `json:"dstFs,omitempty"`
}

TransferSnapshot represents state of an account at point in time.

func (TransferSnapshot) MarshalJSON

func (as TransferSnapshot) MarshalJSON() ([]byte, error)

MarshalJSON implements json.Marshaler interface.

type WrapFn

type WrapFn func(io.Reader) io.Reader

WrapFn wraps an io.Reader (for accounting purposes usually)

func UnWrap

func UnWrap(in io.Reader) (unwrapped io.Reader, wrap WrapFn)

UnWrap unwraps a reader returning unwrapped and wrap, a function to wrap it back up again. If `in` is an Accounter then this function will take the accounting unwrapped and wrap will put it back on again the new Reader passed in.

This allows functions which wrap io.Readers to move the accounting to the end of the wrapped chain of readers. This is very important if buffering is being introduced and if the Reader might be wrapped again.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL