set

package
v1.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 17, 2024 License: MIT Imports: 21 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	ErrInvalidSetID           = "invalid set ID"
	ErrInvalidRequest         = "request lacks Requester or Set"
	ErrInvalidEntry           = "invalid set entry"
	ErrInvalidTransformerPath = "invalid transformer path concatenation"
	ErrNoAddDuringDiscovery   = "can't add set while set is being discovered"

	AttemptsToBeConsideredFailing = 3
)
View Source 
const (
	ErrInvalidTransformer = "invalid transformer"
)

Variables

This section is empty.

Functions

This section is empty.

Types

type DB 

type DB struct {
	DBRO
	// contains filtered or unexported fields
}

DB is used to create and query a database for storing backup sets (lists of files a user wants to have backed up) and their backup status.

func New 

func New(path, backupPath string) (*DB, error)

New returns a *DB that can be used to create or query a set database. Provide the path to the database file.

Optionally, also provide a path to backup the database to.

Returns an error if path exists but can't be opened, or if it doesn't exist and can't be created.

func (*DB) AddOrUpdate 

func (d *DB) AddOrUpdate(set *Set) error

AddOrUpdate adds or updates the given Set to the database.

func (*DB) Backup added in v0.7.0 

func (d *DB) Backup() error

Backup does an on-line backup of the database to the given file path. There's at least 1 second in between backups, and incoming backup requests are noted during an on-going backup, with 1 further backup occurring after the on-going one completes.

func (*DB) Close 

func (d *DB) Close() error

Close closes the database. Be sure to call this to finalise any writes to disk correctly.

func (*DB) Discover added in v0.8.0 

func (d *DB) Discover(setID string, cb DiscoverCallback) (*Set, error)

Discover discovers and stores file entry details for the given set. Immediately tries to record in the db that discovery has started and returns any error from doing that. Actual discovery using your callback on directories will then proceed, as will the stat'ing of pure files, and return the updated set.

func (*DB) EnableRemoteBackups added in v0.7.0 

func (d *DB) EnableRemoteBackups(remotePath string, handler put.Handler)

EnableRemoteBackups causes the backup file to also be backed up to the remote path.

func (*DB) GetMountPointFromPath added in v0.9.0 

func (d *DB) GetMountPointFromPath(path string) string

GetMountPointFromPath determines the mount point for the given path based on the mount points available on the system when the server started. If nothing matches, returns /.

func (*DB) HardlinkPaths added in v0.8.0 

func (d *DB) HardlinkPaths(e *Entry) ([]string, error)

HardlinkPaths returns all known hardlink paths that share the same mountpoint and inode as the entry provided.

func (*DB) HardlinkRemote added in v0.8.0 

func (d *DB) HardlinkRemote(e *Entry) (string, error)

HardlinkRemote gets the remote path of the first hardlink we uploaded that shares the given entry's inode and mountpoint.

func (*DB) SetBackupPath added in v0.7.0 

func (d *DB) SetBackupPath(path string)

SetBackupPath sets the backup path, for if you created the DB without providing one.

func (*DB) SetDirEntries 

func (d *DB) SetDirEntries(setID string, entries []*walk.Dirent) error

SetDirEntries sets the directory paths for the given backup set. Only supply absolute paths to directories.

func (*DB) SetEntryStatus 

func (d *DB) SetEntryStatus(r *put.Request) (*Entry, error)

SetEntryStatus finds the set Entry corresponding to the given Request's Local path, Requester and Set name, and updates its status in the database, and also updates summary status for the Set.

Returns the Entry, which is a reflection of the Request, but contains additional information such as the number of Attempts if the Request is failing and you keep retrying.

Returns an error if a set or entry corresponding to the Request can't be found.

func (*DB) SetError 

func (d *DB) SetError(setID, errMsg string) error

SetError updates a set with the given error message. Returns an error if the setID isn't in the database.

func (*DB) SetFileEntries 

func (d *DB) SetFileEntries(setID string, paths []string) error

SetFileEntries sets the file paths for the given backup set. Only supply absolute paths to files.

func (*DB) SetMinimumTimeBetweenBackups added in v0.7.0 

func (d *DB) SetMinimumTimeBetweenBackups(dur time.Duration)

SetMinimumTimeBetweenBackups sets the minimum time between successive backups. Defaults to 1 second if this method not called.

func (*DB) SetWarning added in v0.8.0 

func (d *DB) SetWarning(setID, warnMsg string) error

SetWarning updates a set with the given warning message. Returns an error if the setID isn't in the database.

type DBRO added in v1.1.0 

type DBRO struct {
	// contains filtered or unexported fields
}

DBRO is the read-only component of the DB struct.

func NewRO added in v1.1.0 

func NewRO(path string) (*DBRO, error)

NewRO returns a *DBRO that can be used to query a set database. Provide the path to the database file.

Returns an error if database can't be opened.

func (*DBRO) GetAll added in v1.1.0 

func (d *DBRO) GetAll() ([]*Set, error)

GetAll returns all the Sets previously added to the database.

func (*DBRO) GetByID added in v1.1.0 

func (d *DBRO) GetByID(id string) *Set

GetByID returns the Sets with the given ID previously added to the database. Returns nil if such a set does not exist.

func (*DBRO) GetByNameAndRequester added in v1.1.0 

func (d *DBRO) GetByNameAndRequester(name, requester string) (*Set, error)

GetByNameAndRequester returns the set with the given name and requester.

Returns nil error when no set found.

func (*DBRO) GetByRequester added in v1.1.0 

func (d *DBRO) GetByRequester(requester string) ([]*Set, error)

GetByRequester returns all the Sets previously added to the database by the given requester.

func (*DBRO) GetDefinedFileEntry added in v1.1.0 

func (d *DBRO) GetDefinedFileEntry(setID string) (*Entry, error)

GetDefinedFileEntry returns the first defined file entry for the given set.

Will return nil Entry if SetFileEntries hasn't been called.

func (*DBRO) GetDirEntries added in v1.1.0 

func (d *DBRO) GetDirEntries(setID string) ([]*Entry, error)

GetDirEntries returns all the dir entries for the given set.

func (*DBRO) GetFailedEntries added in v1.1.0 

func (d *DBRO) GetFailedEntries(setID string) ([]*Entry, int, error)

GetFailedEntries returns up to 10 of the file entries for the given set (both SetFileEntries and SetDiscoveredEntries) that have a failed status. Also returns the number of failed entries that were not returned.

func (*DBRO) GetFileEntries added in v1.1.0 

func (d *DBRO) GetFileEntries(setID string) ([]*Entry, error)

GetFileEntries returns all the file entries for the given set (both SetFileEntries and SetDiscoveredEntries).

func (*DBRO) GetFileEntryForSet added in v1.1.0 

func (d *DBRO) GetFileEntryForSet(setID, filePath string) (*Entry, error)

GetFileEntryForSet returns the file entry for the given path in the given set.

func (*DBRO) GetPureFileEntries added in v1.1.0 

func (d *DBRO) GetPureFileEntries(setID string) ([]*Entry, error)

GetPureFileEntries returns all the file entries for the given set (only SetFileEntries, not SetDiscoveredEntries).

func (*DBRO) LogSetChangesToSlack added in v1.4.0 

func (d *DBRO) LogSetChangesToSlack(slacker Slacker)

type DiscoverCallback added in v0.8.0 

type DiscoverCallback func([]*Entry) ([]*walk.Dirent, error)

DiscoverCallback will receive the sets directory entries and return a list of the files discovered in those directories.

type Entry 

type Entry struct {
	Path        string
	PathForJSON []byte // set by MakeSafeForJSON(); do not set this yourself.
	Size        uint64 // size of local file in bytes.
	Status      EntryStatus
	Type        EntryType
	LastError   string
	LastAttempt time.Time
	Attempts    int
	Inode       uint64
	Dest        string
	// contains filtered or unexported fields
}

Entry holds the status of an entry in a backup set.

func (*Entry) CorrectFromJSON added in v0.6.3 

func (e *Entry) CorrectFromJSON()

CorrectFromJSON copies PathForJSON to Path, so that this value is correct following a decode from JSON.

func (*Entry) InodeStoragePath added in v1.0.3 

func (e *Entry) InodeStoragePath() string

InodeStoragePath returns a relative path that the data for this entry could be stored at if this entry is for a hardlink. The path includes both the inode and the first local path we saw for this inode. This ensures that if an inode gets re-used after previous hardlinks we saw all got deleted locally, the new file will be stored at a new location.

func (*Entry) MakeSafeForJSON added in v0.6.3 

func (e *Entry) MakeSafeForJSON()

MakeSafeForJSON copies Path to PathForJSON, so that if this Entry struct is encoded as JSON, non-UTF8 characters will be preserved. On decode be sure to use CorrectFromJSON().

func (*Entry) ShouldUpload 

func (e *Entry) ShouldUpload(reuploadAfter time.Time) bool

ShouldUpload returns true if this Entry is pending or the last attempt was before the given time. Always returns false if the Type is Abnormal.

type EntryStatus 

type EntryStatus int
const (
	// Pending is an Entry status meaning the file has not yet had an upload
	// attempt.
	Pending EntryStatus = iota

	// UploadingEntry is an Entry status meaning the file has started to upload.
	UploadingEntry

	// Uploaded is an Entry status meaning the file has been uploaded for the
	// first time successfully.
	Uploaded

	// Failed is an Entry status meaning the file failed to upload due to some
	// remote issue.
	Failed

	// Missing is an Entry status meaning the local file is missing so can't be
	// uploaded.
	Missing

	// AbnormalEntry is an Entry status meaning the local files is neither
	// regular nor a symlink (ie. it's a fifo or socket etc.), so shouldn't be
	// uploaded.
	AbnormalEntry

	// Replaced is an Entry status meaning the file has been uploaded previously
	// but was uploaded again because the local file was changed.
	Replaced

	// Skipped is an Entry status meaning the file was not uploaded because it
	// was uploaded previously and hasn't changed since.
	Skipped
)

func (EntryStatus) String 

func (e EntryStatus) String() string

String lets you convert a EntryStatus to a meaningful string.

type EntryType added in v0.8.0 

type EntryType int
const (
	Regular EntryType = iota
	Hardlink
	Symlink
	Directory
	Abnormal // fifos and sockets etc. that we shouldn't upload

	// Unknown is an Entry type meaning the local file is either missing or
	// there was an error trying to get its type.
	Unknown = -1
)

type Error 

type Error struct {
	Msg string
	// contains filtered or unexported fields
}

func (Error) Error 

func (e Error) Error() string

type Set 

type Set struct {
	// An arbitrary (short) name for this backup set.
	Name string

	// The username of the person requesting this backup.
	Requester string

	// The method of transforming local Entries paths in to remote paths, to
	// determine the upload location. "humgen" to use the put.HumgenTransformer,
	// or "prefix=local:remote" to use the put.PrefixTransformer.
	Transformer string

	// Monitor the files and directories and re-upload them whenever they
	// change, checking for changes after the given amount of time. Optional,
	// defaults to unmonitored (a one time upload of Entries).
	MonitorTime time.Duration

	// An optional longer free-text description of this backup set.
	Description string

	// Optional additional metadata which will be applied to every file in the
	// set.
	Metadata map[string]string

	// Delete local paths after successful upload. Optional, defaults to no
	// deletions (ie. do a backup, not a move).
	DeleteLocal bool

	// StartedDiscovery provides the last time that discovery started. This is a
	// read-only value.
	StartedDiscovery time.Time

	// LastDiscovery provides the last time that discovery completed. This is a
	// read-only value.
	LastDiscovery time.Time

	// NumFiles provides the total number of set and discovered files in this
	// set, as of the last discovery. This is a read-only value.
	NumFiles uint64

	// SizeTotal provides the total size (bytes) of set and discovered files in
	// this set, as of the last discovery. This is a read-only value.
	SizeTotal uint64

	// Uploaded provides the total number of set and discovered files in this
	// set that have, for the first time, been uploaded or confirmed uploaded
	// since the last discovery. This is a read-only value.
	Uploaded uint64

	// Replaced is like Uploaded, but for files that had previously been
	// uploaded to iRODS, and now uploaded again because the file on local disk
	// was newer.
	Replaced uint64

	// Skipped is like Uploaded, but for files that had previously been
	// uploaded to iRODS, and were not uploaded again because the file on local
	// disk was the same age.
	Skipped uint64

	// Failed provides the total number of set and discovered files in this set
	// that have failed their upload since the last discovery. This is a
	// read-only value.
	Failed uint64

	// Missing provides the total number of set and discovered files in this set
	// that no longer exist locally since the last discovery. This is a
	// read-only value.
	Missing uint64

	// Abnormal provides the total number of set files in this set that were
	// neither regular files, nor Symlinks (ie. they were fifo or socket files
	// etc).
	Abnormal uint64

	// Symlinks provides the total number of set and discovered files in this
	// set that are symlinks. This is a read-only value.
	Symlinks uint64

	// Hardlinks provides the total number of set and discovered files in this
	// set that are hardlinks and skipped. This is a read-only value.
	Hardlinks uint64

	// Status provides the current status for the set since the last discovery.
	// This is a read-only value.
	Status Status

	// LastCompleted provides the last time that all uploads completed
	// (regardless of success or failure). This is a read-only value.
	LastCompleted time.Time

	// LastCompletedCount provides the count of files on the last upload attempt
	// (those successfully uploaded, those which failed, but not those which
	// were missing locally). This is a read-only value.
	LastCompletedCount uint64

	// LastCompletedSize provides the size of files (bytes) counted in
	// LastCompletedCount. This is a read-only value.
	LastCompletedSize uint64

	// SizeUploaded provides the size of files (bytes) actually uploaded (not
	// skipped) since the last discovery. This is a read-only value.
	SizeUploaded uint64

	// Error holds any error that applies to the whole set, such as an issue
	// with the Transformer. This is a read-only value.
	Error string

	// Warning contains errors that do not stop progression. This is a read-only
	// value.
	Warning string
	// contains filtered or unexported fields
}

Set describes a backup set; a list of files and directories to backup, plus some metadata. All properties are required unless otherwise noted.

func (*Set) Count 

func (s *Set) Count() string

Count provides a string representation of NumFiles, or if 0, returns the LastCompletedCount with a textual note to that effect.

func (*Set) Discovered 

func (s *Set) Discovered() string

Discovered provides a string representation of when discovery last completed, or if it hasn't, when it started, or if it hasn't, says "not started".

func (*Set) DiscoveryCompleted added in v1.4.0 

func (s *Set) DiscoveryCompleted(numFiles uint64)

DiscoveryCompleted should be called when you complete discovering a set. Pass in the number of files you discovered.

func (*Set) HasProblems added in v1.2.0 

func (s *Set) HasProblems() bool

HasProblems returns true if any uploads have failed, if there's an error on the set itself, or if our transformer doesn't work.

Currently does NOT check if all of the user's desired local paths can be transformed, so there might actually be problems.

func (*Set) ID 

func (s *Set) ID() string

ID returns an ID for this set, generated deterministiclly from its Name and Requester. Ie. it is unique between all requesters, and amongst a requester's differently named sets. Sets with the same Name and Requester will have the same ID().

func (*Set) Incomplete added in v1.2.0 

func (s *Set) Incomplete() bool

Incomplete returns true if our Status is not Complete, or if we HasProblems(). (Status can be "Complete" even if there are upload failures, because "Complete" status only means we're not trying to upload any more.)

func (*Set) LogChangesToSlack added in v1.4.0 

func (s *Set) LogChangesToSlack(slacker Slacker)

LogChangesToSlack will cause the set to use the slacker when significant events happen to the set.

func (*Set) MakeTransformer 

func (s *Set) MakeTransformer() (put.PathTransformer, error)

MakeTransformer turns our Transformer string in to a put.HumgenTransformer or a put.PrefixTransformer as appropriate.

func (*Set) Queued added in v1.2.0 

func (s *Set) Queued() bool

Queued returns true if we're either pending discovery or upload. Ie. the set was recently added or updated, but uploads haven't begun yet probably due to uploads for other sets being first in the queue.

func (*Set) RecoveryError added in v1.4.0 

func (s *Set) RecoveryError(err error)

func (*Set) SetError added in v1.4.0 

func (s *Set) SetError(errMsg string)

SetError records the given error against the set, indicating it wont work.

func (*Set) SetWarning added in v1.4.0 

func (s *Set) SetWarning(warnMsg string)

SetWarning records the given warning against the set, indicating it has an issue.

func (*Set) Size 

func (s *Set) Size() string

Size provides a string representation of SizeFiles in a human readable format, or if 0, returns the LastCompletedSize with a textual note to that effect.

func (*Set) SuccessfullyStoredInDB added in v1.4.0 

func (s *Set) SuccessfullyStoredInDB()

SuccessfullyStoredInDB should be called when you successfully store the set in DB.

func (*Set) TransformPath added in v1.1.0 

func (s *Set) TransformPath(path string) (string, error)

func (*Set) UpdateBasedOnEntry added in v1.4.0 

func (s *Set) UpdateBasedOnEntry(entry *Entry, getFileEntries func(string) ([]*Entry, error)) error

UpdateBasedOnEntry updates set status values based on an updated Entry from updateFileEntry(), assuming that request is for one of set's file entries.

func (*Set) UploadedSize added in v1.5.0 

func (s *Set) UploadedSize() string

UploadedSize provides a string representation of SizeUploaded in a human readable format. This is the size of actual uploads (excluding skipped, unlike Size()) since the last discovery.

type SizeAndNumber added in v1.1.0 

type SizeAndNumber struct {
	For    string // Aggregation id (set name or requester name or "total").
	Size   uint64 // Size in bytes.
	Number uint64 // Number of files.
}

SizeAndNumber provides the total Size and Number of files in the set(s) this is a summary of, where "For" is the id of aggregation, eg it might be a set requester name if this was for a breakdown by requester.

Files in the set(s) that began an upload (and we thus know their size) are part of these totals.

func (*SizeAndNumber) SizeTiB added in v1.1.0 

func (s *SizeAndNumber) SizeTiB() float64

SizeTiB returns our Size in TiB.

type Slacker added in v1.4.0 

type Slacker interface {
	SendMessage(level slack.Level, msg string)
}

type Status 

type Status int
const (
	// PendingDiscovery is a Set status meaning the set's entries are pending
	// existence, size and directory content discovery.
	PendingDiscovery Status = iota

	// PendingUpload is a Set status meaning discovery has completed, but no
	// entries have been uploaded since then.
	PendingUpload

	// Uploading is a Set status meaning discovery has completed and upload of
	// entries has started.
	Uploading

	// Failing is a Set status meaning at least 1 of the entries has failed to
	// upload after 3 attempts. Other uploads are ongoing.
	Failing

	// Complete is a Set status meaning all entries have had an upload attempt
	// since the last discovery. (Some uploads may have failed, but they had
	// 3 retries.)
	Complete
)

func (Status) String 

func (s Status) String() string

String lets you convert a Status to a meaningful string.

type Usage added in v1.1.0 

type Usage struct {
	// Total size and number of files across all given sets.
	Total *SizeAndNumber

	// ByRequester is a slice where each SizeAndNumber is the total size and
	// number of files for a particular requester amongst the given sets. The
	// slice is sorted from largest size to smallest.
	ByRequester []*SizeAndNumber

	// BySet is a slice where each SizeAndNumber is the total size and number of
	// files in each set. The "For" equals the set requester concatenated with
	// the set name. The slice is sorted from largest size to smallest.
	BySet []*SizeAndNumber

	// ByMonth is a slice where each SizeAndNumber is the total size and number
	// of files in sets that last completed in a certain calendar month. The
	// slice is sorted in date order. Incomplete sets are ignored.
	ByMonth []*SizeAndNumber
}

Usage is the type returned by UsageSummary() and contains various breakdowns of how much iRODS would be used up by the sets.

func UsageSummary added in v1.1.0 

func UsageSummary(sets []*Set) *Usage

UsageSummary returns a Usage that summaries potential iRODS usage of the given sets, broken down in various ways.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.