put

package
v0.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 20, 2023 License: MIT Imports: 24 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	MetaNamespace = "ibackup:"

	ErrStatFailed = "stat of local path returned strange results"
)
View Source 
const (
	ErrLocalNotAbs  = "local path could not be made absolute"
	ErrRemoteNotAbs = "remote path not absolute"
	ErrReadTimeout  = "local file read timed out"
)
View Source 
const ErrMockMetaFail = "meta fail"
View Source 
const ErrMockPutFail = "put fail"
View Source 
const ErrMockStatFail = "stat fail"
View Source 
const (
	ErrOperationTimeout = "iRODS operation timed out"
)

Variables

This section is empty.

Functions

func HumgenTransformer 

func HumgenTransformer(local string) (string, error)

HumgenTransformer is a PathTransformer that will convert a local "lustre" path to a "canonical" path in the humgen iRODS zone.

This transform is specific to the "humgen" group at the Sanger Institute.

Types

type Baton 

type Baton struct {
	// contains filtered or unexported fields
}

Baton is a Handler that uses Baton (via extendo) to interact with iRODS.

func GetBatonHandler 

func GetBatonHandler() (*Baton, error)

GetBatonHandler returns a Handler that uses Baton to interact with iRODS. If you don't have baton-do in your PATH, you'll get an error.

func (*Baton) AddMeta 

func (b *Baton) AddMeta(path string, meta map[string]string) error

func (*Baton) Cleanup 

func (b *Baton) Cleanup() error

Cleanup stops our clients and closes our client pool.

func (*Baton) CollectionsDone added in v0.6.0 

func (b *Baton) CollectionsDone() error

CollectionsDone closes the connections used for connection creation, and creates new ones for doing puts and metadata operations.

func (*Baton) EnsureCollection 

func (b *Baton) EnsureCollection(collection string) error

EnsureCollection ensures the given collection exists in iRODS, creating it if necessary. You must call Connect() before calling this.

This is safe for calling concurrently, and uses multiple connections. But an artefact is that the error you get might be for a different EnsureCollection() call you made for a different collection. This shouldn't make much difference if you just collect all your errors and don't care about order.

func (*Baton) Put 

func (b *Baton) Put(request *Request) error

Put uploads request Local to the Remote object, overwriting it if it already exists. It calculates and stores the md5 checksum remotely, comparing to the local checksum.

func (*Baton) RemoveMeta 

func (b *Baton) RemoveMeta(path string, meta map[string]string) error

func (*Baton) Stat 

func (b *Baton) Stat(request *Request) (*ObjectInfo, error)

Stat gets mtime and metadata info for the request Remote object.

type Error 

type Error struct {
	// contains filtered or unexported fields
}

func (Error) Error 

func (e Error) Error() string

func (Error) Is added in v0.4.0 

func (e Error) Is(err error) bool

type FileReadTester added in v0.5.0 

type FileReadTester func(ctx context.Context, path string) error

FileReadTester is a function that attempts to open and read the given path, returning any error doing so. Should stop and clean up if the given ctx becomes done before the open and read succeeds.

type Handler 

type Handler interface {
	// EnsureCollection checks if the given collection exists in iRODS, creates
	// it if not, then double-checks it now exists. Must support being called
	// concurrently.
	EnsureCollection(collection string) error

	// CollectionsDone is called after all collections have been created. This
	// method can do things like cleaning up connections created for collection
	// creation. It can also create new connections for subsequent Put() and
	// *Meta calls that are likely to occur.
	CollectionsDone() error

	// Stat checks if the Request's Remote object exists. If it does, records
	// its metadata in the returned ObjectInfo. Returns an error if there was a
	// problem finding out information (but not if the object does not exist).
	Stat(request *Request) (*ObjectInfo, error)

	// Put uploads the Request's Local file to the Remote location, overwriting
	// any existing object, and ensuring that a locally calculated and remotely
	// calculated md5 checksum match.
	Put(request *Request) error

	// RemoveMeta deletes the given metadata from the given object.
	RemoveMeta(path string, meta map[string]string) error

	// AddMeta adds the given metadata to the given object. Given metadata keys
	// should already have been removed with RemoveMeta() from the remote
	// object.
	AddMeta(path string, meta map[string]string) error

	// Cleanup stops any connections created earlier and does any other cleanup
	// needed.
	Cleanup() error
}

Handler is something that knows how to communicate with iRODS and carry out certain operations.

type LocalHandler added in v0.5.0 

type LocalHandler struct {
	// contains filtered or unexported fields
}

LocalHandler satisfies the Handler interface, treating "Remote" as local paths and moving from Local to Remote for the Put().

func GetLocalHandler added in v0.2.0 

func GetLocalHandler() (*LocalHandler, error)

GetLocalHandler returns a Handler that doesn't actually interact with iRODS, but instead simply treats "Remote" as local paths and copies from Local to Remote for any Put()s. For use during tests.

func (*LocalHandler) AddMeta added in v0.5.0 

func (l *LocalHandler) AddMeta(path string, meta map[string]string) error

AddMeta adds the given meta to a map stored under path. Returns an error if metafail == path, or if keys were already defined in the map.

func (*LocalHandler) Cleanup added in v0.5.0 

func (l *LocalHandler) Cleanup() error

Cleanup just records this was called.

func (*LocalHandler) CollectionsDone added in v0.6.0 

func (l *LocalHandler) CollectionsDone() error

CollectionsDone says we connected and prepares us for metadata handling.

func (*LocalHandler) EnsureCollection added in v0.5.0 

func (l *LocalHandler) EnsureCollection(dir string) error

EnsureCollection creates the given dir locally and records that we did this.

func (*LocalHandler) MakeMetaFail added in v0.5.0 

func (l *LocalHandler) MakeMetaFail(remote string)

MakeMetaFail will result in any subsequent *Meta()s for a Request with the given remote path failing.

func (*LocalHandler) MakePutFail added in v0.5.0 

func (l *LocalHandler) MakePutFail(remote string)

MakePutFail will result in any subsequent Put()s for a Request with the given remote path failing.

func (*LocalHandler) MakePutSlow added in v0.5.0 

func (l *LocalHandler) MakePutSlow(local string, dur time.Duration)

MakePutSlow will result in any subsequent Put()s for a Request with the given local path taking the given amount of time.

func (*LocalHandler) MakeStatFail added in v0.5.0 

func (l *LocalHandler) MakeStatFail(remote string)

MakeStatFail will result in any subsequent Stat()s for a Request with the given remote path failing.

func (*LocalHandler) Put added in v0.5.0 

func (l *LocalHandler) Put(request *Request) error

Put just copies from Local to Remote. Returns an error if putFail == Remote.

func (*LocalHandler) RemoveMeta added in v0.5.0 

func (l *LocalHandler) RemoveMeta(path string, meta map[string]string) error

RemoveMeta deletes the given keys from a map stored under path. Returns an error if metaFail == path.

func (*LocalHandler) Stat added in v0.5.0 

func (l *LocalHandler) Stat(request *Request) (*ObjectInfo, error)

Stat returns info about the Remote file, which is a local file on disk. Returns an error if statFail == Remote.

type ObjectInfo 

type ObjectInfo struct {
	Exists bool
	Size   uint64
	Meta   map[string]string
}

func Stat 

func Stat(localPath string) (*ObjectInfo, error)

Stat stats localPath like os.Stat(), but returns information about the file in ObjectInfo Meta (mtime, owner and group information).

func (*ObjectInfo) HasSameModTime 

func (o *ObjectInfo) HasSameModTime(as *ObjectInfo) bool

HasSameModTime tells you if this ObjectInfo has the same ModTime() as the given one. Instead of comparing metadata strings, tries to convert them to times in a way that a comparison would be meaningful, in case the string was not generated by us.

func (*ObjectInfo) ModTime 

func (o *ObjectInfo) ModTime() time.Time

ModTime converts our mtime metadata key value string to a time.Time. Returns time zero if the key isn't present or convertable.

type PathTransformer 

type PathTransformer func(local string) (remote string, err error)

PathTransformer is a function that given a local path, returns the corresponding remote path.

func PrefixTransformer 

func PrefixTransformer(localPrefix, remotePrefix string) PathTransformer

PrefixTransformer returns a PathTransformer that will replace localPrefix in any path given to it with remotePrefix, and return the result.

If the given path does not contain localPrefix, returns the path prefixed with remotePrefix (treating the given path as relative to localPrefix).

type Putter 

type Putter struct {
	// contains filtered or unexported fields
}

Putter is used to Put() files in iRODS.

func New 

func New(handler Handler, requests []*Request) (*Putter, error)

New returns a *Putter that will use the given Handler to Put() all the requests in iRODS. You should defer Cleanup() on the return value. All the incoming requests will have their paths validated (they must be absolute).

func (*Putter) Cleanup 

func (p *Putter) Cleanup() error

Cleanup should be deferred after making a New Putter. It handles things like disconnecting.

func (*Putter) CreateCollections 

func (p *Putter) CreateCollections() error

CreateCollections will determine the minimal set of collections that need to be created to support a future Put() request for our requests, checks if those collections exist in iRODS, and creates them if not.

You MUST call this before Put(), even if you think all collections have already been created.

Tries to create all needed collections, potentially returning multiple errors wrapped in to one.

func (*Putter) Put 

func (p *Putter) Put() (chan *Request, chan *Request, chan *Request)

Put will upload all our request Local files to iRODS at the Remote locations. You ought to call CreateCollections() before calling this.

Existing files in iRODS will be overwritten if the local file's mtime is different to the remote's. If the same, the put will be skipped.

MD5 checksums will be calculated locally and remotely and compared to ensure a perfect upload. The request metadata will then replace any existing metadata with the same keys on the Remote object.

Requests that need to be uploaded are first sent on the first returned channel, with Status set to "uploading".

Each of those will then be send on the second returned channel, with Status set like: 

 "uploaded":   a new object was uploaded to iRODS
	"replaced":   an existing object was replaced in iRODS, because Local was
	              newer than Remote
	"failed":     An upload attempt failed; see Error

The third return channel will receive Requests that didn't need to be uploaded, with Status set like: 

"unmodified": Local and Remote had the same modification time, so nothing
              was done
"missing":    Local path could not be accessed, upload skipped; see Error

func (*Putter) SetFileReadTester added in v0.5.0 

func (p *Putter) SetFileReadTester(tester FileReadTester)

SetFileReadTester sets the function used to see if a file can be opened and read. If this attempt hangs, the function should stop and clean up in response to its context becoming done (when the FileReadTimeout elapses).

The default tester shells out to the "head" command so that we don't leak stuck goroutines.

func (*Putter) SetFileReadTimeout added in v0.5.0 

func (p *Putter) SetFileReadTimeout(timeout time.Duration)

SetFileReadTimeout sets how long to wait on a test open and read of each local file before considering it not possible to upload. The default is 10seconds.

type Request 

type Request struct {
	Local         string
	Remote        string
	LocalForJSON  []byte // set by MakeSafeForJSON(); do not set this yourself.
	RemoteForJSON []byte // set by MakeSafeForJSON(); do not set this yourself.
	Requester     string
	Set           string
	Meta          map[string]string
	Status        RequestStatus
	Size          uint64 // size of Local in bytes, set for you on returned Requests.
	Error         string
	Stuck         *Stuck
	// contains filtered or unexported fields
}

Request represents a local file you would like transferred to a remote iRODS path, and any extra metadata (beyond the defaults which include user, group, mtime, upload date) you'd like to associate with it. Setting Requester and Set will add these to the requesters and sets metadata on upload.

func NewRequestWithTransformedLocal 

func NewRequestWithTransformedLocal(local string, pt PathTransformer) (*Request, error)

NewRequestWithTransformedLocal takes a local path string, uses the given PathTransformer to generate the corresponding remote path, and returns a Request with Local and Remote set.

func (*Request) Clone added in v0.2.0 

func (r *Request) Clone() *Request

Clone returns a copy of this request that can safely be altered without affecting the original.

func (*Request) CorrectFromJSON added in v0.6.3 

func (r *Request) CorrectFromJSON()

CorrectFromJSON copies LocalForJSON and RemoteForJSON to Local and Remote, so that these values are correct following a decode from JSON.

func (*Request) ID added in v0.2.0 

func (r *Request) ID() string

ID returns a deterministic identifier that is unique to this Request's combination of Local, Remote, Requester and Set.

func (*Request) MakeSafeForJSON added in v0.6.3 

func (r *Request) MakeSafeForJSON()

MakeSafeForJSON copies Local and Remote to LocalForJSON and RemoteForJSON, so that if this Request struct is encoded as JSON, non-UTF8 characters will be preserved. On decode be sure to use CorrectFromJSON().

func (*Request) ValidatePaths 

func (r *Request) ValidatePaths() error

ValidatePaths checks that both Local and Remote paths are absolute. (It does NOT check that either path exists.)

type RequestStatus 

type RequestStatus string
const (
	RequestStatusPending    RequestStatus = "pending"
	RequestStatusReserved   RequestStatus = "reserved"
	RequestStatusUploading  RequestStatus = "uploading"
	RequestStatusUploaded   RequestStatus = "uploaded"
	RequestStatusReplaced   RequestStatus = "replaced"
	RequestStatusUnmodified RequestStatus = "unmodified"
	RequestStatusMissing    RequestStatus = "missing"
	RequestStatusFailed     RequestStatus = "failed"
	ErrNotHumgenLustre                    = "not a valid humgen lustre path"
)

type Stuck added in v0.2.0 

type Stuck struct {
	UploadStarted time.Time
	Host          string
	PID           int
}

Stuck is used to provide details of a potentially "stuck" upload Request.

func NewStuck added in v0.2.0 

func NewStuck(uploadStarted time.Time) *Stuck

NewStuck returns a Stuck with the given UploadStarted and the current Host and PID.

func (*Stuck) String added in v0.2.0 

func (s *Stuck) String() string

String returns a message explaining our stuck details.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.