put

Documentation

Index

• Constants
• func HumgenTransformer(local string) (string, error)
• type Baton
  • func GetBatonHandler() (*Baton, error)
  • func (b *Baton) AddMeta(path string, meta map[string]string) error
  • func (b *Baton) Cleanup() error
  • func (b *Baton) Connect() error
  • func (b *Baton) EnsureCollection(collection string) error
  • func (b *Baton) Put(request *Request) error
  • func (b *Baton) RemoveMeta(path string, meta map[string]string) error
  • func (b *Baton) Stat(request *Request) (*ObjectInfo, error)
• type Error
  • func (e Error) Error() string
• type Handler
  • func GetLocalHandler() (Handler, error)
• type ObjectInfo
  • func Stat(localPath string) (*ObjectInfo, error)
  • func (o *ObjectInfo) HasSameModTime(as *ObjectInfo) bool
  • func (o *ObjectInfo) ModTime() time.Time
• type PathTransformer
  • func PrefixTransformer(localPrefix, remotePrefix string) PathTransformer
• type Putter
  • func New(handler Handler, requests []*Request) (*Putter, error)
  • func (p *Putter) Cleanup() error
  • func (p *Putter) CreateCollections() error
  • func (p *Putter) Put() (chan *Request, chan *Request)
• type Request
  • func NewRequestWithTransformedLocal(local string, pt PathTransformer) (*Request, error)
  • func (r *Request) Clone() *Request
  • func (r *Request) ID() string
  • func (r *Request) ValidatePaths() error
• type RequestStatus
• type Stuck
  • func NewStuck(uploadStarted time.Time) *Stuck
  • func (s *Stuck) String() string

Constants

const (
	MetaNamespace = "ibackup:"

	ErrStatFailed = "stat of local path returned strange results"
)

const (
	ErrLocalNotAbs  = "local path could not be made absolute"
	ErrRemoteNotAbs = "remote path not absolute"
)

Functions

func HumgenTransformer

func HumgenTransformer(local string) (string, error)

HumgenTransformer is a PathTransformer that will convert a local "lustre" path to a "canonical" path in the humgen iRODS zone.

This transform is specific to the "humgen" group at the Sanger Institute.

Types

type Baton

type Baton struct {
	// contains filtered or unexported fields
}

Baton is a Handler that uses Baton (via extendo) to interact with iRODS.

func GetBatonHandler

func GetBatonHandler() (*Baton, error)

GetBatonHandler returns a Handler that uses Baton to interact with iRODS. If you don't have baton-do in your PATH, you'll get an error.

func (*Baton) AddMeta

func (b *Baton) AddMeta(path string, meta map[string]string) error

func (*Baton) Cleanup

func (b *Baton) Cleanup() error

Cleanup stops our clients and closes our client pool.

func (*Baton) Connect

func (b *Baton) Connect() error

Connect creates a connection pool and prepares 2 connections to iRODS concurrently, for later use by other methods.

func (*Baton) EnsureCollection

func (b *Baton) EnsureCollection(collection string) error

EnsureCollection ensures the given collection exists in iRODS, creating it if necessary. You must call Connect() before calling this.

func (*Baton) Put

func (b *Baton) Put(request *Request) error

Put uploads request Local to the Remote object, overwriting it if it already exists. It calculates and stores the md5 checksum remotely, comparing to the local checksum.

func (*Baton) RemoveMeta

func (b *Baton) RemoveMeta(path string, meta map[string]string) error

func (*Baton) Stat

func (b *Baton) Stat(request *Request) (*ObjectInfo, error)

Stat gets mtime and metadata info for the request Remote object.

type Error

type Error struct {
	// contains filtered or unexported fields
}

func (Error) Error

func (e Error) Error() string

type Handler

type Handler interface {
	// Connect uses environmental details to make 1 or more connections to
	// iRODS, ready for subsequent use.
	Connect() error

	// Cleanup stops any connections from Connect() and does any other cleanup
	// needed.
	Cleanup() error

	// EnsureCollection checks if the given collection exists in iRODS, creates
	// it if not, then double-checks it now exists.
	EnsureCollection(collection string) error

	// Stat checks if the Request's Remote object exists. If it does, records
	// its metadata in the returned ObjectInfo. Returns an error if there was a
	// problem finding out information (but not if the object does not exist).
	Stat(request *Request) (*ObjectInfo, error)

	// Put uploads the Request's Local file to the Remote location, overwriting
	// any existing object, and ensuring that a locally calculated and remotely
	// calculated md5 checksum match.
	Put(request *Request) error

	// RemoveMeta deletes the given metadata from the given object.
	RemoveMeta(path string, meta map[string]string) error

	// AddMeta adds the given metadata to the given object. Given metadata keys
	// should already have been removed with RemoveMeta() from the remote
	// object.
	AddMeta(path string, meta map[string]string) error
}

Handler is something that knows how to communicate with iRODS and carry out certain operations.

func GetLocalHandler

func GetLocalHandler() (Handler, error)

GetLocalHandler returns a Handler that doesn't actually interact with iRODS, but instead simply treats "Remote" as local paths and copies from Local to Remote for any Put()s. Only for use during tests.

type ObjectInfo

type ObjectInfo struct {
	Exists bool
	Size   uint64
	Meta   map[string]string
}

func Stat

func Stat(localPath string) (*ObjectInfo, error)

Stat stats localPath like os.Stat(), but returns information about the file in ObjectInfo Meta (mtime, owner and group information).

func (*ObjectInfo) HasSameModTime

func (o *ObjectInfo) HasSameModTime(as *ObjectInfo) bool

HasSameModTime tells you if this ObjectInfo has the same ModTime() as the given one. Instead of comparing metadata strings, tries to convert them to times in a way that a comparison would be meaningful, in case the string was not generated by us.

func (*ObjectInfo) ModTime

func (o *ObjectInfo) ModTime() time.Time

ModTime converts our mtime metadata key value string to a time.Time. Returns time zero if the key isn't present or convertable.

type PathTransformer

type PathTransformer func(local string) (remote string, err error)

PathTransformer is a function that given a local path, returns the corresponding remote path.

func PrefixTransformer

func PrefixTransformer(localPrefix, remotePrefix string) PathTransformer

PrefixTransformer returns a PathTransformer that will replace localPrefix in any path given to it with remotePrefix, and return the result.

If the given path does not contain localPrefix, returns the path prefixed with remotePrefix (treating the given path as relative to localPrefix).

type Putter

type Putter struct {
	// contains filtered or unexported fields
}

Putter is used to Put() files in iRODS.

func New

func New(handler Handler, requests []*Request) (*Putter, error)

New returns a *Putter that will use the given Handler to Put() all the requests in iRODS. You should defer Cleanup() on the return value. All the incoming requests will have their paths validated (they must be absolute).

func (*Putter) Cleanup

func (p *Putter) Cleanup() error

Cleanup should be deferred after making a New Putter. It handles things like disconnecting.

func (*Putter) CreateCollections

func (p *Putter) CreateCollections() error

CreateCollections will determine the minimal set of collections that need to be created to support a future Put() request for our requests, checks if those collections exist in iRODS, and creates them if not.

You should call this before Put(), unless you're sure all collections already exist.

Gives up on the first failed operation, and returns that error.

func (*Putter) Put

func (p *Putter) Put() (chan *Request, chan *Request)

Put will upload all our request Local files to iRODS at the Remote locations. You ought to call CreateCollections() before calling this.

Existing files in iRODS will be overwritten if the local file's mtime is different to the remote's. If the same, the put will be skipped.

MD5 checksums will be calculated locally and remotely and compared to ensure a perfect upload. The request metadata will then replace any existing metadata with the same keys on the Remote object.

All our requests are eventually sent on the first returned channel (in potentially a different order to our input Request slice) with Status set on them:

"uploaded":   a new object was uploaded to iRODS
"replaced":   an existing object was replaced in iRODS, because Local was
              newer than Remote
"unmodified": Local and Remote had the same modification time, so nothing
              was done
"missing":    Local path could not be accessed, upload skipped; see Error
"failed":     An upload attempt failed; see Error

The second returned channel receives an input request with Status uploading as soon as we start to upload that request. All input requests are not guaranteed to be send on this channel (eg. not missing or unmodified ones).

type Request

type Request struct {
	Local     string
	Remote    string
	Requester string
	Set       string
	Meta      map[string]string
	Status    RequestStatus
	Size      uint64 // size of Local in bytes, set for you on returned Requests.
	Error     string
	Stuck     *Stuck
	// contains filtered or unexported fields
}

Request represents a local file you would like transferred to a remote iRODS path, and any extra metadata (beyond the defaults which include user, group, mtime, upload date) you'd like to associate with it. Setting Requester and Set will add these to the requesters and sets metadata on upload.

func NewRequestWithTransformedLocal

func NewRequestWithTransformedLocal(local string, pt PathTransformer) (*Request, error)

NewRequestWithTransformedLocal takes a local path string, uses the given PathTransformer to generate the corresponding remote path, and returns a Request with Local and Remote set.

func (*Request) Clone

func (r *Request) Clone() *Request

Clone returns a copy of this request that can safely be altered without affecting the original.

func (*Request) ID

func (r *Request) ID() string

ID returns a deterministic identifier that is unique to this Request's combination of Local, Remote, Requester and Set.

func (*Request) ValidatePaths

func (r *Request) ValidatePaths() error

ValidatePaths checks that both Local and Remote paths are absolute. (It does NOT check that either path exists.)

type RequestStatus

type RequestStatus string

const (
	RequestStatusUploading  RequestStatus = "uploading"
	RequestStatusUploaded   RequestStatus = "uploaded"
	RequestStatusReplaced   RequestStatus = "replaced"
	RequestStatusUnmodified RequestStatus = "unmodified"
	RequestStatusMissing    RequestStatus = "missing"
	RequestStatusFailed     RequestStatus = "failed"
	ErrNotHumgenLustre                    = "not a valid humgen lustre path"
)

type Stuck

type Stuck struct {
	UploadStarted time.Time
	Host          string
	PID           int
}

Stuck is used to provide details of a potentially "stuck" upload Request.

func NewStuck

func NewStuck(uploadStarted time.Time) *Stuck

NewStuck returns a Stuck with the given UploadStarted and the current Host and PID.

func (*Stuck) String

func (s *Stuck) String() string

String returns a message explaining our stuck details.