storage

Documentation

Index

• Constants
• type Backend
• type GSBackend
  • func NewGSBackend(conf config.GSStorage) (*GSBackend, error)
  • func (gs *GSBackend) Get(ctx context.Context, rawurl string, hostPath string, class tes.FileType) error
  • func (gs *GSBackend) PutFile(ctx context.Context, rawurl string, hostPath string) error
  • func (gs *GSBackend) Supports(rawurl string, hostPath string, class tes.FileType) bool
• type LocalBackend
  • func NewLocalBackend(conf config.LocalStorage) (*LocalBackend, error)
  • func (local *LocalBackend) Get(ctx context.Context, url string, hostPath string, class tes.FileType) error
  • func (local *LocalBackend) PutFile(ctx context.Context, url string, hostPath string) error
  • func (local *LocalBackend) Supports(rawurl string, hostPath string, class tes.FileType) bool
• type S3Backend
  • func NewS3Backend(conf config.S3Storage) (*S3Backend, error)
  • func (s3b *S3Backend) Get(ctx context.Context, url string, hostPath string, class tes.FileType) error
  • func (s3b *S3Backend) PutFile(ctx context.Context, url string, hostPath string) error
  • func (s3b *S3Backend) Supports(url string, hostPath string, class tes.FileType) bool
• type Storage
  • func (storage Storage) Get(ctx context.Context, url string, path string, class tes.FileType) error
  • func (storage Storage) Put(ctx context.Context, url string, path string, class tes.FileType) ([]*tes.OutputFileLog, error)
  • func (storage Storage) Supports(url string, path string, class tes.FileType) bool
  • func (storage Storage) WithBackend(b Backend) Storage
  • func (storage Storage) WithConfig(conf config.StorageConfig) (Storage, error)
• type SwiftBackend
  • func NewSwiftBackend(conf config.SwiftStorage) (*SwiftBackend, error)
  • func (sw *SwiftBackend) Get(ctx context.Context, rawurl string, hostPath string, class tes.FileType) error
  • func (sw *SwiftBackend) PutFile(ctx context.Context, rawurl string, hostPath string) error
  • func (sw *SwiftBackend) Supports(rawurl string, hostPath string, class tes.FileType) bool

Constants

const (
	// File represents the file type
	File = tes.FileType_FILE
	// Directory represents the directory type
	Directory = tes.FileType_DIRECTORY
)

const S3Protocol = "s3://"

S3Protocol defines the expected URL prefix for S3, "s3://"

Types

type Backend

type Backend interface {
	Get(ctx context.Context, url string, path string, class tes.FileType) error
	PutFile(ctx context.Context, url string, path string) error
	// Determines whether this backends supports the given request (url/path/class).
	// A backend normally uses this to match the url prefix (e.g. "s3://")
	// TODO would it be useful if this included the request type (Get/Put)?
	Supports(url string, path string, class tes.FileType) bool
}

Backend provides an interface for a storage backend. New storage backends must support this interface.

type GSBackend

type GSBackend struct {
	// contains filtered or unexported fields
}

GSBackend provides access to an GS object store.

func NewGSBackend

func NewGSBackend(conf config.GSStorage) (*GSBackend, error)

NewGSBackend creates an GSBackend client instance, give an endpoint URL and a set of authentication credentials.

func (*GSBackend) Get

func (gs *GSBackend) Get(ctx context.Context, rawurl string, hostPath string, class tes.FileType) error

Get copies an object from GS to the host path.

func (*GSBackend) PutFile

func (gs *GSBackend) PutFile(ctx context.Context, rawurl string, hostPath string) error

PutFile copies an object (file) from the host path to GS.

func (*GSBackend) Supports

func (gs *GSBackend) Supports(rawurl string, hostPath string, class tes.FileType) bool

Supports returns true if this backend supports the given storage request. The Google Storage backend supports URLs which have a "gs://" scheme.

type LocalBackend

type LocalBackend struct {
	// contains filtered or unexported fields
}

LocalBackend provides access to a local-disk storage system.

func NewLocalBackend

func NewLocalBackend(conf config.LocalStorage) (*LocalBackend, error)

NewLocalBackend returns a LocalBackend instance, configured to limit file system access to the given allowed directories.

func (*LocalBackend) Get

func (local *LocalBackend) Get(ctx context.Context, url string, hostPath string, class tes.FileType) error

Get copies a file from storage into the given hostPath.

func (*LocalBackend) PutFile

func (local *LocalBackend) PutFile(ctx context.Context, url string, hostPath string) error

PutFile copies a file from the hostPath into storage.

func (*LocalBackend) Supports

func (local *LocalBackend) Supports(rawurl string, hostPath string, class tes.FileType) bool

Supports indicates whether this backend supports the given storage request. For the LocalBackend, the url must start with "file://"

type S3Backend

type S3Backend struct {
	// contains filtered or unexported fields
}

S3Backend provides access to an S3 object store.

func NewS3Backend

func NewS3Backend(conf config.S3Storage) (*S3Backend, error)

NewS3Backend creates an S3Backend session instance

func (*S3Backend) Get

func (s3b *S3Backend) Get(ctx context.Context, url string, hostPath string, class tes.FileType) error

Get copies an object from S3 to the host path.

func (*S3Backend) PutFile

func (s3b *S3Backend) PutFile(ctx context.Context, url string, hostPath string) error

PutFile copies an object (file) from the host path to S3.

func (*S3Backend) Supports

func (s3b *S3Backend) Supports(url string, hostPath string, class tes.FileType) bool

Supports indicates whether this backend supports the given storage request. For S3, the url must start with "s3://".

type Storage

type Storage struct {
	// contains filtered or unexported fields
}

Storage provides a client for accessing multiple storage systems, i.e. for downloading/uploading task files from S3, GS, local disk, etc.

For a given storage url, the storage backend is usually determined by the url prefix, e.g. "s3://my-bucket/file" will access the S3 backend.

func (Storage) Get

func (storage Storage) Get(ctx context.Context, url string, path string, class tes.FileType) error

Get downloads a file from a storage system at the given "url". The file is downloaded to the given local "path". "class" is either "File" or "Directory".

func (Storage) Put

func (storage Storage) Put(ctx context.Context, url string, path string, class tes.FileType) ([]*tes.OutputFileLog, error)

Put uploads a file to a storage system at the given "url". The file is uploaded from the given local "path". "class" is either "File" or "Directory".

func (Storage) Supports

func (storage Storage) Supports(url string, path string, class tes.FileType) bool

Supports indicates whether the storage supports the given request.

func (Storage) WithBackend

func (storage Storage) WithBackend(b Backend) Storage

WithBackend returns a new child Storage instance with the given backend added.

func (Storage) WithConfig

func (storage Storage) WithConfig(conf config.StorageConfig) (Storage, error)

WithConfig returns a new Storage instance with the given additional configuration.

type SwiftBackend

type SwiftBackend struct {
	// contains filtered or unexported fields
}

SwiftBackend provides access to an sw object store.

func NewSwiftBackend

func NewSwiftBackend(conf config.SwiftStorage) (*SwiftBackend, error)

NewSwiftBackend creates an SwiftBackend client instance, give an endpoint URL and a set of authentication credentials.

func (*SwiftBackend) Get

func (sw *SwiftBackend) Get(ctx context.Context, rawurl string, hostPath string, class tes.FileType) error

Get copies an object from storage to the host path.

func (*SwiftBackend) PutFile

func (sw *SwiftBackend) PutFile(ctx context.Context, rawurl string, hostPath string) error

PutFile copies an object (file) from the host path to storage.

func (*SwiftBackend) Supports

func (sw *SwiftBackend) Supports(rawurl string, hostPath string, class tes.FileType) bool

Supports indicates whether this backend supports the given storage request. For sw, the url must start with "sw://".