transferarchiver

Documentation

Index

• Variables
• func Copy(ctx context.Context, dst io.Writer, src io.Reader) (int64, error)
• type ArchiverOptions
• type ArchiverType
• type Reporter
• type TarArchiver
  • func NewTarArchiver(opts ArchiverOptions) (a *TarArchiver, err error)
  • func (a *TarArchiver) Archive(ctx context.Context, path string, rep Reporter, ...) (err error)
  • func (a *TarArchiver) Index(fn func(k string) error) error
  • func (a *TarArchiver) Unarchive(ctx context.Context, path string, rep Reporter, ...) error

Variables

var (
	//TarArchiverKey configures the one object key returned by the tar Archiver
	TarArchiverKey = "archive.tar"

	//TarArchiverPathSeparator standardizes the header path for cross platform (un)archiving
	TarArchiverPathSeparator = "/"

	//ErrNotADirectory is returned when an archiver was asked to upload a single file without dir
	ErrNotADirectory = errors.New("it is not a directory")

	//ErrNoSuchDirectory is returned when an archiver expected a directory to exist
	ErrNoSuchDirectory = errors.New("directory doesn't exist")

	//ErrEmptyDirectory is returned when the archiver expected the directory to not be empty
	ErrEmptyDirectory = errors.New("directory is empty")

	//ErrDatasetTooLarge is returned when the dataset size is above the sizelimit set in the dataset.
	ErrDatasetTooLarge = "dataset is too big, limit is %s"

	//SizeLimit is the maximum size allowed
	//@TODO: Should be based on customer details?
	SizeLimit = int64(1 * 1024 * 1024 * 1024)
)

Functions

func Copy

func Copy(ctx context.Context, dst io.Writer, src io.Reader) (int64, error)

Copy is taken from http://ixday.github.io/post/golang-cancel-copy/, slightly modified function signature: - context has been added in order to propagate cancelation - I do not return the number of bytes written, has it is not useful in my use case

Types

type ArchiverOptions

type ArchiverOptions struct {
	Type ArchiverType `json:"type"`

	TarArchiverKeyPrefix string `json:"keyPrefix"`

	SizeLimit int64 `json:"sizeLimit"`
}

ArchiverOptions contain options for all stores

type ArchiverType

type ArchiverType string

ArchiverType determines what type the object store will be

const (
	//ArchiverTypeTar uses the tar archiving format
	ArchiverTypeTar ArchiverType = "tar"
)

type Reporter

type Reporter interface {
	StartArchivingProgress(label string, total int64) func(int64)
	StopArchivingProgress()
	StartUnarchivingProgress(label string, total int64, rr io.Reader) io.Reader
	StopUnarchivingProgress()
}

Reporter describes how an archiver reports

type TarArchiver

type TarArchiver struct {
	// contains filtered or unexported fields
}

TarArchiver will archive a directory into a single tar file

func NewTarArchiver

func NewTarArchiver(opts ArchiverOptions) (a *TarArchiver, err error)

NewTarArchiver will setup the tar archiver

func (*TarArchiver) Archive

func (a *TarArchiver) Archive(ctx context.Context, path string, rep Reporter, fn func(k string, r io.ReadSeeker, nbytes int64) error) (err error)

Archive will archive a directory at 'path' into readable objects 'r' and calls 'fn' for each

func (*TarArchiver) Index

func (a *TarArchiver) Index(fn func(k string) error) error

Index calls 'fn' for all object keys that are part of the archive

func (*TarArchiver) Unarchive

func (a *TarArchiver) Unarchive(ctx context.Context, path string, rep Reporter, fn func(k string, w io.WriterAt) error) error

Unarchive will take a file system path and call 'fn' for each object that it needs for unarchiving. It writes to a temporary directory first and then moves this to the final location