v1data

Documentation

Index

• Constants
• Variables
• type Client
  • func NewClient(ops DataOps) *Client
  • func (c *Client) Download(ctx context.Context, bucket, key string) (io.ReadCloser, error)
  • func (c *Client) Exists(ctx context.Context, bucket, key string) (has bool, err error)
  • func (c *Client) MetadataDownload(ctx context.Context, bucket, root string) (*v1payload.Metadata, error)
  • func (c *Client) MetadataExists(ctx context.Context, bucket, root string) (bool, error)
  • func (c *Client) MetadataUpload(ctx context.Context, bucket, root string, m *v1payload.Metadata) error
  • func (c *Client) Upload(ctx context.Context, bucket, key string, body io.ReadSeeker) error
• type DataOps
• type IndexReader
  • func NewIndexReader(r io.Reader) *IndexReader
  • func (r *IndexReader) ReadKey() (Key, error)
• type IndexWriter
  • func NewIndexWriter(w io.Writer) *IndexWriter
  • func (w *IndexWriter) WriteKey(k Key) error
• type Key
  • func (k Key) ToString() string
• type KeyReader
• type KeyWriter

Constants

const (
	//LogGroup is the group for each log statement.
	LogGroup = "DataClient"
	//NoOfRetries is the amount of retries when uploading or downloading to S3.
	NoOfRetries = 2
)

const (
	//IndexObjectKey is the key of the object that contains an index of all the chunks of a dataset.
	IndexObjectKey = "index"
	//UploadPolynomal is the polynomal that is used for chunked uploading.
	UploadPolynomal = 0x3DA3358B4DC173
)

const (
	//MetadataObjectKey is the key of the object that contains an a dataset's metadata.
	MetadataObjectKey = "metadata"
)

Variables

var ZeroKey = Key{}

ZeroKey is an empty key.

Types

type Client

type Client struct {
	DataOps DataOps
}

Client holds a reference to an AWS session

func NewClient

func NewClient(ops DataOps) *Client

NewClient creates a new data client that is capable of uploading and downloading (multiple) files.

func (*Client) Download

func (c *Client) Download(ctx context.Context, bucket, key string) (io.ReadCloser, error)

Download downloads a single object.

func (*Client) Exists

func (c *Client) Exists(ctx context.Context, bucket, key string) (has bool, err error)

Exists checks if a given object key exists.

func (*Client) MetadataDownload

func (c *Client) MetadataDownload(ctx context.Context, bucket, root string) (*v1payload.Metadata, error)

MetadataDownload downloads a dataset's metadata.

func (*Client) MetadataExists

func (c *Client) MetadataExists(ctx context.Context, bucket, root string) (bool, error)

MetadataExists checks if the metadata object exists.

func (*Client) MetadataUpload

func (c *Client) MetadataUpload(ctx context.Context, bucket, root string, m *v1payload.Metadata) error

MetadataUpload uploads a dataset's metadata.

func (*Client) Upload

func (c *Client) Upload(ctx context.Context, bucket, key string, body io.ReadSeeker) error

Upload uploads a single object.

type DataOps

type DataOps interface {
	Upload(ctx context.Context, bucket, key string, body io.ReadSeeker) error
	Download(ctx context.Context, bucket, key string) (body io.ReadCloser, err error)
	Exists(ctx context.Context, bucket, key string) (exists bool, err error)
}

DataOps is an interface to a set of data operations. The interface can be implemented to store / retrieve data from different data backends.

type IndexReader

type IndexReader struct {
	// contains filtered or unexported fields
}

IndexReader can be used to read keys from the "index" object.

func NewIndexReader

func NewIndexReader(r io.Reader) *IndexReader

NewIndexReader creates a new IndexReader.

func (*IndexReader) ReadKey

func (r *IndexReader) ReadKey() (Key, error)

ReadKey reads Keys from the provided io.Reader.

type IndexWriter

type IndexWriter struct {
	// contains filtered or unexported fields
}

IndexWriter can be used to write keys to the "index" object.

func NewIndexWriter

func NewIndexWriter(w io.Writer) *IndexWriter

NewIndexWriter creates a new IndexWriter.

func (*IndexWriter) WriteKey

func (w *IndexWriter) WriteKey(k Key) error

WriteKey writes a Key to the io.WriteCloser.

type Key

type Key [sha256.Size]byte

Key is the identifier of a chunk of data.

func (Key) ToString

func (k Key) ToString() string

ToString returns the string representation of a key.

type KeyReader

type KeyReader interface {
	ReadKey() (Key, error)
}

KeyReader can be implemented by objects capable of reading Keys.

type KeyWriter

type KeyWriter interface {
	WriteKey(Key) error
}

KeyWriter can be implemented by objects capable of writing Keys.