fileset

Documentation

Index

• type FileSet
  • func (f *FileSet) Close() error
  • func (f *FileSet) Delete(path string)
  • func (f *FileSet) StartTag(tag string)
  • func (f *FileSet) Write(data []byte) (int, error)
  • func (f *FileSet) WriteHeader(hdr *tar.Header) error
• type Option
  • func WithMemThreshold(threshold int64) Option
  • func WithParent(parentName string) Option
  • func WithRoot(root string) Option
• type Reader
  • func (r *Reader) Close() error
  • func (r *Reader) Next() (*index.Header, error)
  • func (r *Reader) Peek() (*index.Header, error)
  • func (r *Reader) PeekTag() (*index.Tag, error)
  • func (r *Reader) Read(data []byte) (int, error)
• type Storage
  • func NewStorage(objC obj.Client, chunks *chunk.Storage) *Storage
  • func (s *Storage) New(ctx context.Context, name string, opts ...Option) *FileSet
  • func (s *Storage) NewIndexReader(ctx context.Context, fileSet, idxPrefix string) *index.Reader
  • func (s *Storage) NewIndexWriter(ctx context.Context, fileSet string) *index.Writer
  • func (s *Storage) NewReader(ctx context.Context, fileSet, idxPrefix string) *Reader
  • func (s *Storage) NewWriter(ctx context.Context, fileSet string) *Writer
• type Writer
  • func (w *Writer) Close() error
  • func (w *Writer) CopyFiles(r *Reader, pathBound ...string) error
  • func (w *Writer) CopyTags(r *Reader, tagBound ...string) error
  • func (w *Writer) StartTag(id string)
  • func (w *Writer) Write(data []byte) (int, error)
  • func (w *Writer) WriteHeader(hdr *index.Header) error

Types

type FileSet

type FileSet struct {
	// contains filtered or unexported fields
}

FileSet is a set of files. This may be a full filesystem or a subfilesystem (e.g. datum / datum set / shard).

func (*FileSet) Close

func (f *FileSet) Close() error

Close closes the file set.

func (*FileSet) Delete

func (f *FileSet) Delete(path string)

Delete deletes a file from the file set. (bryce) might need to delete ancestor directories in certain cases.

func (*FileSet) StartTag

func (f *FileSet) StartTag(tag string)

StartTag starts a tag for the next set of files.

func (*FileSet) Write

func (f *FileSet) Write(data []byte) (int, error)

Write writes to the current file in the tar stream.

func (*FileSet) WriteHeader

func (f *FileSet) WriteHeader(hdr *tar.Header) error

WriteHeader writes a tar header and prepares to accept the file's contents. (bryce) should we prevent directories from being written here?

type Option

type Option func(f *FileSet)

Option configures a file set.

func WithMemThreshold

func WithMemThreshold(threshold int64) Option

WithMemThreshold sets the memory threshold of the file set.

func WithParent

func WithParent(parentName string) Option

WithParent sets the parent of the file set.

func WithRoot

func WithRoot(root string) Option

WithRoot sets the root path of the file set.

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader reads the serialized format of a fileset.

func (*Reader) Close

func (r *Reader) Close() error

Close closes the reader.

func (*Reader) Next

func (r *Reader) Next() (*index.Header, error)

Next returns the next header, and prepares the file's content for reading.

func (*Reader) Peek

func (r *Reader) Peek() (*index.Header, error)

Peek returns the next header without progressing the reader.

func (*Reader) PeekTag

func (r *Reader) PeekTag() (*index.Tag, error)

PeekTag returns the next tag without progressing the reader.

func (*Reader) Read

func (r *Reader) Read(data []byte) (int, error)

Read reads from the current file in the tar stream.

type Storage

type Storage struct {
	// contains filtered or unexported fields
}

Storage is the abstraction that manages fileset storage.

func NewStorage

func NewStorage(objC obj.Client, chunks *chunk.Storage) *Storage

NewStorage creates a new Storage.

func (*Storage) New

func (s *Storage) New(ctx context.Context, name string, opts ...Option) *FileSet

New creates a new in-memory fileset.

func (*Storage) NewIndexReader

func (s *Storage) NewIndexReader(ctx context.Context, fileSet, idxPrefix string) *index.Reader

NewIndexReader creates a new index.Reader.

func (*Storage) NewIndexWriter

func (s *Storage) NewIndexWriter(ctx context.Context, fileSet string) *index.Writer

NewIndexWriter creates a new index.Writer.

func (*Storage) NewReader

func (s *Storage) NewReader(ctx context.Context, fileSet, idxPrefix string) *Reader

NewReader creates a new Reader.

func (*Storage) NewWriter

func (s *Storage) NewWriter(ctx context.Context, fileSet string) *Writer

NewWriter creates a new Writer.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer writes the serialized format of a fileset. The serialized format of a fileset consists of indexes and content which are both realized as compressed tar stream chunks.

func (*Writer) Close

func (w *Writer) Close() error

Close closes the writer. (bryce) is not closing the tar writer the right choice here? We cannot include it in the last file's range because it will effect the hash. Our indexing is what will exit the reading of the tar stream, not the end of tar entry (two empty 512 bytes).

func (*Writer) CopyFiles

func (w *Writer) CopyFiles(r *Reader, pathBound ...string) error

CopyFiles does a cheap copy of files from a reader to a writer. (bryce) need to handle delete operations.

func (*Writer) CopyTags

func (w *Writer) CopyTags(r *Reader, tagBound ...string) error

CopyTags does a cheap copy of tagged file data from a reader to a writer.

func (*Writer) StartTag

func (w *Writer) StartTag(id string)

StartTag starts a tag for the next set of bytes (used for the reverse index, mapping file output to datums).

func (*Writer) Write

func (w *Writer) Write(data []byte) (int, error)

Write writes to the current file in the tar stream.

func (*Writer) WriteHeader

func (w *Writer) WriteHeader(hdr *index.Header) error

WriteHeader writes a tar header and prepares to accept the file's contents.