ztream

Documentation

Overview

Package ztream implements zip files for streaming data to.

Index

• Variables
• type CorruptError
  • func (e *CorruptError) Error() string
• type Entry
• type Options
• type Stream
  • func Create(path string, opt Options) (s *Stream, err error)
  • func Open(path string, opt Options) (s *Stream, err error)
  • func (s *Stream) Append(name string, data []byte) (Entry, error)
  • func (s *Stream) Close() error
  • func (s *Stream) Contents() (entries []Entry, err error)
  • func (s *Stream) Read(e Entry, buf []byte) (err error)
  • func (s *Stream) Sync() error
  • func (s *Stream) Wipe(name string) error
• type Verifier
• type VerifyError

Variables

var (
	ErrStreamFull        = errors.New("zstream: the provided data does not fit in the stream")
	ErrBuffNotSufficient = errors.New("zstream: the provided buffer is not long enough to read the data")
)

var DefaultOptions = Options{
	FileSize:             1 << 27,
	Verifier:             nil,
	SampleCompressSize:   1024 * 4,
	CompressionThreshold: 0.75,
	CompressionLevel:     flate.BestSpeed,
}

Types

type CorruptError

type CorruptError struct {
	File   string
	Offset int
	Err    string
}

A CorruptError is returned if any unexpected data is found in the file.

func (*CorruptError) Error

func (e *CorruptError) Error() string

type Entry

type Entry struct {
	Name   string
	Offset int32 // Offset to the actuall data, not the header.
	// The data is deflated if and only if CompressedSize < UncompressedSize
	CompressedSize   int32
	UncompressedSize int32
}

type Options

type Options struct {
	// The size of the zip file that should be allocated when creating a new file. Has no
	// effect when opening an existing ztream.
	FileSize int32
	// If non nill used to verify all exisiting data in a ztream that is opened. Has no
	// effect when creating a new ztream.
	Verifier Verifier
	// If SampleCompressSize > 0 the ztream tries to compress part of any appended data to
	// see if it is worthwile to compress the data to save space. If <= 0 compression is disabled.
	// When compression is enabled the memory usage is increased since a buffer must be maintained
	// for the compressed data to avoid disk seeks.
	SampleCompressSize int
	// Only compress the data if it is shorter than CompressionThreshold*len, else write the
	// uncompressed data.
	CompressionThreshold float32
	// CompressionLevel to use - specified as given by the flate package.
	CompressionLevel int
}

Options to configure the ztream.

type Stream

type Stream struct {
	// contains filtered or unexported fields
}

A Stream can only b

func Create

func Create(path string, opt Options) (s *Stream, err error)

Create h

func Open

func Open(path string, opt Options) (s *Stream, err error)

Open op

func (*Stream) Append

func (s *Stream) Append(name string, data []byte) (Entry, error)

Append tries to append a file with the given name and data to the file. NOTE that this will NOT commit the data to disk, Sync() MUST be called, this allows seceral data pieces to be written at once to avoid disk seek. If it does not fit ErrStreamFull is returned.

func (*Stream) Close

func (s *Stream) Close() error

Close ensures that the written file is a valid zip and everything is commited to disk.

func (*Stream) Contents

func (s *Stream) Contents() (entries []Entry, err error)

Contents returns a list of the file names contained in the ztream, in the same order as stored on disk. Note that data appended but not Synced will not be returned here.

func (*Stream) Read

func (s *Stream) Read(e Entry, buf []byte) (err error)

Read out the Entry, optimized for sequential reading of entries after each other. Note that buf must be large enough to hold the uncompressed data or it is an error of type ErrBuffNotSufficient.

func (*Stream) Sync

func (s *Stream) Sync() error

Sync flushes out all the Appended data to the underlying disk (writes and syncs). Note that the end of file dictionary will not be written until Close is called, since data can be recovered by scanning. Note that after an error here 0 or more of the data pieces Appended since last Sync may be missing from the file - they must be read and checked to ensure they are present.

func (*Stream) Wipe

func (s *Stream) Wipe(name string) error

Wipe removes the given file and overwrites the data twice to ensure that it is gone. Note that this is an expensive call. It is not expected to be used often so ok that it is slow.

type Verifier

type Verifier interface {
	io.Writer

	Reset()
	// Match should return true if the name provided is a valid name for the data stream written
	// to the Verifier since the last call to Reset.
	Match(name string) bool
}

A Verifier that is used to check that the data stored in the ztream is stored under the same names as the Verifier deterministically would give.

type VerifyError

type VerifyError struct {
}

A VerifyError is returned if the name stored in the ztream does not match the name given by the Verifier.