kzip

Documentation

Overview

Package kzip implements the kzip compilation storage file format.

The package exports two types of interest: A kzip.Reader can be used to read the contents of an existing kzip archive, and a kzip.Writer can be used to construct a new kzip archive.

Reading an Archive:

r, err := kzip.NewReader(file, size)
...

// Look up a compilation record by its digest.
unit, err := r.Lookup(unitDigest)
...

// Scan all the compilation records stored.
err := r.Scan(func(unit *kzip.Unit) error {
   if hasInterestingProperty(unit) {
      doStuffWith(unit)
   }
   return nil
})

// Open a reader for a stored file.
rc, err := r.Open(fileDigest)
...
defer rc.Close()

// Read the complete contents of a stored file.
bits, err := r.ReadAll(fileDigest)
...

Writing an Archive:

w, err := kzip.NewWriter(file)
...

// Add a compilation record and (optional) index data.
udigest, err := w.AddUnit(unit, nil)
...

// Add file contents.
fdigest, err := w.AddFile(file)
...

Index

• Variables
• func Scan(f File, scan func(*Reader, *Unit) error, opts ...ScanOption) error
• type Encoding
  • func EncodingFor(v string) (Encoding, error)
  • func (e Encoding) String() string
• type File
• type Reader
  • func NewReader(r io.ReaderAt, size int64) (*Reader, error)
  • func (r *Reader) Lookup(unitDigest string) (*Unit, error)
  • func (r *Reader) Open(fileDigest string) (io.ReadCloser, error)
  • func (r *Reader) ReadAll(fileDigest string) ([]byte, error)
  • func (r *Reader) Scan(f func(*Unit) error, opts ...ScanOption) error
• type ScanOption
  • func ReadConcurrency(n int) ScanOption
• type Unit
• type Writer
  • func NewWriteCloser(wc io.WriteCloser, options ...WriterOption) (*Writer, error)
  • func NewWriter(w io.Writer, options ...WriterOption) (*Writer, error)
  • func (w *Writer) AddFile(r io.Reader) (string, error)
  • func (w *Writer) AddUnit(cu *apb.CompilationUnit, index *apb.IndexedCompilation_Index) (string, error)
  • func (w *Writer) Close() error
• type WriterOption
  • func WithEncoding(e Encoding) WriterOption

Variables

var ErrDigestNotFound = errors.New("digest not found")

ErrDigestNotFound is returned when a requested compilation unit or file digest is not found.

var ErrUnitExists = errors.New("unit already exists")

ErrUnitExists is returned by AddUnit when adding the same compilation multiple times.

Functions

func Scan

func Scan(f File, scan func(*Reader, *Unit) error, opts ...ScanOption) error

Scan is a convenience function that creates a *Reader from f and invokes its Scan method with the given callback. Each invocation of scan is passed the reader associated with f, along with the current compilation unit.

Types

type Encoding

type Encoding int

Encoding describes how compilation units will be encoded when written to a kzip.

const (
	// EncodingJSON specifies to use JSON encoding
	EncodingJSON Encoding = 1
	// EncodingProto specifies to use Proto encoding
	EncodingProto Encoding = 2
	// EncodingAll specifies to encode using all known encodings
	EncodingAll Encoding = EncodingJSON | EncodingProto
)

func EncodingFor

func EncodingFor(v string) (Encoding, error)

EncodingFor converts a string to an Encoding.

func (Encoding) String

func (e Encoding) String() string

String stringifies an Encoding

type File

type File interface {
	io.ReaderAt
	io.Seeker
}

A File represents the file capabilities needed to scan a kzip file.

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

A Reader permits reading and scanning compilation records and file contents stored in a .kzip archive. The Lookup and Scan methods are mutually safe for concurrent use by multiple goroutines.

func NewReader

func NewReader(r io.ReaderAt, size int64) (*Reader, error)

NewReader constructs a new Reader that consumes zip data from r, whose total size in bytes is given.

func (*Reader) Lookup

func (r *Reader) Lookup(unitDigest string) (*Unit, error)

Lookup returns the specified compilation from the archive, if it exists. If the requested digest is not in the archive, ErrDigestNotFound is returned.

func (*Reader) Open

func (r *Reader) Open(fileDigest string) (io.ReadCloser, error)

Open opens a reader on the contents of the specified file digest. If the requested digest is not in the archive, ErrDigestNotFound is returned. The caller must close the reader when it is no longer needed.

func (*Reader) ReadAll

func (r *Reader) ReadAll(fileDigest string) ([]byte, error)

ReadAll returns the complete contents of the file with the specified digest. It is a convenience wrapper for Open followed by ioutil.ReadAll.

func (*Reader) Scan

func (r *Reader) Scan(f func(*Unit) error, opts ...ScanOption) error

Scan scans all the compilations stored in the archive, and invokes f for each compilation record. If f reports an error, the scan is terminated and that error is propagated to the caller of Scan. At most 1 invocation of f will occur at any one time.

type ScanOption

type ScanOption interface {
	// contains filtered or unexported methods
}

A ScanOption configures the behavior of scanning a kzip file.

func ReadConcurrency

func ReadConcurrency(n int) ScanOption

ReadConcurrency returns a ScanOption that configures the max concurrency of reading compilation units within a kzip archive.

type Unit

type Unit struct {
	Digest string
	Proto  *apb.CompilationUnit
	Index  *apb.IndexedCompilation_Index
}

A Unit represents a compilation record read from a kzip archive.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

A Writer permits construction of a .kzip archive.

func NewWriteCloser

func NewWriteCloser(wc io.WriteCloser, options ...WriterOption) (*Writer, error)

NewWriteCloser behaves as NewWriter, but arranges that when the *Writer is closed it also closes wc.

func NewWriter

func NewWriter(w io.Writer, options ...WriterOption) (*Writer, error)

NewWriter constructs a new empty Writer that delivers output to w. The AddUnit and AddFile methods are safe for use by concurrent goroutines.

func (*Writer) AddFile

func (w *Writer) AddFile(r io.Reader) (string, error)

AddFile copies the complete contents of r into the archive as a new file entry, returning the hex-encoded SHA256 digest of the file's contents.

func (*Writer) AddUnit

func (w *Writer) AddUnit(cu *apb.CompilationUnit, index *apb.IndexedCompilation_Index) (string, error)

AddUnit adds a new compilation record to be added to the archive, returning the hex-encoded SHA256 digest of the unit's contents. It is legal for index to be nil, in which case no index terms will be added.

If the same compilation is added multiple times, AddUnit returns the digest of the duplicated compilation along with ErrUnitExists to all callers after the first. The existing unit is not modified.

func (*Writer) Close

func (w *Writer) Close() error

Close closes the writer, flushing any remaining unwritten data out to the underlying zip file. It is safe to close w arbitrarily many times; all calls after the first will report nil.

type WriterOption

type WriterOption func(*Writer)

WriterOption describes options when creating a Writer

func WithEncoding

func WithEncoding(e Encoding) WriterOption

WithEncoding sets the encoding to be used by a Writer