riegeli

Documentation

Overview

Package riegeli implements a Reader and Writer for the Riegeli records format.

C++ implementation: https://github.com/google/riegeli Format spec: https://github.com/google/riegeli/blob/master/doc/riegeli_records_file_format.md

Index

• Constants
• Variables
• type CompressionType
  • func BrotliCompression(level int) CompressionType
  • func ZSTDCompression(level int) CompressionType
• type ReadSeeker
  • func NewReadSeeker(r io.ReadSeeker) ReadSeeker
• type Reader
  • func NewReader(r io.Reader) Reader
• type RecordPosition
• type Writer
  • func NewWriter(w io.Writer, opts *WriterOptions) *Writer
  • func NewWriterAt(w io.Writer, pos int, opts *WriterOptions) *Writer
  • func (w *Writer) Close() error
  • func (w *Writer) Flush() error
  • func (w *Writer) Put(rec []byte) error
  • func (w *Writer) PutProto(msg proto.Message) error
• type WriterOptions
  • func ParseOptions(s string) (*WriterOptions, error)
  • func (o *WriterOptions) String() string

Constants

const (
	DefaultChunkSize uint64 = 1 << 20

	DefaultBrotliLevel = 9
	DefaultZSTDLevel   = 9
)

Defaults for the WriterOptions.

Variables

var DefaultCompression = BrotliCompression(DefaultBrotliLevel)

DefaultCompression is the default Compression for the WriterOptions.

Types

type CompressionType

type CompressionType interface {
	fmt.Stringer
	// contains filtered or unexported methods
}

CompressionType is the type of compression used for encoding Riegeli chunks.

var (
	// NoCompression indicates that no compression will be used to encode chunks.
	NoCompression CompressionType = &compressionLevel{noCompression, 0}
)

func BrotliCompression

func BrotliCompression(level int) CompressionType

BrotliCompression returns a CompressionType for Brotli compression with the given quality level. If level < 0 || level > 11, then the DefaultBrotliLevel will be used.

func ZSTDCompression

func ZSTDCompression(level int) CompressionType

ZSTDCompression returns a CompressionType for zstd compression with the given compression level. If level < 0 || level > 22 (outside of the levels specified by the zstdlib spec), then the DefaultZSTDLevel will be used.

type ReadSeeker

type ReadSeeker interface {
	Reader

	// Seek interprets pos as an offset to a record within the Riegeli file.  pos
	// must be between 0 and the file's size.  If pos is between records, Seek will
	// position the reader to the next record in the file.
	Seek(pos int64) error

	// SeekToRecord seeks to the given RecordPosition.
	SeekToRecord(pos RecordPosition) error
}

ReadSeeker is a Riegeli records file reader able to seek to arbitrary positions.

func NewReadSeeker

func NewReadSeeker(r io.ReadSeeker) ReadSeeker

NewReadSeeker returns a Riegeli ReadSeeker for r.

type Reader

type Reader interface {
	// RecordsMetadata returns the optional metadata from the underlying Riegeli
	// file.  If not found, an empty RecordsMetadata is returned and err == nil.
	RecordsMetadata() (*rmpb.RecordsMetadata, error)

	// Next reads and returns the next Riegeli record from the underlying io.Reader.
	Next() ([]byte, error)

	// NextProto reads, unmarshals, and returns the next proto.Message from the
	// underlying io.Reader.
	NextProto(msg proto.Message) error

	// Position returns the current position of the Reader.
	Position() (RecordPosition, error)
}

Reader is a sequential Riegeli records file reader.

func NewReader

func NewReader(r io.Reader) Reader

NewReader returns a Riegeli Reader for r.

type RecordPosition

type RecordPosition struct {
	// ChunkBegin is the starting offset of a chunk within a Riegeli file.
	ChunkBegin int64

	// RecordIndex is the index of a record within the chunk starting at
	// ChunkBegin.
	RecordIndex int64
}

A RecordPosition is a pointer to the starting offset of a record within a Riegeli file.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer is a Riegeli records file writer.

func NewWriter

func NewWriter(w io.Writer, opts *WriterOptions) *Writer

NewWriter returns a Riegeli Writer for a new Riegeli file to be written to w.

func NewWriterAt

func NewWriterAt(w io.Writer, pos int, opts *WriterOptions) *Writer

NewWriterAt returns a Riegeli Writer at the given byte offset within w.

func (*Writer) Close

func (w *Writer) Close() error

Close releases all resources associated with Writer. Any buffered records will be flushed before releasing any resources.

func (*Writer) Flush

func (w *Writer) Flush() error

Flush writes any buffered records to the underlying io.Writer.

func (*Writer) Put

func (w *Writer) Put(rec []byte) error

Put writes/buffers the given []byte as a Riegili record.

func (*Writer) PutProto

func (w *Writer) PutProto(msg proto.Message) error

PutProto writes/buffers the given proto.Message as a Riegili record.

type WriterOptions

type WriterOptions struct {
	// Desired uncompressed size of a chunk which groups records.
	ChunkSize uint64

	// Compression is the type of compression used for encoding chunks.
	Compression CompressionType

	// Transpose determines whether Protocol Buffer messages have their component
	// key-value entries encoded in separate buffers for better compression.
	Transpose bool
}

WriterOptions customizes the behavior of a Riegeli Writer.

func ParseOptions

func ParseOptions(s string) (*WriterOptions, error)

ParseOptions decodes a WriterOptions from text:

options ::= option? ("," option?)*
option ::=
  "default" |
  "transpose" (":" ("true" | "false"))? |
  "uncompressed" |
  "brotli" (":" brotli_level)? |
  "zstd" (":" zstd_level)? |
  "chunk_size" ":" chunk_size
brotli_level ::= integer 0..11 (default 9)
zstd_level ::= integer 0..22 (default 9)
chunk_size ::= positive integer

func (*WriterOptions) String

func (o *WriterOptions) String() string

String encodes the WriterOptions as text.