pack

README

Proto-Pack format Version 2.0

Header

name	type	description
magic	byte[16]	"ProtoPack\r\n2.0\n\0"

The header contains both types of new-lines, which is common in file headers to detect corruption caused by automatic new-line conversions. The header is followed by arbitrary number of variable-sized chunks. Chunks can be either object instance or type definition depending on the sign of the size field (encoded as protobuf's variable-length zigzag).

Object instance chunk (size>0)

name	type	description
size	sint32	Total size of the chunk excluding this size field.
parent	sint32	If >= 0: New root object instance. If < 0: Relative index of chunk which is the parent.
type	sint32	If > 0: Type index. The object has no children. If < 0: Negated type index if it may have children. If == 0: Terminates children list of the parent.
data	byte[]	Proto message.

Objects may form a tree. Parent object is specified as back-reference to previous chunk (-1 is the previous chunk). The referenced chunk must also be an object and it must allow addition of children. Values of parent>=0 define a root object. Values of parent>0 may be used in the future to add information to the root object.

Negated type index denotes object which may have children. The children list must be terminated by chunk with type==0. Object with positive type can not have children and does not need terminator.

As an optimization, if size is small enough to cover only the parent field, type field is implicitly set to 0 (i.e. it is list terminator).

Type definition chunk (size<0)

name	type	description
size	sint32	Negated total size of the chunk excluding this size field.
name	string	Fully qualified type name as proto string.
desc	byte[]	Proto descriptor of one message type of the given name.

The format is self-describing. All objects are stored as typed proto messages, where the type must be first described by type definition chunk. Types are assigned indices based on the order in the file (starting with 1).

Documentation

Overview

Package pack provides methods to deal with self describing files of proto data.

The file format consists of a magic marker, followed by a Header. After that is a repeated sequence of uvarint length, tag and matching encoded message pair. Some section tags will also be followed by a string. The tag 0 is special, and marks a type entry, the body will be a descriptor.DescriptorProto.

Index

• Constants
• Variables
• func CheckMagic(from *bufio.Reader) bool
• func Read(ctx context.Context, from io.Reader, events Events, forceDynamic bool) error
• type Dynamic
  • func (d Dynamic) Format(f fmt.State, r rune)
  • func (Dynamic) ProtoMessage()
  • func (d *Dynamic) Reset()
  • func (d Dynamic) String() string
  • func (d *Dynamic) Unmarshal(data []byte) error
• type ErrUnknownType
  • func (e ErrUnknownType) Error() string
• type ErrUnsupportedVersion
  • func (e ErrUnsupportedVersion) Error() string
• type Events
• type Version
• type Writer
  • func NewWriter(to io.Writer) (*Writer, error)
  • func (w *Writer) BeginChildGroup(ctx context.Context, msg proto.Message, parentID uint64) (id uint64, err error)
  • func (w *Writer) BeginGroup(ctx context.Context, msg proto.Message) (id uint64, err error)
  • func (w *Writer) ChildObject(ctx context.Context, msg proto.Message, parentID uint64) error
  • func (w *Writer) EndGroup(ctx context.Context, id uint64) error
  • func (w *Writer) Object(ctx context.Context, msg proto.Message) error

Constants

const (
	// ErrIncorrectMagic is the error returned when the file header is not matched.
	ErrIncorrectMagic = fault.Const("Incorrect pack magic header")
)

Variables

var (
	// MinMajorVersion is the current minimum supported major version of pack files.
	MinMajorVersion = 2

	// MaxMajorVersion is the current maximum supported major version of pack files.
	MaxMajorVersion = 2
)

Functions

func CheckMagic

func CheckMagic(from *bufio.Reader) bool

CheckMagic checks whether the given stream starts with a pack header. This function will peek the header from the stream without adjusting it's position. The buffer on the reader needs to be at least maxHeaderSize bytes.

func Read

func Read(ctx context.Context, from io.Reader, events Events, forceDynamic bool) error

Read reads the pack file from the supplied stream. This function will read the header from the stream, adjusting it's position. It may read extra bytes from the stream into an internal buffer.

Types

type Dynamic

type Dynamic struct {
	Desc   *descriptor.DescriptorProto
	Fields map[string]interface{}
	// contains filtered or unexported fields
}

Dynamic is a dynamic proto message.

func (Dynamic) Format

func (d Dynamic) Format(f fmt.State, r rune)

Format implements fmt.Formatter to print the full value chain.

func (Dynamic) ProtoMessage

func (Dynamic) ProtoMessage()

func (*Dynamic) Reset

func (d *Dynamic) Reset()

func (Dynamic) String

func (d Dynamic) String() string

func (*Dynamic) Unmarshal

func (d *Dynamic) Unmarshal(data []byte) error

type ErrUnknownType

type ErrUnknownType struct{ TypeName string }

ErrUnknownType is the error returned by Reader.Unmarshal() when it encounters an unknown proto type.

func (ErrUnknownType) Error

func (e ErrUnknownType) Error() string

type ErrUnsupportedVersion

type ErrUnsupportedVersion struct{ Version Version }

ErrUnsupportedVersion is the error returned when the header version is one this package cannot handle.

func (ErrUnsupportedVersion) Error

func (e ErrUnsupportedVersion) Error() string

type Events

type Events interface {
	// BeginGroup is called to start a new root group with the given identifier.
	BeginGroup(ctx context.Context, msg proto.Message, id uint64) error

	// BeginChildGroup is called to start a new group with the given identifier
	// as a child of the group with the parent identifier.
	BeginChildGroup(ctx context.Context, msg proto.Message, id, parentID uint64) error

	// EndGroup finalizes the group with the given identifier. It is illegal to
	// attempt to add children to the group after this is called.
	// EndGroup should be closed immediately after the last child has been added
	// to the group.
	EndGroup(ctx context.Context, id uint64) error

	// Object is called to declare an object outside of any group.
	Object(ctx context.Context, msg proto.Message) error

	// ChildObject is called to declare an object in the group with the given
	// identifier.
	ChildObject(ctx context.Context, msg proto.Message, parentID uint64) error
}

Events describes the events used to construct groups and objects that are stored in a proto-pack stream.

type Version

type Version struct {
	Major int // Major version is incremented for format breaking changes.
	Minor int // Minor version is incremented backwards compatible changes.
}

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer is the type for a pack file writer. They should only be constructed by NewWriter.

func NewWriter

func NewWriter(to io.Writer) (*Writer, error)

NewWriter constructs and returns a new Writer that writes to the supplied output stream. This method will write the packfile magic and header to the underlying stream.

func (*Writer) BeginChildGroup

func (w *Writer) BeginChildGroup(ctx context.Context, msg proto.Message, parentID uint64) (id uint64, err error)

BeginChildGroup is called to start a new group as a child of the group with the parent identifier.

func (*Writer) BeginGroup

func (w *Writer) BeginGroup(ctx context.Context, msg proto.Message) (id uint64, err error)

BeginGroup is called to start a new root group.

func (*Writer) ChildObject

func (w *Writer) ChildObject(ctx context.Context, msg proto.Message, parentID uint64) error

ChildObject is called to declare an object in the group with the given identifier.

func (*Writer) EndGroup

func (w *Writer) EndGroup(ctx context.Context, id uint64) error

EndGroup finalizes the group with the given identifier. It is illegal to attempt to add children to the group after this is called. EndGroup should be closed immediately after the last child has been added to the group.

func (*Writer) Object

func (w *Writer) Object(ctx context.Context, msg proto.Message) error

Object is called to declare an object outside of any group.