dbin

README

StreamingFast dbin Library

dbin is a simple file storage format to pack a stream of protobuf messages. It is part of StreamingFast.

Usage

See example usage in merger

Latest file format (v1)

First four magic bytes:

• 'd', 'b', 'i', 'n'

Next single byte:

• file format version, current is 0x01

Next 2 bytes are big-endian uint16, the length of the proto definition type, to follow:

• 0x1900

Next bytes are the name of the proto def:

• "sf.ethereum.type.v1.Block"

Rest of the file, a sequence of:

• Length-prefixed messages, with each length specified as 4 bytes big-endian uint32.
• Followed by message of that length

EOF reached when no more bytes exist after the last message boundary.

Prior file format (v0)

Version 0 of the dbin for mat was:

First four magic bytes:

• 'd', 'b', 'i', 'n'

Next single byte:

• file format version, was 0x00.

Next 5 bytes were the content type, including 3 letters and 2 numbers.

Rest of the file, a sequence of:

• Length-prefixed messages, with each length specified as 4 bytes big-endian uint32.
• Followed by message of that length

EOF reached when no more bytes exist after the last message boundary.

Contributing

Issues and PR in this repo related strictly to the dbin library.

Report any protocol-specific issues in their respective repositories

Please first refer to the general StreamingFast contribution guide, if you wish to contribute to this code base.

License

Apache 2.0

Documentation

Index

• type Reader
  • func NewFileReader(filename string) (*Reader, error)
  • func NewReader(r io.Reader) *Reader
  • func (r *Reader) Close() error
  • func (r *Reader) ReadHeader() (contentType string, err error)
  • func (r *Reader) ReadMessage() ([]byte, error)
• type Writer
  • func NewWriter(w io.Writer) *Writer
  • func (w *Writer) Close() error
  • func (w *Writer) WriteHeader(contentType string) error
  • func (w *Writer) WriteMessage(msg []byte) error

Types

type Reader

type Reader struct {
	io.Reader
	// contains filtered or unexported fields
}

func NewFileReader

func NewFileReader(filename string) (*Reader, error)

func NewReader

func NewReader(r io.Reader) *Reader

func (*Reader) Close

func (r *Reader) Close() error

func (*Reader) ReadHeader

func (r *Reader) ReadHeader() (contentType string, err error)

func (*Reader) ReadMessage

func (r *Reader) ReadMessage() ([]byte, error)

ReadMessage reads next message from byte stream

type Writer

type Writer struct {
	io.Writer
	// contains filtered or unexported fields
}

func NewWriter

func NewWriter(w io.Writer) *Writer

func (*Writer) Close

func (w *Writer) Close() error

func (*Writer) WriteHeader

func (w *Writer) WriteHeader(contentType string) error

func (*Writer) WriteMessage

func (w *Writer) WriteMessage(msg []byte) error