gblob

package module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 3, 2024 License: MIT Imports: 4 Imported by: 6

README

Go Blob

Go Reference Build Status Go Report Card

A package that provides utilities for writing and reading primitive types to and from byte sequences.

User's Guide

For a more complete documentation, check the GoDocs.

Block API

The Block API allows one to place values of concrete primitive types at specific offsets inside a byte slice. It is up to the caller to ensure that the slice has the necessary size.

Example:

offset := 4
block := make(gblob.LittleEndianBlock, 32)
block.SetFloat32(offset, 3.5)
fmt.Println(block.Float32(offset))

This API is similar to Go's built-in binary.ByteOrder. The difference here is that the gblob API is slightly more compact, it has helper functions for more primitive types and allows one to pass an offset into the byte slice for better readability.

There are two implementations available - LittleEndianBlock and BigEndianBlock, depending on the desired byte order.

TypedWriter / TypedReader API

The TypedWriter API allows one to write concrete primitive types to an io.Writer. No padding or alignment is added to the output.

Example:

var buffer bytes.Buffer
writer := gblob.NewLittleEndianWriter(&buffer)
writer.WriteUint64(0x13743521FA954321)

There are two implementations available - NewLittleEndianWriter and NewBigEndianWriter, depending on the desired byte order.

The TypedReader API allows one to read concrete primitive types from an io.Reader.

Example:

buffer := bytes.NewBuffer([]uint8{
  0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 0x20,
})
reader := gblob.NewLittleEndianReader(buffer)
value, err := reader.ReadUint64()

There are two implementations available - NewLittleEndianReader and NewBigEndianReader, depending on the desired byte order.

PackedEncoder / PackedDecoder API

The PackedEncoder API allows one to marshal a data structure to a binary sequence.

Example:

var buffer bytes.Buffer
gblob.NewLittleEndianPackedEncoder(&buffer).Encode(struct{
  A float32
  B uint64
}{
  A: 3.14,
  B: 0xFF003344FF003344,
})

There are two implementations available - NewLittleEndianPackedEncoder and NewBigEndianPackedEncoder, depending on the desired byte order.

This is similar to Go's bytes.Write, except that it supports slices, maps and strings.

The PackedDecoder API allows one to unmarshal a data structure from a binary sequence that was previously marshaled through the PackedEncoder API.

Example:

buffer := bytes.NewBuffer([]uint8{
  0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 0x20,
})
var target uint64
gblob.NewLittleEndianPackedDecoder(buffer).Decode(&target)

There are two implementations available - NewLittleEndianPackedDecoder and NewBigEndianPackedDecoder, depending on the desired byte order.

This is similar to Go's bytes.Read, except that it supports slices, maps and strings.

Performance

Following are some benchmark results. They compare this library against Go's binary and gob packages, since those are closest in terms of features. Results are based on the following hardware:

goos: linux
goarch: amd64
pkg: github.com/mokiat/gblob
cpu: AMD Ryzen 7 3700X 8-Core Processor

You can find the benchmark tests in the bench_test.go file. You can run them with the following command:

go test -benchmem -run=^$ -bench ^Benchmark github.com/mokiat/gblob

Adjust the ^Benchmark regex to focus on specific benchmark sets and use the -cpuprofile bench.pprof flag if profiling is desired.

Take the results that follow with a grain of salt. Do your own measurements for your particular use case.

Block API

Following is a benchmark comparison between Block and Go's binary.ByteOrder API.

Approach Time per Operation Allocated Memory per Operation Allocation Count per Operation
Block 1493 ns/op 0 B/op 0 allocs/op
binary.ByteOrder 1496 ns/op 0 B/op 0 allocs/op

Both APIs provide similar performance so it boils down to ease of use of and personal preference.

TypedWriter

Following is a benchmark comparison between TypedWriter and Go's binary.Write functions.

Approach Time per Operation Allocated Memory per Operation Allocation Count per Operation
TypedWriter 7753 ns/op 0 B/op 0 allocs/op
binary.Write 39768 ns/op 4096 B/op 1024 allocs/op

The TypedWriter does not allocate any memory per write operation and it also runs about 3-4 times faster. In reality, it allocates an initial buffer of size 8 bytes that it reuses to achieve these results.

TypedReader

Following is a benchmark comparison between TypedReader and Go's binary.Read functions.

Approach Time per Operation Allocated Memory per Operation Allocation Count per Operation
TypedReader 11595 ns/op 0 B/op 0 allocs/op
binary.Read 53088 ns/op 4096 B/op 1024 allocs/op

The TypedReader does not allocate any memory per read and it also runs 4-5 times faster. This is again achieved by having the TypedReader allocate an initial buffer of 8 bytes that it reuses.

PackedEncoder

When serializing larger types with PackedEncoder vs binary.Write, the performance difference is negligible, though the memory aspect remains the same as before. More interesting in this case is the comparison with gob.Encoder, due to a similar feature set.

WARNING: The scenario that is compared is the one where a new instance of an Encoder is created for each Encode performed (i.e. gob.NewEncoder(out).Encode(&target)). This is arguably the more common scenario (saving an asset / writing a response). When a gob.Encoder is reused to encode multiple sequences of items to a stream, it is significantly faster than PackedEncoder, likely due to caching.

Approach Time per Operation Allocated Memory per Operation Allocation Count per Operation
PackedEncoder 7644113 ns/op 179158 B/op 3072 allocs/op
gob.Encoder 11349161 ns/op 2408841 B/op 38912 allocs/op

Here the gob.Encoder performs worse, especially when memory is concerned.

PackedDecoder

Following is the comparison between PackedDecoder and gob.Decoder.

WARNING: The scenario that is compared is the one where a new instance of a Decoder is created for each Decode performed (i.e. gob.NewDecoder(in).Decode(&target)). This is arguably the more common scenario (loading an asset / reading a request). When a gob.Decoder is reused to read multiple sequences of items from a stream, it is significantly faster than PackedDecoder, likely due to caching.

Approach Time per Operation Allocated Memory per Operation Allocation Count per Operation
PackedDecoder 17483112 ns/op 2261422 B/op 5120 allocs/op
gob.Decoder 45206563 ns/op 11466272 B/op 236565 allocs/op

The gob.Decoder performs werse, especially when memory is concerned.

Documentation

Overview

Package gblob (Go Blob) provides utilities for writing and reading binary primitives. While Go does have encoding/binary and encoding/gob packages, they have certain limitations that this package attempts to overcome at the cost of other limitations.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type BigEndianBlock 

type BigEndianBlock []byte

BigEndianBlock represents a fixed-size block of bytes that holds values encoded in Big Endian order.

func (BigEndianBlock) Float32 

func (b BigEndianBlock) Float32(offset int) float32

Float32 returns the float32 value at the specified offset.

func (BigEndianBlock) Float64 

func (b BigEndianBlock) Float64(offset int) float64

Float64 returns the float64 value at the specified offset.

func (BigEndianBlock) Int16 

func (b BigEndianBlock) Int16(offset int) int16

Int16 returns the int16 value at the specified offset.

func (BigEndianBlock) Int32 

func (b BigEndianBlock) Int32(offset int) int32

Int32 returns the int32 value at the specified offset.

func (BigEndianBlock) Int64 

func (b BigEndianBlock) Int64(offset int) int64

Int64 returns the int64 value at the specified offset.

func (BigEndianBlock) Int8 

func (b BigEndianBlock) Int8(offset int) int8

Int8 returns the int8 value at the specified offset.

func (BigEndianBlock) SetFloat32 

func (b BigEndianBlock) SetFloat32(offset int, value float32)

SetFloat32 places the float32 value at the specified offset.

func (BigEndianBlock) SetFloat64 

func (b BigEndianBlock) SetFloat64(offset int, value float64)

SetFloat64 places the float64 value at the specified offset.

func (BigEndianBlock) SetInt16 

func (b BigEndianBlock) SetInt16(offset int, value int16)

SetInt16 places the int16 value at the specified offset.

func (BigEndianBlock) SetInt32 

func (b BigEndianBlock) SetInt32(offset int, value int32)

SetInt32 places the int32 value at the specified offset.

func (BigEndianBlock) SetInt64 

func (b BigEndianBlock) SetInt64(offset int, value int64)

SetInt64 places the int64 value at the specified offset.

func (BigEndianBlock) SetInt8 

func (b BigEndianBlock) SetInt8(offset int, value int8)

SetInt8 places the int8 value at the specified offset.

func (BigEndianBlock) SetUint16 

func (b BigEndianBlock) SetUint16(offset int, value uint16)

SetUint16 places the uint16 value at the specified offset.

func (BigEndianBlock) SetUint32 

func (b BigEndianBlock) SetUint32(offset int, value uint32)

SetUint32 places the uint32 value at the specified offset.

func (BigEndianBlock) SetUint64 

func (b BigEndianBlock) SetUint64(offset int, value uint64)

SetUint64 places the uint64 value at the specified offset.

func (BigEndianBlock) SetUint8 

func (b BigEndianBlock) SetUint8(offset int, value uint8)

SetUint8 places the uint8 value at the specified offset.

func (BigEndianBlock) Uint16 

func (b BigEndianBlock) Uint16(offset int) uint16

Uint16 returns the uint16 value at the specified offset.

func (BigEndianBlock) Uint32 

func (b BigEndianBlock) Uint32(offset int) uint32

Uint32 returns the uint32 value at the specified offset.

func (BigEndianBlock) Uint64 

func (b BigEndianBlock) Uint64(offset int) uint64

Uint64 returns the uint64 value at the specified offset.

func (BigEndianBlock) Uint8 

func (b BigEndianBlock) Uint8(offset int) uint8

Uint8 returns the uint8 value at the specified offset.

type Block 

type Block interface {

	// Uint8 returns the uint8 value at the specified offset.
	Uint8(offset int) uint8

	// SetUint8 places the uint8 value at the specified offset.
	SetUint8(offset int, value uint8)

	// Int8 returns the int8 value at the specified offset.
	Int8(offset int) int8

	// SetInt8 places the int8 value at the specified offset.
	SetInt8(offset int, value int8)

	// Uint16 returns the uint16 value at the specified offset.
	Uint16(offset int) uint16

	// SetUint16 places the uint16 value at the specified offset.
	SetUint16(offset int, value uint16)

	// Int16 returns the int16 value at the specified offset.
	Int16(offset int) int16

	// SetInt16 places the int16 value at the specified offset.
	SetInt16(offset int, value int16)

	// Uint32 returns the uint32 value at the specified offset.
	Uint32(offset int) uint32

	// SetUint32 places the uint32 value at the specified offset.
	SetUint32(offset int, value uint32)

	// Int32 returns the int32 value at the specified offset.
	Int32(offset int) int32

	// SetInt32 places the int32 value at the specified offset.
	SetInt32(offset int, value int32)

	// Uint64 returns the uint64 value at the specified offset.
	Uint64(offset int) uint64

	// SetUint64 places the uint64 value at the specified offset.
	SetUint64(offset int, value uint64)

	// Int64 returns the int64 value at the specified offset.
	Int64(offset int) int64

	// SetInt64 places the int64 value at the specified offset.
	SetInt64(offset int, value int64)

	// Float32 returns the float32 value at the specified offset.
	Float32(offset int) float32

	// SetFloat32 places the float32 value at the specified offset.
	SetFloat32(offset int, value float32)

	// Float64 returns the float64 value at the specified offset.
	Float64(offset int) float64

	// SetFloat64 places the float64 value at the specified offset.
	SetFloat64(offset int, value float64)
}

Block represents a fixed-size block of bytes that holds values encoded in a particular order.

type LittleEndianBlock 

type LittleEndianBlock []byte

LittleEndianBlock represents a fixed-size block of bytes that holds values encoded in Little Endian order.

func (LittleEndianBlock) Float32 

func (b LittleEndianBlock) Float32(offset int) float32

Float32 returns the float32 value at the specified offset.

func (LittleEndianBlock) Float64 

func (b LittleEndianBlock) Float64(offset int) float64

Float64 returns the float64 value at the specified offset.

func (LittleEndianBlock) Int16 

func (b LittleEndianBlock) Int16(offset int) int16

Int16 returns the int16 value at the specified offset.

func (LittleEndianBlock) Int32 

func (b LittleEndianBlock) Int32(offset int) int32

Int32 returns the int32 value at the specified offset.

func (LittleEndianBlock) Int64 

func (b LittleEndianBlock) Int64(offset int) int64

Int64 returns the int64 value at the specified offset.

func (LittleEndianBlock) Int8 

func (b LittleEndianBlock) Int8(offset int) int8

Int8 returns the int8 value at the specified offset.

func (LittleEndianBlock) SetFloat32 

func (b LittleEndianBlock) SetFloat32(offset int, value float32)

SetFloat32 places the float32 value at the specified offset.

func (LittleEndianBlock) SetFloat64 

func (b LittleEndianBlock) SetFloat64(offset int, value float64)

SetFloat64 places the float64 value at the specified offset.

func (LittleEndianBlock) SetInt16 

func (b LittleEndianBlock) SetInt16(offset int, value int16)

SetInt16 places the int16 value at the specified offset.

func (LittleEndianBlock) SetInt32 

func (b LittleEndianBlock) SetInt32(offset int, value int32)

SetInt32 places the int32 value at the specified offset.

func (LittleEndianBlock) SetInt64 

func (b LittleEndianBlock) SetInt64(offset int, value int64)

SetInt64 places the int64 value at the specified offset.

func (LittleEndianBlock) SetInt8 

func (b LittleEndianBlock) SetInt8(offset int, value int8)

SetInt8 places the int8 value at the specified offset.

func (LittleEndianBlock) SetUint16 

func (b LittleEndianBlock) SetUint16(offset int, value uint16)

SetUint16 places the uint16 value at the specified offset.

func (LittleEndianBlock) SetUint32 

func (b LittleEndianBlock) SetUint32(offset int, value uint32)

SetUint32 places the uint32 value at the specified offset.

func (LittleEndianBlock) SetUint64 

func (b LittleEndianBlock) SetUint64(offset int, value uint64)

SetUint64 places the uint64 value at the specified offset.

func (LittleEndianBlock) SetUint8 

func (b LittleEndianBlock) SetUint8(offset int, value uint8)

SetUint8 places the uint8 value at the specified offset.

func (LittleEndianBlock) Uint16 

func (b LittleEndianBlock) Uint16(offset int) uint16

Uint16 returns the uint16 value at the specified offset.

func (LittleEndianBlock) Uint32 

func (b LittleEndianBlock) Uint32(offset int) uint32

Uint32 returns the uint32 value at the specified offset.

func (LittleEndianBlock) Uint64 

func (b LittleEndianBlock) Uint64(offset int) uint64

Uint64 returns the uint64 value at the specified offset.

func (LittleEndianBlock) Uint8 

func (b LittleEndianBlock) Uint8(offset int) uint8

Uint8 returns the uint8 value at the specified offset.

type PackedDecodable added in v0.3.0 

type PackedDecodable interface {

	// DecodePacked decodes the receiver from the specified reader.
	DecodePacked(reader TypedReader) error
}

PackedDecodable is an interface that can be implemented by types that want to provide a custom packed decoding.

type PackedDecoder added in v0.2.0 

type PackedDecoder struct {
	// contains filtered or unexported fields
}

PackedDecoder decodes arbitrary Go objects from binary form by going through each field in sequence and deserializing it without any padding.

func NewBigEndianPackedDecoder added in v0.2.0 

func NewBigEndianPackedDecoder(in io.Reader) *PackedDecoder

NewBigEndianPackedDecoder creates a new PackedDecoder that is configured to read its input in Big Endian order.

func NewLittleEndianPackedDecoder added in v0.2.0 

func NewLittleEndianPackedDecoder(in io.Reader) *PackedDecoder

NewLittleEndianPackedDecoder creates a new PackedDecoder that is configured to read its input in Little Endian order.

func (*PackedDecoder) Decode added in v0.2.0 

func (d *PackedDecoder) Decode(target interface{}) error

Decode decodes the specified target value from the Reader.

type PackedEncodable added in v0.3.0 

type PackedEncodable interface {

	// EncodePacked encodes the receiver into the specified writer.
	EncodePacked(writer TypedWriter) error
}

PackedEncodable is an interface that can be implemented by types that want to provide a custom packed encoding.

type PackedEncoder added in v0.2.0 

type PackedEncoder struct {
	// contains filtered or unexported fields
}

PackedEncoder encodes arbitrary Go objects in binary form by going through each field in sequence and serializing it without any padding.

func NewBigEndianPackedEncoder added in v0.2.0 

func NewBigEndianPackedEncoder(out io.Writer) *PackedEncoder

NewBigEndianPackedEncoder creates a new PackedEncoder that is configured to write its output in Big Endian order.

func NewLittleEndianPackedEncoder added in v0.2.0 

func NewLittleEndianPackedEncoder(out io.Writer) *PackedEncoder

NewLittleEndianPackedEncoder creates a new PackedEncoder that is configured to write its output in Little Endian order.

func (*PackedEncoder) Encode added in v0.2.0 

func (e *PackedEncoder) Encode(source any) error

Encode encodes the specified source value into the Writer.

type TypedReader 

type TypedReader interface {

	// ReadUint8 reads a single uint8 from the source.
	ReadUint8() (uint8, error)

	// ReadInt8 reads a single int8 from the source.
	ReadInt8() (int8, error)

	// ReadUint16 reads a single uint16 from the source.
	ReadUint16() (uint16, error)

	// ReadInt16 reads a single int16 from the source.
	ReadInt16() (int16, error)

	// ReadUint32 reads a single uint32 from the source.
	ReadUint32() (uint32, error)

	// ReadInt32 reads a single int32 from the source.
	ReadInt32() (int32, error)

	// ReadUint64 reads a single uint64 from the source.
	ReadUint64() (uint64, error)

	// ReadInt64 reads a single int64 from the source.
	ReadInt64() (int64, error)

	// ReadFloat32 reads a single float32 from the source.
	ReadFloat32() (float32, error)

	// ReadFloat64 reads a single float64 from the source.
	ReadFloat64() (float64, error)

	// ReadBytes reads exactly len(target) bytes from the source and places
	// them inside target.
	ReadBytes(target []byte) error
}

TypedReader represents a reader that can parse specific Go types from a byte sequence.

The endianness depends on the actual implementation.

func NewBigEndianReader 

func NewBigEndianReader(in io.Reader) TypedReader

NewBigEndianReader returns an implementation of TypedReader that reads from the specified in Reader in Big Endian order.

func NewLittleEndianReader 

func NewLittleEndianReader(in io.Reader) TypedReader

NewLittleEndianReader returns an implementation of TypedReader that reads from the specified in Reader in Little Endian order.

type TypedWriter 

type TypedWriter interface {

	// WriteUint8 writes a single uint8 to the target.
	WriteUint8(uint8) error

	// WriteInt8 writes a single int8 to the target.
	WriteInt8(int8) error

	// WriteUint16 writes a single uint16 to the target.
	WriteUint16(uint16) error

	// WriteInt16 writes a single int16 to the target.
	WriteInt16(int16) error

	// WriteUint32 writes a single uint32 to the target.
	WriteUint32(uint32) error

	// WriteInt32 writes a single int32 to the target.
	WriteInt32(int32) error

	// WriteUint64 writes a single uint64 to the target.
	WriteUint64(uint64) error

	// WriteInt64 writes a single int64 to the target.
	WriteInt64(int64) error

	// WriteFloat32 writes a single float32 to the target.
	WriteFloat32(float32) error

	// WriteFloat64 writes a single float64 to the target.
	WriteFloat64(float64) error

	// WriteBytes writes len(bytes) from source to the target.
	WriteBytes(source []byte) error
}

TypedWriter represents a writer that can serialize specific Go types to a byte sequence.

The endianness depends on the actual implementation.

func NewBigEndianWriter 

func NewBigEndianWriter(out io.Writer) TypedWriter

NewBigEndianWriter returns an implementation of TypedWriter that writes to the specified out Writer in Big Endian order.

func NewLittleEndianWriter 

func NewLittleEndianWriter(out io.Writer) TypedWriter

NewLittleEndianWriter returns an implementation of TypedWriter that writes to the specified out Writer in Little Endian order.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.