bytebuilder

README

ByteBuilder

A bytes.Buffer with the capability of writing any object's exact memory layout (with every padding) into it.

WARNING

This is not a proper serialization protocol implementation, and has never intended to be one. Any data acquired with this method is usually only meaningful under the same CPU architecture, the same OS, AND the same compiler.

Cases when you should NOT use this library:

• If you want to dump structs to a persistent storage only to restore them later in the same program, use encoding/gob.
• To encode/decode between basic types and a binary buffer, use encoding/binary.
• For reading/writing machine-readable config files, use mapstructure, encoding/json, or encoding/xml.
• For exchanging data between network services, use Protocol Buffers or Cap'n Proto.

If an API supplied by this library does some sort of type conversion but does not alter its data, then the returned object is READ ONLY. DO NOT attempt to write to them.

Usage

bytebuilder.WriteObject

Takes a io.Writer and a pointer to an object. Writes the object into the Writer.

bytebuilder.WriteObject is well-defined (relies only on public methods) although unsafe (bypasses type safety). Generics support is required, so this function is only available on Golang 1.18+.

package main

import (
	"bytes"
	"fmt"
	"github.com/jamesits/go-bytebuilder"
	"unsafe"
)

type SomeStruct struct {
	Value1 uint16
	// there is an implicit padding
	Value2 uint64
}

func main() {
	s := SomeStruct{
		Value1: 100,
		Value2: 500,
	}
	fmt.Printf("size of s is %d\n", unsafe.Sizeof(s))
	
	// using the Writer wrapper
	var b bytes.Buffer
	_, _ = bytebuilder.WriteObject(&b, &s)
	fmt.Printf("size of b is %d\n", b.Len())
	
	// you can write more data into the same buffer
	b.Write([]byte{1, 1, 4, 5, 1, 4})
}

bytebuilder.ByteBuilder

ByteBuilder provides a StringBuilder-like API to concentrate multiple objects into one big buffer.

bytebuilder.ByteBuilder{}.WriteObject writes the object into the buffer. It relies on internal knowledge of the Golang runtime implementation and unsafe. It does not require generics, and works on all recent Golang versions.

Other methods of a ByteBuilder works exactly the same as a bytes.Buffer, so you can use Write() or other methods for certain use cases.

package main

import (
	"fmt"
	"unsafe"

	"github.com/jamesits/go-bytebuilder"
)

type SomeStruct struct {
	Value1 uint16
	// there is an implicit padding
	Value2 uint64
}

func main() {
	s := SomeStruct{
		Value1: 100,
		Value2: 500,
	}
	fmt.Printf("size of s is %d\n", unsafe.Sizeof(s))
	
	// using the ByteBuilder
	var bb bytebuilder.ByteBuilder
	_, _ = bb.WriteObject(s)
	fmt.Printf("size of bb is %d\n", bb.Len())
	
	// you can write more objects into it
	_, _ = bb.WriteObject(s)
	
	// or use it as a generic buffer
	_, _ = bb.Write([]byte{1, 1, 4, 5, 1, 4})
	
	fmt.Printf("Buffer: %v\n", bb.Bytes())
}

Marshal / Unmarshal

package main

import (
	"fmt"
	"github.com/jamesits/go-bytebuilder"
)

type SomeStruct struct {
	Value1 uint16
	// there is an implicit padding
	Value2 uint64
}

func main() {
	var err error

	s := SomeStruct{
		Value1: 100,
		Value2: 500,
	}

	b, err := bytebuilder.Marshal(&s)
	if err != nil {
		panic(err)
	}

	var s2 SomeStruct
	err = bytebuilder.Unmarshal(b, &s2)
	if err != nil {
		panic(err)
	}

	fmt.Printf("s2.Value1 = %d, s2.Value2 = %d\n", s2.Value1, s2.Value2)
}

Real-world Examples

Say we have the following type definitions in C:

typedef struct {
    uint64_t id;
    char name[32];
} student;

typedef struct {
    char name[32];
    uint8_t student_count;
    student students[1];
} class;

And we have a C function that takes an argument of type *class.

It is impossible to express structs with flexible array members natively in Golang. But if your C compiler's padding rules happens to be the same as Golang's, this library offers you a shortcut besides your regular byte operations.

package main

import (
	"bytes"
	"github.com/jamesits/go-bytebuilder"
)

type Student struct {
	Id uint64
	Name [32]byte
}

type Class struct {
	Name [32]byte
	StudentCount uint8
}

func main() {
	class := Class{
		Name: [32]byte{'c', 'l', 'a', 's', 's'},
		StudentCount: 2,
	}
	
	alice := Student{
		Id: 1,
		Name: [32]byte{'A', 'l', 'i', 'c', 'e'},
	}
	
	bob := Student{
		Id: 2,
		Name: [32]byte{'B', 'o', 'b'},
	}
	
	var buf bytes.Buffer
	_, _ = bytebuilder.WriteObject(&buf, &class)
	_, _ = bytebuilder.WriteObject(&buf, &alice)
	_, _ = bytebuilder.WriteObject(&buf, &bob)
	
	// now buf.Bytes() contains exactly what you need to give to the native function
	fmt.Printf("%v\n", buf.Bytes())
}

Documentation

Index

• Variables
• func CarCdr[T any](s []byte) (car *T, cdr []byte)
• func Copy[T any](b []byte, object *T) (n int, err error)
• func Marshal[T any](v *T) ([]byte, error)
• func NewArbitraryByteArray(length uintptr, ptr uintptr) []byte
• func ReadPartial[T any](r io.Reader, object *T) (n int, err error)
• func Skip(reader io.Reader, count int64) (int64, error)
• func SliceCast[Ts, Td any](s []Ts) (d []Td)
• func Unmarshal[T any](data []byte, v *T) (err error)
• func WriteObject[T any](w io.Writer, object *T) (n int, err error)
• type ByteBuilder
  • func (b *ByteBuilder) WriteObject(obj any) (n int, err error)

Variables

var SizeMismatch = errors.New("size mismatch")

Functions

func CarCdr

func CarCdr[T any](s []byte) (car *T, cdr []byte)

CarCdr returns a (readonly) object converted from your input slice, and the remaining (readonly) slice. It assumes the slice is created from a readonly pinned memory area.

func Copy

func Copy[T any](b []byte, object *T) (n int, err error)

Copy an object to a byte buffer.

func Marshal

func Marshal[T any](v *T) ([]byte, error)

Marshal returns a (shallow) copy of the exact internal memory representation of v.

func NewArbitraryByteArray

func NewArbitraryByteArray(length uintptr, ptr uintptr) []byte

NewArbitraryByteArray creates a (readonly) array from any memory location and length. It assumes the memory behind the ptr is allocated elsewhere, readonly and pinned.

func ReadPartial

func ReadPartial[T any](r io.Reader, object *T) (n int, err error)

ReadPartial reads from an io.Reader directly into an object.

func Skip

func Skip(reader io.Reader, count int64) (int64, error)

Skip certain length from the reader.

func SliceCast

func SliceCast[Ts, Td any](s []Ts) (d []Td)

SliceCast converts between different types of slices. It assumes the source slice is created from a readonly pinned memory area. Note: edge cases are not tested for, use with care.

func Unmarshal

func Unmarshal[T any](data []byte, v *T) (err error)

Unmarshal copies the data into the memory pointed to by v.

func WriteObject

func WriteObject[T any](w io.Writer, object *T) (n int, err error)

WriteObject write any object's internal memory representation into an io.Writer. Arguments: - w: io.Writer - object: a pointer to the source object

Types

type ByteBuilder

type ByteBuilder struct {
	bytes.Buffer
}

ByteBuilder is a `bytes.Buffer` with the capability of writing any object into it.

func (*ByteBuilder) WriteObject

func (b *ByteBuilder) WriteObject(obj any) (n int, err error)

WriteObject writes any object's internal memory representation into the buffer. Arguments: - obj: the source object itself