README
¶
Very fast, very unsafe serialization for Go
This package provides a fast way to load large amounts of data into Go structs. As shown in the benchmarks, memdump can load datasets containing millions of small structs at over 1 GB/s.
However, the price you pay for decoding performance is:
- you cannot load structs that contain maps or interfaces
- data migration is not supported
- your data is not portable across machine architectures (64 bit vs 32 bit, big-endian vs small-endian)
- encoding performance is merely on par with typical json or gob performance
Memdump was designed for use in a caching layer, where the data was encoded and decoded by the same machine, and where decoding performance mattered more than encoding performance.
Benchmarks
The benchmarks were measured by encoding and decoding a tree containing 2,097,151 nodes. See below for further details.
Decode
gob 28.17 MB/s (39.8 MB in 1.41s)
json 30.17 MB/s (113.8 MB in 3.77s)
memdump 1031.54 MB/s (113.2 MB in 0.11s)
Encode
gob 37.07 MB/s (39.8 MB in 1.07s)
json 77.20 MB/s (113.8 MB in 1.47s)
memdump 61.25 MB/s (113.2 MB in 1.85s)
The tree nodes were as follows:
type treeNode struct {
Label string
Weight int
Path []pathComponent
Children []*treeNode
}
To reproduce these results:
$ go get -u github.com/alexflint/go-memdump
$ go build github.com/alexflint/go-memdump/bench -o /tmp/bench
$ /tmp/bench
Quick start
go get github.com/alexflint/go-memdump
Write data to a file:
type data struct {
Foo string
Bar int
}
mydata := data{Foo: "abc", Bar: 123}
w, _ := os.Create("/tmp/data.memdump", 0777)
// note that you must pass a pointer when encoding
memdump.Encode(w, &args)
Load data from a file:
var mydata *data
r, _ := os.Open("/tmp/data.memdump")
// note that you muss pass a pointer to a pointer when decoding
memdump.Decode(r, &mydata)
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrBufferTooSmall = errors.New(
"cannot read into buffer of size less than 16 bytes (due to delim)")
ErrBufferTooSmall is returned by delimetedReader if the input buffer is smaller than the length of the delimeter
var ( // ErrIncompatibleLayout is returned by decoders when the object on the wire has // an in-memory layout that is not compatible with the requested Go type. ErrIncompatibleLayout = errors.New("attempted to load data with incompatible layout") )
var ErrUnexpectedEOF = errors.New(
"got EOF before finding the delimeter")
ErrUnexpectedEOF is returned by delimetedReader if EOF is reached before finding a delimeter
Functions ¶
Types ¶
type Decoder ¶
type Decoder struct {
// contains filtered or unexported fields
}
Decoder reads memdumps from the provided reader
func NewDecoder ¶
NewDecoder creates a Decoder that reads memdumps
func (*Decoder) Decode ¶
Decode reads an object of the specified type from the input. The object passed to Decode must be a pointer to the type was originally passed to Encode().
type DelimitedReader ¶
type DelimitedReader struct {
// contains filtered or unexported fields
}
DelimitedReader reads delimited segments
func NewDelimitedReader ¶
func NewDelimitedReader(r io.Reader) *DelimitedReader
NewDelimitedReader creates a reader for delimited segments
func (*DelimitedReader) Next ¶
func (r *DelimitedReader) Next() ([]byte, error)
Next returns the next segment, or (nil, io.EOF) if there are no more segments. The data is only valid until the next call to Next().
type Encoder ¶
type Encoder struct {
// contains filtered or unexported fields
}
Encoder writes memdumps to the provided writer
func NewEncoder ¶
NewEncoder creates an Encoder that writes memdumps to the provided writer
type HeterogeneousDecoder ¶
type HeterogeneousDecoder struct {
// contains filtered or unexported fields
}
HeterogeneousDecoder reads memdumps from the provided reader
func NewHeterogeneousDecoder ¶
func NewHeterogeneousDecoder(r io.Reader) *HeterogeneousDecoder
NewHeterogeneousDecoder creates a HeterogeneousDecoder that reads memdumps
func (*HeterogeneousDecoder) Decode ¶
func (d *HeterogeneousDecoder) Decode(dest interface{}) error
Decode reads an object of the specified type from the input. The object passed to Decode must be a pointer to the type was originally passed to Encode().
func (*HeterogeneousDecoder) DecodePtr ¶
func (d *HeterogeneousDecoder) DecodePtr(typ reflect.Type) (interface{}, error)
DecodePtr reads an object of the specified type from the input and returns a pointer to it. The provided type must be the result of calling reflect.TypeOf(x) where x is the object originally passed to Encode(). The return valoue will be of type *x
type HeterogeneousEncoder ¶
type HeterogeneousEncoder struct {
// contains filtered or unexported fields
}
HeterogeneousEncoder writes memdumps to the provided writer
func NewHeterogeneousEncoder ¶
func NewHeterogeneousEncoder(w io.Writer) *HeterogeneousEncoder
NewHeterogeneousEncoder creates an HeterogeneousEncoder that writes memdumps to the provided writer
func (*HeterogeneousEncoder) Encode ¶
func (e *HeterogeneousEncoder) Encode(obj interface{}) error
Encode writes a memdump of the provided object to output. You must pass a pointer to the object you wish to encode. To encode a pointer, pass a double-pointer.
Source Files
Directories