README
¶
Very fast, very unsafe serialization for Go
This package provides a fast way to load large amounts of data into Go structs. Memdump can load datasets containing millions of small structs at over 1 GB/s (compared to ~30 MB/s for gob or json).
The price you pay is:
- you cannot load structs that contain maps or interfaces
- your data is not portable across machine architectures (64 bit vs 32 bit, big-endian vs small-endian)
Benchmarks
The benchmarks were measured by encoding and decoding a tree containing 2,097,151 small structs. Code is under the bench dir.
Decode
gob 28.17 MB/s (39.8 MB in 1.41s)
json 30.17 MB/s (113.8 MB in 3.77s)
memdump 1031.54 MB/s (113.2 MB in 0.11s)
Encode
gob 37.07 MB/s (39.8 MB in 1.07s)
json 77.20 MB/s (113.8 MB in 1.47s)
memdump 61.25 MB/s (113.2 MB in 1.85s)
To reproduce these results:
$ go run ./bench/summarize.go
Quick start
go get github.com/alexflint/go-memdump
Write data to a file:
type data struct {
Foo string
Bar int
}
w, err := os.Create("/tmp/data.memdump")
if err != nil {
...
}
// note that you must pass a pointer when encoding
mydata := data{Foo: "abc", Bar: 123}
memdump.Encode(w, &mydata)
Load data from a file:
r, err := os.Open("/tmp/data.memdump")
if err != nil {
...
}
// note that you muss pass a pointer to a pointer when decoding
var mydata *data
memdump.Decode(r, &mydata)
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrBufferTooSmall = errors.New(
"cannot read into buffer of size less than 16 bytes (due to delim)")
ErrBufferTooSmall is returned by delimetedReader if the input buffer is smaller than the length of the delimeter
var ( // ErrIncompatibleLayout is returned by decoders when the object on the wire has // an in-memory layout that is not compatible with the requested Go type. ErrIncompatibleLayout = errors.New("attempted to load data with incompatible layout") )
var ErrUnexpectedEOF = errors.New(
"got EOF before finding the delimeter")
ErrUnexpectedEOF is returned by delimetedReader if EOF is reached before finding a delimeter
Functions ¶
Types ¶
type Decoder ¶
type Decoder struct {
// contains filtered or unexported fields
}
Decoder reads memdumps from the provided reader
func NewDecoder ¶
NewDecoder creates a Decoder that reads memdumps
func (*Decoder) Decode ¶
Decode reads an object of the specified type from the input. The object passed to Decode must be a pointer to the type was originally passed to Encode().
type DelimitedReader ¶
type DelimitedReader struct {
// contains filtered or unexported fields
}
DelimitedReader reads delimited segments
func NewDelimitedReader ¶
func NewDelimitedReader(r io.Reader) *DelimitedReader
NewDelimitedReader creates a reader for delimited segments
func (*DelimitedReader) Next ¶
func (r *DelimitedReader) Next() ([]byte, error)
Next returns the next segment, or (nil, io.EOF) if there are no more segments. The data is only valid until the next call to Next().
type Encoder ¶
type Encoder struct {
// contains filtered or unexported fields
}
Encoder writes memdumps to the provided writer
func NewEncoder ¶
NewEncoder creates an Encoder that writes memdumps to the provided writer. Each object passed to Encode must be of the same type.
type HeterogeneousDecoder ¶
type HeterogeneousDecoder struct {
// contains filtered or unexported fields
}
HeterogeneousDecoder reads memdumps from the provided reader
func NewHeterogeneousDecoder ¶
func NewHeterogeneousDecoder(r io.Reader) *HeterogeneousDecoder
NewHeterogeneousDecoder creates a HeterogeneousDecoder that reads memdumps
func (*HeterogeneousDecoder) Decode ¶
func (d *HeterogeneousDecoder) Decode(dest interface{}) error
Decode reads an object of the specified type from the input. The object passed to Decode must be a pointer to the type was originally passed to Encode().
func (*HeterogeneousDecoder) DecodePtr ¶
func (d *HeterogeneousDecoder) DecodePtr(typ reflect.Type) (interface{}, error)
DecodePtr reads an object of the specified type from the input and returns a pointer to it. The provided type must be the result of calling reflect.TypeOf(x) where x is the object originally passed to Encode(). The return valoue will be of type *x
type HeterogeneousEncoder ¶
type HeterogeneousEncoder struct {
// contains filtered or unexported fields
}
HeterogeneousEncoder writes memdumps to the provided writer
func NewHeterogeneousEncoder ¶
func NewHeterogeneousEncoder(w io.Writer) *HeterogeneousEncoder
NewHeterogeneousEncoder creates an HeterogeneousEncoder that writes memdumps to the provided writer
func (*HeterogeneousEncoder) Encode ¶
func (e *HeterogeneousEncoder) Encode(obj interface{}) error
Encode writes a memdump of the provided object to output. You must pass a pointer to the object you wish to encode. To encode a pointer, pass a double-pointer.
Source Files
Directories