serialize

Documentation

Overview

Package serialize provides methods for reading and writing concatenable, bytewise-sortable forms of the datatypes defined in the datastore package.

Index

• Constants
• Variables
• func Increment(bstr []byte) ([]byte, bool)
• func Invert(bs []byte) []byte
• func Join(itms ...[]byte) []byte
• func ReadGeoPoint(buf ReadBuffer) (gp ds.GeoPoint, err error)
• func ReadIndexColumn(buf ReadBuffer) (c ds.IndexColumn, err error)
• func ReadIndexDefinition(buf ReadBuffer) (i ds.IndexDefinition, err error)
• func ReadKey(buf ReadBuffer, context KeyContext, inKC ds.KeyContext) (ret *ds.Key, err error)
• func ReadKeyTok(buf ReadBuffer) (ret ds.KeyTok, err error)
• func ReadProperty(buf ReadBuffer, context KeyContext, kc ds.KeyContext) (p ds.Property, err error)
• func ReadPropertyMap(buf ReadBuffer, context KeyContext, kc ds.KeyContext) (pm ds.PropertyMap, err error)
• func ReadTime(buf ReadBuffer) (time.Time, error)
• func ToBytes(i interface{}) []byte
• func ToBytesErr(i interface{}) ([]byte, error)
• func ToBytesWithContext(i interface{}) []byte
• func ToBytesWithContextErr(i interface{}) ([]byte, error)
• func WriteGeoPoint(buf WriteBuffer, gp ds.GeoPoint) (err error)
• func WriteIndexColumn(buf WriteBuffer, c ds.IndexColumn) (err error)
• func WriteIndexDefinition(buf WriteBuffer, i ds.IndexDefinition) (err error)
• func WriteIndexProperty(buf WriteBuffer, context KeyContext, p ds.Property) error
• func WriteKey(buf WriteBuffer, context KeyContext, k *ds.Key) (err error)
• func WriteKeyTok(buf WriteBuffer, tok ds.KeyTok) (err error)
• func WriteProperty(buf WriteBuffer, context KeyContext, p ds.Property) error
• func WritePropertyMap(buf WriteBuffer, context KeyContext, pm ds.PropertyMap) (err error)
• func WriteTime(buf WriteBuffer, t time.Time) error
• type InvertibleBuffer
  • func Invertible(b WriteBuffer) InvertibleBuffer
• type KeyContext
• type ReadBuffer
• type SerializedPmap
  • func PropertyMapPartially(k *ds.Key, pm ds.PropertyMap) (ret SerializedPmap)
• type SerializedPslice
  • func PropertySlice(vals ds.PropertySlice) SerializedPslice
  • func (s SerializedPslice) Len() int
  • func (s SerializedPslice) Less(i, j int) bool
  • func (s SerializedPslice) Swap(i, j int)
• type WriteBuffer

Constants

const MaxIndexColumns = 64

MaxIndexColumns is the maximum number of sort columns (e.g. sort orders) that ReadIndexDefinition is willing to deserialize. 64 was chosen as a likely-astronomical number.

const ReadKeyNumToksReasonableLimit = 50

ReadKeyNumToksReasonableLimit is the maximum number of Key tokens that ReadKey is willing to read for a single key.

const ReadPropertyMapReasonableLimit uint64 = 30000

ReadPropertyMapReasonableLimit sets a limit on the number of rows and number of properties per row which can be read by ReadPropertyMap. The total number of Property objects readable by this method is this number squared (e.g. Limit rows * Limit properties)

Variables

var WritePropertyMapDeterministic = false

WritePropertyMapDeterministic allows tests to make WritePropertyMap deterministic.

Functions

func Increment

func Increment(bstr []byte) ([]byte, bool)

Increment attempts to increment a copy of bstr as if adding 1 to an integer.

If it overflows, the returned []byte will be all 0's, and the overflow bool will be true.

func Invert

func Invert(bs []byte) []byte

Invert simply inverts all the bytes in bs.

func Join

func Join(itms ...[]byte) []byte

Join is a convenience invocation of bytes.Join(itms, nil)

func ReadGeoPoint

func ReadGeoPoint(buf ReadBuffer) (gp ds.GeoPoint, err error)

ReadGeoPoint reads a GeoPoint from the buffer.

func ReadIndexColumn

func ReadIndexColumn(buf ReadBuffer) (c ds.IndexColumn, err error)

ReadIndexColumn reads an IndexColumn from the buffer.

func ReadIndexDefinition

func ReadIndexDefinition(buf ReadBuffer) (i ds.IndexDefinition, err error)

ReadIndexDefinition reads an IndexDefinition from the buffer.

func ReadKey

func ReadKey(buf ReadBuffer, context KeyContext, inKC ds.KeyContext) (ret *ds.Key, err error)

ReadKey deserializes a key from the buffer. The value of context must match the value of context that was passed to WriteKey when the key was encoded. If context == WithoutContext, then the appid and namespace parameters are used in the decoded Key. Otherwise they're ignored.

func ReadKeyTok

func ReadKeyTok(buf ReadBuffer) (ret ds.KeyTok, err error)

ReadKeyTok reads a KeyTok from the buffer. You usually want ReadKey instead of this.

func ReadProperty

func ReadProperty(buf ReadBuffer, context KeyContext, kc ds.KeyContext) (p ds.Property, err error)

ReadProperty reads a Property from the buffer. `context` and `kc` behave the same way they do for ReadKey, but only have an effect if the decoded property has a Key value.

func ReadPropertyMap

func ReadPropertyMap(buf ReadBuffer, context KeyContext, kc ds.KeyContext) (pm ds.PropertyMap, err error)

ReadPropertyMap reads a PropertyMap from the buffer. `context` and friends behave the same way that they do for ReadKey.

func ReadTime

func ReadTime(buf ReadBuffer) (time.Time, error)

ReadTime reads a time.Time from the buffer.

func ToBytes

func ToBytes(i interface{}) []byte

ToBytes serializes i to a byte slice, if it's one of the type supported by this library. If an error is encountered (e.g. `i` is not a supported type), this method panics.

Key types will be serialized using the 'WithoutContext' option (e.g. their encoded forms will not contain AppID or Namespace).

func ToBytesErr

func ToBytesErr(i interface{}) ([]byte, error)

ToBytesErr serializes i to a byte slice, if it's one of the type supported by this library, otherwise it returns an error.

Key types will be serialized using the 'WithoutContext' option (e.g. their encoded forms will not contain AppID or Namespace).

func ToBytesWithContext

func ToBytesWithContext(i interface{}) []byte

ToBytesWithContext serializes i to a byte slice, if it's one of the type supported by this library. If an error is encountered (e.g. `i` is not a supported type), this method panics.

Key types will be serialized using the 'WithContext' option (e.g. their encoded forms will not contain AppID or Namespace).

func ToBytesWithContextErr

func ToBytesWithContextErr(i interface{}) ([]byte, error)

ToBytesWithContextErr serializes i to a byte slice, if it's one of the type supported by this library, otherwise it returns an error.

Key types will be serialized using the 'WithContext' option (e.g. their encoded forms will contain AppID and Namespace).

func WriteGeoPoint

func WriteGeoPoint(buf WriteBuffer, gp ds.GeoPoint) (err error)

WriteGeoPoint writes a GeoPoint to the buffer.

func WriteIndexColumn

func WriteIndexColumn(buf WriteBuffer, c ds.IndexColumn) (err error)

WriteIndexColumn writes an IndexColumn to the buffer.

func WriteIndexDefinition

func WriteIndexDefinition(buf WriteBuffer, i ds.IndexDefinition) (err error)

WriteIndexDefinition writes an IndexDefinition to the buffer

func WriteIndexProperty

func WriteIndexProperty(buf WriteBuffer, context KeyContext, p ds.Property) error

WriteIndexProperty writes a Property to the buffer as its native index type. `context` behaves the same way that it does for WriteKey, but only has an effect if `p` contains a Key as its IndexValue.

func WriteKey

func WriteKey(buf WriteBuffer, context KeyContext, k *ds.Key) (err error)

WriteKey encodes a key to the buffer. If context is WithContext, then this encoded value will include the appid and namespace of the key.

func WriteKeyTok

func WriteKeyTok(buf WriteBuffer, tok ds.KeyTok) (err error)

WriteKeyTok writes a KeyTok to the buffer. You usually want WriteKey instead of this.

func WriteProperty

func WriteProperty(buf WriteBuffer, context KeyContext, p ds.Property) error

WriteProperty writes a Property to the buffer. `context` behaves the same way that it does for WriteKey, but only has an effect if `p` contains a Key as its IndexValue.

func WritePropertyMap

func WritePropertyMap(buf WriteBuffer, context KeyContext, pm ds.PropertyMap) (err error)

WritePropertyMap writes an entire PropertyMap to the buffer. `context` behaves the same way that it does for WriteKey.

If WritePropertyMapDeterministic is true, then the rows will be sorted by property name before they're serialized to buf (mostly useful for testing, but also potentially useful if you need to make a hash of the property data).

Write skips metadata keys.

func WriteTime

func WriteTime(buf WriteBuffer, t time.Time) error

WriteTime writes a time.Time to the buffer.

The supplied time is rounded via datastore.RoundTime and written as a microseconds-since-epoch integer to comform to datastore storage standards.

Types

type InvertibleBuffer

type InvertibleBuffer interface {
	WriteBuffer
	SetInvert(inverted bool)
}

InvertibleBuffer is just like Buffer, except that it also has a stateful Invert() method, which will cause all reads and writes to/from it to be inverted (e.g. every byte XOR 0xFF).

Implementing queries requires manipulating the index entries (e.g. synthesizing them, parsing them, etc.). In particular, when you have a reverse-sorted field (e.g. high to low instead of low to high), it's achieved by having all the bits inverted.

All the serialization formats include delimiter information, which the parsers only know to parse non-inverted. If we don't have this buffer, we'd basically have to invert every byte in the []byte array when we're trying to decode a reverse-ordered field (including the bytes of all fields after the one we intend to parse) so that the parser can consume as many bytes as it needs (and it only knows the number of bytes it needs as it decodes them). This InvertibleBuffer lets that happen on the fly without having to flip the whole []byte.

If you know you need it, you'll know it's the right thing. If you're not sure then you definitely don't need it!

func Invertible

func Invertible(b WriteBuffer) InvertibleBuffer

Invertible returns an InvertibleBuffer based on the Buffer.

type KeyContext

type KeyContext bool

KeyContext controls whether the various Write and Read serializtion routines should encode the context of Keys (read: the appid and namespace). Frequently the appid and namespace of keys are known in advance and so there's no reason to redundantly encode them.

const (
	WithContext    KeyContext = true
	WithoutContext            = false
)

With- and WithoutContext indicate if the serialization method should include context for Keys. See KeyContext for more information.

type ReadBuffer

type ReadBuffer interface {
	Len() int

	Read([]byte) (int, error)
	ReadByte() (byte, error)
}

ReadBuffer is the interface which corresponds to the subset of *bytes.Reader that this package requires.

type SerializedPmap

type SerializedPmap map[string]SerializedPslice

SerializedPmap maps from

prop name -> [<serialized DSProperty>, ...]

includes special values '__key__' and '__ancestor__' which contains all of the ancestor entries for this key.

func PropertyMapPartially

func PropertyMapPartially(k *ds.Key, pm ds.PropertyMap) (ret SerializedPmap)

PropertyMapPartially turns a regular PropertyMap into a SerializedPmap. Essentially all the []Property's become SerializedPslice, using cmpbin and datastore/serialize's encodings.

type SerializedPslice

type SerializedPslice [][]byte

SerializedPslice is all of the serialized DSProperty values in ASC order.

func PropertySlice

func PropertySlice(vals ds.PropertySlice) SerializedPslice

PropertySlice serializes a single row of a DSProperty map.

It does not differentiate between single- and multi- properties.

func (SerializedPslice) Len

func (s SerializedPslice) Len() int

func (SerializedPslice) Less

func (s SerializedPslice) Less(i, j int) bool

func (SerializedPslice) Swap

func (s SerializedPslice) Swap(i, j int)

type WriteBuffer

type WriteBuffer interface {
	ReadBuffer

	String() string
	Bytes() []byte

	Grow(int)

	Write([]byte) (int, error)
	WriteByte(c byte) error
	WriteString(s string) (int, error)
}

WriteBuffer is the interface which corresponds to the subset of *bytes.Buffer that this package requires.