state

package
v0.0.0-...-ba09d25 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 29, 2021 License: Apache-2.0, MIT Imports: 11 Imported by: 0

README

State Encoding and Decoding

The state package implements the encoding and decoding of data structures for go_stateify. This package is designed for use cases other than the standard encoding packages, e.g. gob and json. Principally:

  • This package operates on complex object graphs and accurately serializes and restores all relationships. That is, you can have things like: intrusive pointers, cycles, and pointer chains of arbitrary depths. These are not handled appropriately by existing encoders. This is not an implementation flaw: the formats themselves are not capable of representing these graphs, as they can only generate directed trees.

  • This package allows installing order-dependent load callbacks and then resolves that graph at load time, with cycle detection. Similarly, there is no analogous feature possible in the standard encoders.

  • This package handles the resolution of interfaces, based on a registered type name. For interface objects type information is saved in the serialized format. This is generally true for gob as well, but it works differently.

Here's an overview of how encoding and decoding works.

Encoding

Encoding produces a statefile, which contains a list of chunks of the form (header, payload). The payload can either be some raw data, or a series of encoded wire objects representing some object graph. All encoded objects are defined in the wire subpackage.

Encoding of an object graph begins with encodeState.Save.

1. Memory Map & Encoding

To discover relationships between potentially interdependent data structures (for example, a struct may contain pointers to members of other data structures), the encoder first walks the object graph and constructs a memory map of the objects in the input graph. As this walk progresses, objects are queued in the pending list and items are placed on the deferred list as they are discovered. No single object will be encoded multiple times, but the discovered relationships between objects may change as more parts of the overall object graph are discovered.

The encoder starts at the root object and recursively visits all reachable objects, recording the address ranges containing the underlying data for each object. This is stored as a segment set (addrSet), mapping address ranges to the of the object occupying the range; see encodeState.values. Note that there is special handling for zero-sized types and map objects during this process.

Additionally, the encoder assigns each object a unique identifier which is used to indicate relationships between objects in the statefile; see objectID in encode.go.

2. Type Serialization

The enoder will subsequently serialize all information about discovered types, including field names. These are used during decoding to reconcile these types with other internally registered types.

3. Object Serialization

With a full address map, and all objects correctly encoded, all object encodings are serialized. The assigned objectIDs aren't explicitly encoded in the statefile. The order of object messages in the stream determine their IDs.

Example

Given the following data structure definitions:

type system struct {
    o *outer
    i *inner
}

type outer struct {
    a  int64
    cn *container
}

type container struct {
    n    uint64
    elem *inner
}

type inner struct {
    c    container
    x, y uint64
}

Initialized like this:

o := outer{
    a: 10,
    cn: nil,
}
i := inner{
    x: 20,
    y: 30,
    c: container{},
}
s := system{
    o: &o,
    i: &i,
}

o.cn = &i.c
o.cn.elem = &i

Encoding will produce an object stream like this:

g0r1 = struct{
     i: g0r3,
     o: g0r2,
}
g0r2 = struct{
     a: 10,
     cn: g0r3.c,
}
g0r3 = struct{
     c: struct{
             elem: g0r3,
             n: 0u,
     },
     x: 20u,
     y: 30u,
}

Note how g0r3.c is correctly encoded as the underlying container object for inner.c, and how the pointer from outer.cn points to it, despite system.i being discovered after the pointer to it in system.o.cn. Also note that decoding isn't strictly reliant on the order of encoded object stream, as long as the relationship between objects are correctly encoded.

Decoding

Decoding reads the statefile and reconstructs the object graph. Decoding begins in decodeState.Load. Decoding is performed in a single pass over the object stream in the statefile, and a subsequent pass over all deserialized objects is done to fire off all loading callbacks in the correctly defined order. Note that introducing cycles is possible here, but these are detected and an error will be returned.

Decoding is relatively straight forward. For most primitive values, the decoder constructs an appropriate object and fills it with the values encoded in the statefile. Pointers need special handling, as they must point to a value allocated elsewhere. When values are constructed, the decoder indexes them by their objectIDs in decodeState.objectsByID. The target of pointers are resolved by searching for the target in this index by their objectID; see decodeState.register. For pointers to values inside another value (fields in a pointer, elements of an array), the decoder uses the accessor path to walk to the appropriate location; see walkChild.

Documentation

Overview

Package state provides functionality related to saving and loading object graphs. For most types, it provides a set of default saving / loading logic that will be invoked automatically if custom logic is not defined.

Kind             Support
----             -------
Bool             default
Int              default
Int8             default
Int16            default
Int32            default
Int64            default
Uint             default
Uint8            default
Uint16           default
Uint32           default
Uint64           default
Float32          default
Float64          default
Complex64        default
Complex128       default
Array            default
Chan             custom
Func             custom
Interface        default
Map              default
Ptr              default
Slice            default
String           default
Struct           custom (*) Unless zero-sized.
UnsafePointer    custom

See README.md for an overview of how encoding and decoding works.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Failf

func Failf(fmtStr string, v ...interface{})

Failf is a wrapper around panic that should be used to generate errors that can be caught during saving and loading.

func IsZeroValue

func IsZeroValue(val interface{}) bool

IsZeroValue checks if the given value is the zero value.

This function is used by the stateify tool.

func ReadHeader

func ReadHeader(r wire.Reader) (length uint64, object bool, err error)

ReadHeader reads an object header.

Each object written to the statefile is prefixed with a header. See WriteHeader for more information; these functions are exported to allow non-state writes to the file to play nice with debugging tools.

func Register

func Register(t Type)

Register registers a type.

This must be called on init and only done once.

func WriteHeader

func WriteHeader(w wire.Writer, length uint64, object bool) error

WriteHeader writes a header.

Each object written to the statefile should be prefixed with a header. In order to generate statefiles that play nicely with debugging tools, raw writes should be prefixed with a header with object set to false and the appropriate length. This will allow tools to skip these regions.

Types

type ErrState

type ErrState struct {
	// contains filtered or unexported fields
}

ErrState is returned when an error is encountered during encode/decode.

func (*ErrState) Error

func (e *ErrState) Error() string

Error returns a sensible description of the state error.

func (*ErrState) Unwrap

func (e *ErrState) Unwrap() error

Unwrap implements standard unwrapping.

type SaverLoader

type SaverLoader interface {
	// StateSave saves the state of the object to the given Map.
	StateSave(Sink)

	// StateLoad loads the state of the object.
	StateLoad(Source)
}

SaverLoader must be implemented by struct types.

type Sink

type Sink struct {
	// contains filtered or unexported fields
}

Sink is used for Type.StateSave.

func (Sink) Context

func (s Sink) Context() context.Context

Context returns the context object provided at save time.

func (Sink) Save

func (s Sink) Save(slot int, objPtr interface{})

Save adds the given object to the map.

You should pass always pointers to the object you are saving. For example:

type X struct {
	A int
	B *int
}
func (x *X) StateTypeInfo(m Sink) state.TypeInfo {
	return state.TypeInfo{
		Name:   "pkg.X",
		Fields: []string{
			"A",
			"B",
		},
	}
}
func (x *X) StateSave(m Sink) {
	m.Save(0, &x.A) // Field is A.
	m.Save(1, &x.B) // Field is B.
}
func (x *X) StateLoad(m Source) {
	m.Load(0, &x.A) // Field is A.
	m.Load(1, &x.B) // Field is B.
}

func (Sink) SaveValue

func (s Sink) SaveValue(slot int, obj interface{})

SaveValue adds the given object value to the map.

This should be used for values where pointers are not available, or casts are required during Save/Load.

For example, if we want to cast external package type P.Foo to int64:

func (x *X) StateSave(m Sink) {
	m.SaveValue(0, "A", int64(x.A))
}
func (x *X) StateLoad(m Source) {
	m.LoadValue(0, new(int64), func(x interface{}) {
		x.A = P.Foo(x.(int64))
	})
}

type Source

type Source struct {
	// contains filtered or unexported fields
}

Source is used for Type.StateLoad.

func (Source) AfterLoad

func (s Source) AfterLoad(fn func())

AfterLoad schedules a function execution when all objects have been allocated and their automated loading and customized load logic have been executed. fn will not be executed until all of current object's dependencies' AfterLoad() logic, if exist, have been executed.

func (Source) Context

func (s Source) Context() context.Context

Context returns the context object provided at load time.

func (Source) Load

func (s Source) Load(slot int, objPtr interface{})

Load loads the given object passed as a pointer..

See Sink.Save for an example.

func (Source) LoadValue

func (s Source) LoadValue(slot int, objPtr interface{}, fn func(interface{}))

LoadValue loads the given object value from the map.

See Sink.SaveValue for an example.

func (Source) LoadWait

func (s Source) LoadWait(slot int, objPtr interface{})

LoadWait loads the given objects from the map, and marks it as requiring all AfterLoad executions to complete prior to running this object's AfterLoad.

See Sink.Save for an example.

type Stats

type Stats struct {
	// contains filtered or unexported fields
}

Stats tracks encode / decode timing.

This currently provides a meaningful String function and no other way to extract stats about individual types.

All exported receivers accept nil.

func Load

func Load(ctx context.Context, r wire.Reader, rootPtr interface{}) (Stats, error)

Load loads a checkpoint.

func Save

func Save(ctx context.Context, w wire.Writer, rootPtr interface{}) (Stats, error)

Save saves the given object state.

func (*Stats) String

func (s *Stats) String() string

String returns a table representation of the stats.

type Type

type Type interface {
	// StateTypeName returns the type's name.
	//
	// This is used for matching type information during encoding and
	// decoding, as well as dynamic interface dispatch. This should be
	// globally unique.
	StateTypeName() string

	// StateFields returns information about the type.
	//
	// Fields is the set of fields for the object. Calls to Sink.Save and
	// Source.Load must be made in-order with respect to these fields.
	//
	// This will be called at most once per serialization.
	StateFields() []string
}

Type is an interface that must be implemented by Struct objects. This allows these objects to be serialized while minimizing runtime reflection required.

All these methods can be automatically generated by the go_statify tool.

Directories

Path Synopsis
Package pretty is a pretty-printer for state streams.
Package pretty is a pretty-printer for state streams.
Package statefile defines the state file data stream.
Package statefile defines the state file data stream.
Package tests tests the state packages.
Package tests tests the state packages.
Package wire contains a few basic types that can be composed to serialize graph information for the state package.
Package wire contains a few basic types that can be composed to serialize graph information for the state package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL