kvo

package
v0.3.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 26, 2024 License: MIT Imports: 8 Imported by: 0

Documentation

Overview

Package kvo (stands for Key Value Objects) manages trees of objects mainly represented as key-value pairs.

It has 3 goals:

1. Manipulation of data objects in a generic way (e.g. handling all integer ledger stats with the same code).

2. A compact binary data representation that can be read from a memory-mapped file without deserialization.

3. A diff/change format that can be used within a transaction (including to serve reads) and then quickly merged with the memory-mapped data when doing updates.

Potentially we also want to have a diff format that can be persisted long-term as part of change history. This might or might not match #3.

Index

Constants

View Source 
const TimeOffsetMicros = 62_135_596_800_000_000

TimeOffsetMicros is offset to Time.UnixMicro() that kvo stores, chosen such that time.Time{}.UnixMicro() = -TimeOffsetMicros. With this offset, 0 micros correspond to zero time instead of Unix epoch, and we can treat all times as unsigned integers.

Variables

View Source 
var (
	Int64  = NewIntType[int64]("int64")
	Uint64 = NewIntType[uint64]("uint64")
)

Functions

func Dump added in v0.3.1 

func Dump(m AnyMap) string

func Hex added in v0.3.3 

func Hex(v uint64) string

func HexString 

func HexString(data []uint64) string

func TimeToUint64 added in v0.3.1 

func TimeToUint64(t time.Time) uint64

func Uint64ToTime added in v0.3.1 

func Uint64ToTime(u uint64) time.Time

Types

type AnyMap 

type AnyMap interface {
	Model() *Model
	IsMissing() bool
	Keys() []uint64
	Get(key uint64) uint64
	GetAnyMap(key uint64) AnyMap
	Dump() string
}

type AnyRecord 

type AnyRecord interface {
	AnyRoot() AnyMap
	Packable
}

type AnyType 

type AnyType interface {
	Name() string
	// contains filtered or unexported methods
}

type FloatValue 

type FloatValue interface {
	float32 | ~float64
}

type ImmutableMap 

type ImmutableMap struct {
	// contains filtered or unexported fields
}

func (ImmutableMap) Dump added in v0.3.1 

func (m ImmutableMap) Dump() string

func (ImmutableMap) Get 

func (m ImmutableMap) Get(key uint64) uint64

func (ImmutableMap) GetAnyMap 

func (m ImmutableMap) GetAnyMap(key uint64) AnyMap

func (ImmutableMap) GetMap added in v0.3.3 

func (m ImmutableMap) GetMap(key uint64) ImmutableMap

func (ImmutableMap) IsMissing 

func (m ImmutableMap) IsMissing() bool

func (ImmutableMap) Items added in v0.3.3 

func (m ImmutableMap) Items() func(yield func(k, v uint64) bool)

func (ImmutableMap) KeyModel 

func (m ImmutableMap) KeyModel(key uint64) *Model

func (ImmutableMap) Keys added in v0.3.1 

func (m ImmutableMap) Keys() []uint64

func (ImmutableMap) Model added in v0.3.1 

func (m ImmutableMap) Model() *Model

func (ImmutableMap) RecordData added in v0.3.1 

func (m ImmutableMap) RecordData() ImmutableRecordData

func (ImmutableMap) RecordWithThisRoot added in v0.3.1 

func (m ImmutableMap) RecordWithThisRoot() ImmutableRecord

type ImmutableObjectData 

type ImmutableObjectData []uint64

ImmutableObjectData represents a single object within ImmutableRecordData. It can be a map, an array or a set. Could even be a string or a blob, but we'd lack the exact size info, and those are better represented as []byte.

func (ImmutableObjectData) MapGet 

func (m ImmutableObjectData) MapGet(key uint64) uint64

func (ImmutableObjectData) MapKeys added in v0.3.1 

func (m ImmutableObjectData) MapKeys() []uint64

type ImmutableRecord 

type ImmutableRecord struct {
	// contains filtered or unexported fields
}

ImmutableRecord is a high-level interface to immutableRecordData, representing an immutable a tree of objects in a binary format suitable for storing on disk and optimized for random access.

func EmptyRecord 

func EmptyRecord(rootModel *Model) ImmutableRecord

func LoadRecord 

func LoadRecord(data []byte, rootModel *Model) ImmutableRecord

func (ImmutableRecord) AnyRoot 

func (r ImmutableRecord) AnyRoot() AnyMap

func (ImmutableRecord) Data 

func (r ImmutableRecord) Data() ImmutableRecordData

func (ImmutableRecord) Pack 

func (r ImmutableRecord) Pack() ImmutableRecordData

func (ImmutableRecord) Root 

func (r ImmutableRecord) Root() ImmutableMap

type ImmutableRecordData 

type ImmutableRecordData []uint64

ImmutableRecordData provides for random access to an immutable tree of objects stored in a binary format suitable for mmap'ing.

The format is as follows: 

 record -> flags:32 count:32 (objectKind:4 offset:28 size:32)... object...

 object -> map | arrayOrSet | stringOrBlob

	map -> key1 ... keyN valueOrRef1 ... valueOrRefN  (N determined by map size; keys are sorted)
	arrayOrSet -> valueOrRef1 ... valueOrRefN  (N is determined by array/set size; set elements are sorted)
	stringOrBlob is just raw bytes  (this is why object directory stores sizes in bytes)

 valueOrRef -> uint:64 | subobject:64

	subobject -> inlineStringOrData:64 | objectRef:64
	  inlineStringOrData -> alwaysOne:1 zeros:4 byteCount:3 inlineBytes:[7]byte
	  objectRef -> alwaysZero:1 zeroes:31 objectIndex:32

Note that only maps are implements so far.

func LoadRecordData 

func LoadRecordData(data []byte) ImmutableRecordData

func (ImmutableRecordData) Bytes 

func (r ImmutableRecordData) Bytes() []byte

func (ImmutableRecordData) HexString 

func (r ImmutableRecordData) HexString() string

func (ImmutableRecordData) Object 

func (r ImmutableRecordData) Object(i int) ImmutableObjectData

func (ImmutableRecordData) ObjectCount 

func (r ImmutableRecordData) ObjectCount() int

func (ImmutableRecordData) Pack 

func (r ImmutableRecordData) Pack() ImmutableRecordData

func (ImmutableRecordData) Record 

func (data ImmutableRecordData) Record(rootModel *Model) ImmutableRecord

func (ImmutableRecordData) RootObject added in v0.3.1 

func (r ImmutableRecordData) RootObject() ImmutableObjectData

func (ImmutableRecordData) Uints 

func (r ImmutableRecordData) Uints() []uint64

type IntegerValue 

type IntegerValue interface {
	~int | ~int8 | ~int16 | ~int32 | ~int64 | ~uint | ~uint8 | ~uint16 | ~uint32 | ~uint64 | ~uintptr
}

type List 

type List struct {
	// contains filtered or unexported fields
}

TODO

type Model 

type Model struct {
	// contains filtered or unexported fields
}

func NewModel 

func NewModel(schema *Schema, compositeType *Type[any], build func(b *ModelBuilder)) *Model

func (*Model) MustPropByCode 

func (model *Model) MustPropByCode(key uint64) PropImpl

func (*Model) Name 

func (model *Model) Name() string

type ModelBuilder 

type ModelBuilder struct {
	// contains filtered or unexported fields
}

func (*ModelBuilder) Prop 

func (b *ModelBuilder) Prop(propCode PropCode, opts ...any)

type ModelDefinition 

type ModelDefinition struct {
	Name  string
	Props []*PropInstance
}

type MutableMap 

type MutableMap struct {
	// contains filtered or unexported fields
}

MutableMap allows mutating a map within a MutableRecord.

func NewRecord 

func NewRecord(model *Model) MutableMap

NewRecord sets up an empty mutable record with the given root model and an empty root map, and returns the root map.

func UpdateRecord 

func UpdateRecord(orig ImmutableRecord) MutableMap

UpdateRecord sets up a mutable variant of the given record and returns its root map.

func (MutableMap) Dump added in v0.3.1 

func (m MutableMap) Dump() string

func (MutableMap) Get 

func (m MutableMap) Get(key uint64) uint64

func (MutableMap) GetAnyMap 

func (m MutableMap) GetAnyMap(key uint64) AnyMap

func (MutableMap) IsEmpty added in v0.3.1 

func (m MutableMap) IsEmpty() bool

func (MutableMap) IsMissing 

func (m MutableMap) IsMissing() bool

func (MutableMap) Keys added in v0.3.1 

func (m MutableMap) Keys() []uint64

func (MutableMap) Model added in v0.3.1 

func (m MutableMap) Model() *Model

func (MutableMap) Record 

func (m MutableMap) Record() *MutableRecord

func (MutableMap) Set 

func (m MutableMap) Set(key, value uint64)

func (MutableMap) UpdateMap 

func (m MutableMap) UpdateMap(key uint64) MutableMap

type MutableRecord 

type MutableRecord struct {
	// contains filtered or unexported fields
}

MutableRecord is a builder of ImmutableRecordDatas. It represents a delta of a tree of objects in a way that's fast to write and reasonably fast to read.

(E.g. we're using unsorted keys, so reading performance could be improved by maybe sorting keys before reading.)

func (*MutableRecord) IsDirty added in v0.3.1 

func (rec *MutableRecord) IsDirty() bool

func (*MutableRecord) Original 

func (rec *MutableRecord) Original() ImmutableRecord

func (*MutableRecord) Pack 

func (rec *MutableRecord) Pack() ImmutableRecordData

Pack produces an on-disk binary encoding of the updated record, merging the changes recorded in MutableRecord with the original record.

func (*MutableRecord) PackedRecord added in v0.3.2 

func (rec *MutableRecord) PackedRecord() ImmutableRecord

func (*MutableRecord) PackedRoot added in v0.3.1 

func (rec *MutableRecord) PackedRoot() ImmutableMap

func (*MutableRecord) Root 

func (rec *MutableRecord) Root() MutableMap

type Packable 

type Packable interface {
	// Pack returns ImmutableRecordData of this record. Must return nil if
	// called on a nil struct pointer.
	Pack() ImmutableRecordData
}

type PropBuilder 

type PropBuilder struct{}

type PropCode 

type PropCode = uint64

func NewProp 

func NewProp[T any](schema *Schema, code PropCode, name string, typ *Type[T], build func(b *PropBuilder)) PropCode

type PropFlag 

type PropFlag int
const (
	PropDense PropFlag = 1 << iota
	PropRequired
)

type PropImpl 

type PropImpl interface {
	Name() string
	Code() PropCode
	AnyType() AnyType
	TypeModel() *Model
}

type PropInstance 

type PropInstance struct {
	Prop     PropImpl
	Dense    bool
	Required bool
	// contains filtered or unexported fields
}

func (*PropInstance) PropInstance 

func (pi *PropInstance) PropInstance() *PropInstance

type PropOption 

type PropOption interface {
	// contains filtered or unexported methods
}

type PropOptions 

type PropOptions struct {
}

type ScalarConverter 

type ScalarConverter[T any] interface {
	ValueToScalar(value T) uint64
	ScalarToValue(scalar uint64) T
}

type ScalarProp 

type ScalarProp[T any] struct {
	// contains filtered or unexported fields
}

func (*ScalarProp[T]) AnyType 

func (p *ScalarProp[T]) AnyType() AnyType

func (*ScalarProp[T]) Code 

func (p *ScalarProp[T]) Code() PropCode

func (*ScalarProp[T]) Name 

func (p *ScalarProp[T]) Name() string

func (*ScalarProp[T]) ScalarToValue 

func (p *ScalarProp[T]) ScalarToValue(scalar uint64) T

func (*ScalarProp[T]) TypeModel 

func (p *ScalarProp[T]) TypeModel() *Model

func (*ScalarProp[T]) ValueToScalar 

func (p *ScalarProp[T]) ValueToScalar(value T) uint64

type Schema 

type Schema struct {
	// contains filtered or unexported fields
}

func (*Schema) MustPropByCode 

func (schema *Schema) MustPropByCode(code PropCode) PropImpl

func (*Schema) PropByCode 

func (schema *Schema) PropByCode(code PropCode) PropImpl

type Type 

type Type[T any] struct {
	// contains filtered or unexported fields
}

func NewEntityType 

func NewEntityType(schema *Schema, name string) *Type[any]

func NewFloatType 

func NewFloatType[T FloatValue](name string) *Type[T]

func NewIntType 

func NewIntType[T IntegerValue](name string) *Type[T]

func NewScalarSubtype 

func NewScalarSubtype[T any](name string, base *Type[T]) *Type[T]

func NewScalarType 

func NewScalarType[T any](name string, conv ScalarConverter[T]) *Type[T]

func (*Type[T]) Name 

func (typ *Type[T]) Name() string

func (*Type[T]) ScalarConverter 

func (typ *Type[T]) ScalarConverter() ScalarConverter[T]

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.