record

README

Heirarchical Caching Data Model

The goals of the record package and the model generator are:

1. Explicitly define a conceptual data model
2. Generate types that mirror the conceptual data model
3. Move caching into the data model

Conceptual Data Model

Entities

An entity is a record that is composed of a set of attributes. There are two basic categories of attributes: complex and simple. A complex attribute is one that itself has attributes, such as an entity or a chain. A simple attribute represents a single value or a simple collection of values. Since an attribute can be an entity, which itself has attributes, the data model is heirarchical.

Simple Attributes

Simple attributes are state or index attributes. A state attribute is considered to be part of the proper state of the entity, whereas index attributes exist soley to index other records.

Simple attributes can be atomic values, sets, or lists. Atomic values may themselves be structured, but they are persisted atomically so they are considered to be a single value by the CDM. Sets and lists are ordered and unordered collections of values, respectively.

Parameterized Attributes

An attribute of an entity may be parameterized. A parameterized attribute is effectively a map or dictionary where the key is a tuple of the parameters. For example, the root entity in a personnel database may have an attribute, Person, with the a string parameter, Name. accessed as model.Person(name).

Implementation

The generator produces a struct type for every entity, with a method for every attribute. The return type for a complex attribute is the corresponding entity struct type. The return type for a simple attribute is one of the simple attribute types defined in the record package. Parameterized attributes are backed by a map and single-valued attributes are backed by a simple field. The first time an attribute is accessed, an object of the appropriate type is created. Subsequent accesses will return the same object.

Example

The following is an example of a data model that could be used to track personnel.

- name: Model
  type: entity
  root: true
  attributes:
    - name: Person
      type: entity
      parameters:
        - name: Surname
          type: string
        - name: GivenName
          type: string
      attributes:
        - name: State
          type: state
          dataType: PersonState
          pointer: true
        - name: Dependents
          type: state
          dataType: string
          collection: set

type Model struct{...}

func (*Model) Person(surname, givenName string) *ModelPerson

type ModelPerson struct{...}

func (*ModelPerson) State() *record.Value[*PersonState]
func (*ModelPerson) Dependents() *record.Set[string]

Documentation

Index

• Variables
• func CompareHash(u, v [32]byte) int
• func CompareTxid(u, v *url.TxID) int
• func CompareUrl(u, v *url.URL) int
• func MapKeyBytes(v []byte) [32]byte
• func MapKeyUrl(v *url.URL) [32]byte
• func ParseString(s string) (string, error)
• func Struct[T any, PT ptrBinaryValue[T]]() encodableValue[PT]
• func Union[T encoding.BinaryValue](unmarshal func([]byte) (T, error)) encodableValue[T]
• func UnionFactory[T encoding.BinaryValue](unmarshal func([]byte) (T, error)) func() encodableValue[T]
• func Wrapped[T any](funcs *wrapperFuncs[T]) encodableValue[T]
• func WrappedFactory[T any](funcs *wrapperFuncs[T]) func() encodableValue[T]
• type ChainHead
  • func (v *ChainHead) Copy() *ChainHead
  • func (v *ChainHead) CopyAsInterface() interface{}
  • func (v *ChainHead) Equal(u *ChainHead) bool
  • func (v *ChainHead) MarshalJSON() ([]byte, error)
  • func (v *ChainHead) UnmarshalJSON(data []byte) error
• type Counted
  • func NewCounted[T any](logger log.Logger, store Store, key Key, namefmt string, ...) Counted[T]
• type Key
  • func (k Key) Append(v ...interface{}) Key
  • func (k Key) Hash() storage.Key
  • func (k Key) MarshalJSON() ([]byte, error)
  • func (k Key) String() string
• type KvStore
  • func (s KvStore) GetValue(key Key, value ValueWriter) error
  • func (s KvStore) PutValue(key Key, value ValueReader) error
• type List
  • func NewList[T any](logger log.Logger, store Store, key Key, namefmt string, ...) List[T]
• type Record
• type Set
  • func NewSet[T any](logger log.Logger, store Store, key Key, namefmt string, ...) Set[T]
• type Store
• type Value
  • func NewValue[T any](logger log.Logger, store Store, key Key, name string, allowMissing bool, ...) Value[T]
• type ValueMarshaller
• type ValueReader
• type ValueUnmarshaller
• type ValueWriter

Variables

var BytesWrapper = &wrapperFuncs[[]byte]{
	copy:      func(v []byte) []byte { u := make([]byte, len(v)); copy(u, v); return u },
	marshal:   oldMarshal(encoding.MarshalBytes),
	unmarshal: encoding.UnmarshalBytes,
}

BytesWrapper defines un/marshalling functions for byte slice fields.

var ErrNotEnoughData = errors.BadRequest.With("not enough data")

var ErrOverflow = errors.BadRequest.With("overflow")

var HashWrapper = &wrapperFuncs[[32]byte]{
	copy:      copyValue[[32]byte],
	marshal:   oldMarshalPtr(encoding.MarshalHash),
	unmarshal: encoding.UnmarshalHash,
}

HashWrapper defines un/marshalling functions for hash fields.

var StringWrapper = &wrapperFuncs[string]{
	copy:      copyValue[string],
	marshal:   oldMarshal(encoding.MarshalString),
	unmarshal: encoding.UnmarshalString,
}

StringWrapper defines un/marshalling functions for string fields.

var TxidWrapper = &wrapperFuncs[*url.TxID]{
	copy:      copyValue[*url.TxID],
	marshal:   marshalAsString[*url.TxID],
	unmarshal: unmarshalFromString(url.ParseTxID),
}

TxidWrapper defines un/marshalling functions for txid fields.

var UintWrapper = &wrapperFuncs[uint64]{
	copy:      copyValue[uint64],
	marshal:   oldMarshal(encoding.MarshalUint),
	unmarshal: encoding.UnmarshalUint,
}

UintWrapper defines un/marshalling functions for uint fields.

var UrlWrapper = &wrapperFuncs[*url.URL]{
	copy:      copyValue[*url.URL],
	marshal:   marshalAsString[*url.URL],
	unmarshal: unmarshalFromString(url.Parse),
}

UrlWrapper defines un/marshalling functions for url fields.

Functions

func CompareHash

func CompareHash(u, v [32]byte) int

func CompareTxid

func CompareTxid(u, v *url.TxID) int

func CompareUrl

func CompareUrl(u, v *url.URL) int

func MapKeyBytes

func MapKeyBytes(v []byte) [32]byte

func MapKeyUrl

func MapKeyUrl(v *url.URL) [32]byte

func ParseString

func ParseString(s string) (string, error)

func Struct

func Struct[T any, PT ptrBinaryValue[T]]() encodableValue[PT]

Struct returns an encodable value for the given encodable struct-type.

func Union

func Union[T encoding.BinaryValue](unmarshal func([]byte) (T, error)) encodableValue[T]

Union returns an encodable value for the given encodable union type.

func UnionFactory

func UnionFactory[T encoding.BinaryValue](unmarshal func([]byte) (T, error)) func() encodableValue[T]

UnionFactory curries Union.

func Wrapped

func Wrapped[T any](funcs *wrapperFuncs[T]) encodableValue[T]

Wrapped returns an encodable value for the given type using the given wrapper functions.

func WrappedFactory

func WrappedFactory[T any](funcs *wrapperFuncs[T]) func() encodableValue[T]

WrappedFactory curries Wrapped.

Types

type ChainHead

type ChainHead struct {
	Count    int64    `json:"count,omitempty" form:"count" query:"count" validate:"required"`
	Pending  [][]byte `json:"pending,omitempty" form:"pending" query:"pending" validate:"required"`
	HashList [][]byte `json:"hashList,omitempty" form:"hashList" query:"hashList" validate:"required"`
}

func (*ChainHead) Copy

func (v *ChainHead) Copy() *ChainHead

func (*ChainHead) CopyAsInterface

func (v *ChainHead) CopyAsInterface() interface{}

func (*ChainHead) Equal

func (v *ChainHead) Equal(u *ChainHead) bool

func (*ChainHead) MarshalJSON

func (v *ChainHead) MarshalJSON() ([]byte, error)

func (*ChainHead) UnmarshalJSON

func (v *ChainHead) UnmarshalJSON(data []byte) error

type Counted

type Counted[T any] interface {
	Record
	Get(int) (T, error)
	Put(T) error
	Count() (int, error)
	GetAll() ([]T, error)
	Last() (int, T, error)
	Overwrite([]T) error
}

Counted records an insertion-ordered list of values as separate records plus a record for the count.

func NewCounted

func NewCounted[T any](logger log.Logger, store Store, key Key, namefmt string, new func() encodableValue[T]) Counted[T]

NewCounted returns a new counted using the given encodable value type.

type Key

type Key []interface{}

A Key is the key for a record.

func (Key) Append

func (k Key) Append(v ...interface{}) Key

Append creates a child key of this key.

func (Key) Hash

func (k Key) Hash() storage.Key

Hash converts the record key to a storage key.

func (Key) MarshalJSON

func (k Key) MarshalJSON() ([]byte, error)

MarshalJSON is implemented so keys are formatted nicely by zerolog.

func (Key) String

func (k Key) String() string

String returns a human-readable string for the key.

type KvStore

type KvStore struct {
	Store storage.KeyValueTxn
}

KvStore is a Store that reads/writes from/to a key-value store.

func (KvStore) GetValue

func (s KvStore) GetValue(key Key, value ValueWriter) error

GetValue loads the raw value and calls value.LoadBytes.

func (KvStore) PutValue

func (s KvStore) PutValue(key Key, value ValueReader) error

PutValue marshals the value and stores it.

type List

type List[T any] interface {
	Value[[]T]
	Add(...T) error
}

List records an unordered list of values as a single record.

func NewList

func NewList[T any](logger log.Logger, store Store, key Key, namefmt string, encoder encodableValue[T]) List[T]

NewList returns a new list using the given encoder and comparison.

type Record

type Record interface {
	// Resolve resolves the record or a child record.
	Resolve(key Key) (Record, Key, error)
	// IsDirty returns true if the record has been modified.
	IsDirty() bool
	// Commit writes any modifications to the store.
	Commit() error
}

A Record is a component of a data model.

type Set

type Set[T any] interface {
	List[T]
	Remove(T) error
	Index(T) (int, error)
	Find(T) (T, error)
}

Set records an ordered list of values as a single record.

func NewSet

func NewSet[T any](logger log.Logger, store Store, key Key, namefmt string, encoder encodableValue[T], cmp func(u, v T) int) Set[T]

NewSet returns a new set using the given encoder and comparison.

type Store

type Store interface {
	// GetValue loads the value from the underlying store and writes it. Byte
	// stores call LoadBytes(data) and value stores call LoadValue(v, false).
	GetValue(key Key, value ValueWriter) error
	// PutValue gets the value from the reader and stores it. A byte store
	// marshals the value and stores the bytes. A value store finds the
	// appropriate value and calls LoadValue(v, true).
	PutValue(key Key, value ValueReader) error
}

A Store loads and stores values.

type Value

type Value[T any] interface {
	Record
	Key(int) any
	Get() (T, error)
	GetAs(any) error
	Put(T) error
}

Value records a value.

func NewValue

func NewValue[T any](logger log.Logger, store Store, key Key, name string, allowMissing bool, ev encodableValue[T]) Value[T]

NewValue returns a new value using the given encodable value.

type ValueMarshaller

type ValueMarshaller[T any] func(value T) ([]byte, error)

type ValueReader

type ValueReader interface {
	// GetValue returns the value.
	GetValue() (value encoding.BinaryValue, version int, err error)
}

A ValueReader holds a readable value.

type ValueUnmarshaller

type ValueUnmarshaller[T any] func(data []byte) (T, error)

type ValueWriter

type ValueWriter interface {
	// LoadValue stores the value of the reader into the receiver.
	LoadValue(value ValueReader, put bool) error
	// LoadBytes unmarshals a value from bytes into the receiver.
	LoadBytes(data []byte) error
}

A ValueWriter holds a writable value.