ard

package
v0.1.68 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 29, 2023 License: Apache-2.0 Imports: 19 Imported by: 0

README

Agnostic Raw Data (ARD)

This library is also implemented in Python.

And check out the ardconv ARD conversion tool.

What is "agnostic raw data"?

Agnostic

ARD comprises data types that are "agnostic", meaning that they can be trivially used by practically any programming language, stored in practically any database, and can also be transmitted in a wide variety of formats.

The following data types are supported:

  • strings (Unicode)
  • byte arrays
  • signed integers
  • unsigned integers
  • floats
  • booleans
  • nulls

As well as two nestable structures:

  • lists
  • maps (unordered)

Note that map keys do not have to be strings and indeed can be arbitrarily complex. Such keys might be impossible to use in hashtable implementations in some programming languages. In such cases maps can be stored as lists of key/value tuples.

Raw

Data validation is out of scope for ARD. There's no schema. The idea is to support arbitrary data of any structure and size. Once the ARD is made available other layers can validate its structure and otherwise process the values.

This library does support such schema validation via conversion to Go structs using a reflector.

Data

This is about data as opposed to the representation of data. What's the difference? ARD does not define how the data is stored or transmitted. Thus ARD in itself is not concerned with the endiannes or precision of integers and floats, and also not concerned with character encodings (compare the Unicode standard for data vs. the UTF-8 standard for encoding that data).

ARD and Representation Formats

CBOR and MessagePack

CBOR and MessagePack support everything! Though note that they are not human-readable.

YAML

YAML supports a rich set of primitive types (when it includes the common JSON schema), so most ARD will survive a round trip to YAML.

YAML, however, does not distinguish between signed and unsigned integers.

Byte arrays can also be problematic. Some parsers support the optional !!binary type, but others may not. Encoded strings (e.g. using Base64) can be used instead to ensure portability.

Also note that some YAML 1.1 implementations support ordered maps (!!omap vs. !!map). These will lose their order when converted to ARD, so it's best to standardize on arbitrary order (!!map). YAML 1.2 does not support !!omap by default, so this use case may become less and less common.

JSON

JSON can be read into ARD. However, because JSON has fewer types and more limitations than YAML (no signed and unsigned integers, only floats; map keys can only be strings), ARD will lose quite a bit of type information when translated into JSON.

We overcome this challenge by extending JSON with some conventions for encoding extra types. See our conventions here or in the Python ARD library.

XML

XML does not have a type system. Arbitrary XML cannot be parsed into ARD.

However, we support certain conventions that enforce such compatibility.

ARD and Programming Languages

Go

Unfortunately, the most popular Go YAML parser does not easily support arbitrarily complex keys (see this issue). We provide an independent library, yamlkeys, to make this easier.

Python

Likewise, the Python ruamel.yaml parser does not easily support arbitrarily complex keys. We solve this by extending ruamel.yaml in our Python ARD library.

JavaScript

See the discussion of JSON, above (JSON stands for "JavaScript Object Notation"). A straightforward way to work with ARD in JavaScript is via our ARD-compatible JSON conventions. However, it may also be possible to create a library of classes to support ARD features.

Documentation

Index

Constants

View Source 
const (
	CompatibleJSONIntegerCode  = "$ard.integer"
	CompatibleJSONUIntegerCode = "$ard.uinteger"
	CompatibleJSONBytesCode    = "$ard.bytes"
	CompatibleJSONMapCode      = "$ard.map"
)
View Source 
const (
	FieldPathType = iota
	MapPathType
	ListPathType
	SequencedListPathType
)
View Source 
const (
	CompatibleXMLNilTag           = "nil"
	CompatibleXMLBytesTag         = "bytes"
	CompatibleXMLListTag          = "list"
	CompatibleXMLMapTag           = "map"
	CompatibleXMLMapEntryTag      = "entry"
	CompatibleXMLMapEntryKeyTag   = "key"
	CompatibleXMLMapEntryValueTag = "value"
)

Variables

View Source 
var NoNode = &Node{nil, nil, "", false, false}
View Source 
var TypeValidators = map[TypeName]TypeValidator{
	TypeMap:       IsMap,
	TypeList:      IsList,
	TypeString:    IsString,
	TypeBoolean:   IsBoolean,
	TypeInteger:   IsInteger,
	TypeFloat:     IsFloat,
	TypeNull:      IsNull,
	TypeBytes:     IsBytes,
	TypeTimestamp: IsTimestamp,
}
View Source 
var TypeZeroes = map[TypeName]Value{
	TypeMap:       make(Map),
	TypeList:      List{},
	TypeString:    "",
	TypeBoolean:   false,
	TypeInteger:   int(0),
	TypeFloat:     float64(0.0),
	TypeNull:      nil,
	TypeBytes:     []byte{},
	TypeTimestamp: time.Time{},
}

Functions

func Decode added in v0.1.19 

func Decode(code string, format string, locate bool) (Value, Locator, error)

func DecodeCBOR added in v0.1.20 

func DecodeCBOR(code string) (Value, Locator, error)

The code should be in Base64

func DecodeCompatibleJSON added in v0.1.19 

func DecodeCompatibleJSON(code string, locate bool) (Value, Locator, error)

func DecodeCompatibleJSONBytes added in v0.1.15 

func DecodeCompatibleJSONBytes(code StringMap) ([]byte, bool)

func DecodeCompatibleJSONInteger added in v0.1.15 

func DecodeCompatibleJSONInteger(code StringMap) (int64, bool)

func DecodeCompatibleJSONUInteger added in v0.1.15 

func DecodeCompatibleJSONUInteger(code StringMap) (uint64, bool)

func DecodeCompatibleXML added in v0.1.19 

func DecodeCompatibleXML(code string, locate bool) (Value, Locator, error)

func DecodeJSON added in v0.1.19 

func DecodeJSON(code string, locate bool) (Value, Locator, error)

func DecodeMessagePack added in v0.1.62 

func DecodeMessagePack(code string) (Value, Locator, error)

The code should be in Base64

func DecodeYAML added in v0.1.19 

func DecodeYAML(code string, locate bool) (Value, Locator, error)

func Equals 

func Equals(a Value, b Value) bool

func FindYAMLNode 

func FindYAMLNode(node *yaml.Node, path ...PathElement) *yaml.Node

func FromCompatibleXML added in v0.1.18 

func FromCompatibleXML(element *etree.Element) (any, error)

func IsBoolean 

func IsBoolean(value Value) bool

bool

func IsBytes added in v0.1.62 

func IsBytes(value Value) bool

func IsFloat 

func IsFloat(value Value) bool

float64, float32

func IsInteger 

func IsInteger(value Value) bool

int64, int32, int16, int8, int, uint64, uint32, uint16, uint8, uint

func IsList 

func IsList(value Value) bool

List = []any

func IsMap 

func IsMap(value Value) bool

Map = map[any]any

func IsNull 

func IsNull(value Value) bool

func IsPrimitiveType added in v0.1.60 

func IsPrimitiveType(value Value) bool

func IsString 

func IsString(value Value) bool

string

func IsTimestamp 

func IsTimestamp(value Value) bool

time.Time

func MergeMaps 

func MergeMaps(target Map, source Map, mergeLists bool)

func MergeStringMaps 

func MergeStringMaps(target StringMap, source StringMap, mergeLists bool)

func Read 

func Read(reader io.Reader, format string, locate bool) (Value, Locator, error)

func ReadCBOR added in v0.1.20 

func ReadCBOR(reader io.Reader) (Value, Locator, error)

func ReadCompatibleJSON added in v0.1.16 

func ReadCompatibleJSON(reader io.Reader, locate bool) (Value, Locator, error)

func ReadCompatibleXML added in v0.1.18 

func ReadCompatibleXML(reader io.Reader, locate bool) (Value, Locator, error)

func ReadJSON 

func ReadJSON(reader io.Reader, locate bool) (Value, Locator, error)

func ReadMessagePack added in v0.1.62 

func ReadMessagePack(reader io.Reader) (Value, Locator, error)

func ReadYAML 

func ReadYAML(reader io.Reader, locate bool) (Value, Locator, error)

func StringMapPutNested 

func StringMapPutNested(map_ StringMap, key string, value string) error

TODO: use Node instead

func ToCompatibleXML added in v0.1.18 

func ToCompatibleXML(value any) any

func ToFloat64 added in v0.1.61 

func ToFloat64(value Value) float64

float64, float32

func ToInt64 added in v0.1.61 

func ToInt64(value Value) int64

int64, int32, int16, int8, int

func ToUInt64 added in v0.1.61 

func ToUInt64(value Value) uint64

uint64, uint32, uint16, uint8, uint

func ToYAMLDocumentNode 

func ToYAMLDocumentNode(value Value, verbose bool) (*yaml.Node, error)

func ToYAMLNode 

func ToYAMLNode(value Value, verbose bool) (*yaml.Node, bool)

func ValueToString 

func ValueToString(value Value) string

Types

type CompatibleJSONBytes added in v0.1.15 

type CompatibleJSONBytes []byte

func (CompatibleJSONBytes) MarshalJSON added in v0.1.15 

func (self CompatibleJSONBytes) MarshalJSON() ([]byte, error)

json.Marshaler interface

type CompatibleJSONInteger added in v0.1.15 

type CompatibleJSONInteger int64

func (CompatibleJSONInteger) MarshalJSON added in v0.1.15 

func (self CompatibleJSONInteger) MarshalJSON() ([]byte, error)

json.Marshaler interface

type CompatibleJSONMap added in v0.1.15 

type CompatibleJSONMap Map

func (CompatibleJSONMap) MarshalJSON added in v0.1.15 

func (self CompatibleJSONMap) MarshalJSON() ([]byte, error)

json.Marshaler interface

type CompatibleJSONMapEntry added in v0.1.15 

type CompatibleJSONMapEntry struct {
	Key   Value `json:"key"`
	Value Value `json:"value"`
}

func DecodeCompatibleJSONMapEntry added in v0.1.15 

func DecodeCompatibleJSONMapEntry(entry Value) (*CompatibleJSONMapEntry, bool)

type CompatibleJSONUInteger added in v0.1.15 

type CompatibleJSONUInteger uint64

func (CompatibleJSONUInteger) MarshalJSON added in v0.1.15 

func (self CompatibleJSONUInteger) MarshalJSON() ([]byte, error)

json.Marshaler interface

type CompatibleXMLBytes added in v0.1.62 

type CompatibleXMLBytes struct {
	// contains filtered or unexported fields
}

func (CompatibleXMLBytes) MarshalXML added in v0.1.62 

func (self CompatibleXMLBytes) MarshalXML(encoder *xml.Encoder, start xml.StartElement) error

xml.Marshaler interface

type CompatibleXMLList added in v0.1.18 

type CompatibleXMLList struct {
	Entries []any
}

func (CompatibleXMLList) MarshalXML added in v0.1.18 

func (self CompatibleXMLList) MarshalXML(encoder *xml.Encoder, start xml.StartElement) error

xml.Marshaler interface

type CompatibleXMLMap added in v0.1.18 

type CompatibleXMLMap struct {
	// contains filtered or unexported fields
}

func (CompatibleXMLMap) MarshalXML added in v0.1.18 

func (self CompatibleXMLMap) MarshalXML(encoder *xml.Encoder, start xml.StartElement) error

xml.Marshaler interface

type CompatibleXMLMapEntry added in v0.1.18 

type CompatibleXMLMapEntry struct {
	// contains filtered or unexported fields
}

func NewCompatibleXMLMapEntry added in v0.1.20 

func NewCompatibleXMLMapEntry(element *etree.Element) (CompatibleXMLMapEntry, error)

func (CompatibleXMLMapEntry) MarshalXML added in v0.1.18 

func (self CompatibleXMLMapEntry) MarshalXML(encoder *xml.Encoder, start xml.StartElement) error

xml.Marshaler interface

type CompatibleXMLNil added in v0.1.62 

type CompatibleXMLNil struct{}

func (CompatibleXMLNil) MarshalXML added in v0.1.62 

func (self CompatibleXMLNil) MarshalXML(encoder *xml.Encoder, start xml.StartElement) error

xml.Marshaler interface

type FromARD added in v0.1.62 

type FromARD interface {
	FromARD(reflector *Reflector) (any, error)
}

type List 

type List = []Value

func ReadAllYAML added in v0.1.19 

func ReadAllYAML(reader io.Reader) (List, error)

type Locator 

type Locator interface {
	Locate(path ...PathElement) (int, int, bool)
}

type Map 

type Map = map[Value]Value

func DecodeCompatibleJSONMap added in v0.1.15 

func DecodeCompatibleJSONMap(code StringMap) (Map, bool)

func StringMapToMap added in v0.1.20 

func StringMapToMap(stringMap StringMap) Map

Recursive

type Node 

type Node struct {
	Value Value
	// contains filtered or unexported fields
}

func NewNode 

func NewNode(data any) *Node

func (*Node) Append 

func (self *Node) Append(value Value) bool

func (*Node) Boolean 

func (self *Node) Boolean() (bool, bool)

func (*Node) Bytes added in v0.1.38 

func (self *Node) Bytes() ([]byte, bool)

func (*Node) ConvertSimilar added in v0.1.61 

func (self *Node) ConvertSimilar() *Node

func (*Node) EnsureMap added in v0.1.68 

func (self *Node) EnsureMap(keys ...string) (Map, bool)

func (*Node) Float 

func (self *Node) Float() (float64, bool)

Supports .ConvertSimilar()

func (*Node) Get 

func (self *Node) Get(keys ...string) *Node

func (*Node) Integer 

func (self *Node) Integer() (int64, bool)

Supports .ConvertSimilar()

func (*Node) List 

func (self *Node) List() (List, bool)

func (*Node) Map 

func (self *Node) Map() (Map, bool)

func (*Node) NilMeansZero added in v0.1.61 

func (self *Node) NilMeansZero() *Node

func (*Node) Put 

func (self *Node) Put(key string, value Value) bool

func (*Node) String 

func (self *Node) String() (string, bool)

func (*Node) StringList added in v0.1.30 

func (self *Node) StringList() ([]string, bool)

func (*Node) StringMap 

func (self *Node) StringMap() (StringMap, bool)

Supports .ConvertSimilar()

func (*Node) UnsignedInteger 

func (self *Node) UnsignedInteger() (uint64, bool)

Supports .ConvertSimilar()

type Path 

type Path []PathElement

func (Path) Append 

func (self Path) Append(element PathElement) Path

func (Path) AppendField 

func (self Path) AppendField(name string) Path

func (Path) AppendList 

func (self Path) AppendList(index int) Path

func (Path) AppendMap 

func (self Path) AppendMap(name string) Path

func (Path) AppendSequencedList 

func (self Path) AppendSequencedList(index int) Path

func (Path) String 

func (self Path) String() string

fmt.Stringer interface

type PathElement 

type PathElement struct {
	Type  PathElementType
	Value any // string for FieldPathType and MapPathType, int for ListPathType and SequencedListPathType
}

func NewFieldPathElement 

func NewFieldPathElement(name string) PathElement

func NewListPathElement 

func NewListPathElement(index int) PathElement

func NewMapPathElement 

func NewMapPathElement(name string) PathElement

func NewSequencedListPathElement 

func NewSequencedListPathElement(index int) PathElement

type PathElementType 

type PathElementType uint8

type ReflectField added in v0.1.62 

type ReflectField struct {
	// contains filtered or unexported fields
}

type ReflectFields added in v0.1.62 

type ReflectFields map[string]ReflectField // ARD name

func (ReflectFields) GetField added in v0.1.62 

func (self ReflectFields) GetField(structValue reflect.Value, name string) reflect.Value

type Reflector added in v0.1.61 

type Reflector struct {
	IgnoreMissingStructFields bool
	NilMeansZero              bool
	StructFieldTags           []string // in order
	StructFieldNameMapper     StructFieldNameMapperFunc
	// contains filtered or unexported fields
}

func NewReflector added in v0.1.61 

func NewReflector() *Reflector

func (*Reflector) NewReflectFields added in v0.1.62 

func (self *Reflector) NewReflectFields(type_ reflect.Type) ReflectFields

func (*Reflector) Pack added in v0.1.62 

func (self *Reflector) Pack(value Value, packedValuePtr any) error

Fills in Go struct fields from ARD maps

func (*Reflector) PackReflect added in v0.1.62 

func (self *Reflector) PackReflect(value Value, packedValue reflect.Value) error

func (*Reflector) Unpack added in v0.1.62 

func (self *Reflector) Unpack(packedValue any) (Value, error)

Converts Go structs to ARD maps

func (*Reflector) UnpackReflect added in v0.1.62 

func (self *Reflector) UnpackReflect(packedValue reflect.Value) (Value, error)

type StringMap 

type StringMap = map[string]Value

func EnsureStringMaps 

func EnsureStringMaps(stringMap StringMap) StringMap

Ensure data adheres to map[string]any (JSON encoding does not support map[any]any)

func MapToStringMap added in v0.1.20 

func MapToStringMap(map_ Map) StringMap

Recursive

type StructFieldNameMapperFunc added in v0.1.61 

type StructFieldNameMapperFunc func(fieldName string) string

type ToARD added in v0.1.62 

type ToARD interface {
	ToARD(reflector *Reflector) (any, error)
}

type TypeName 

type TypeName string
const (
	NoType TypeName = ""

	// Failsafe schema: https://yaml.org/spec/1.2/spec.html#id2802346
	TypeMap    TypeName = "ard.map"
	TypeList   TypeName = "ard.list"
	TypeString TypeName = "ard.string"

	// JSON schema: https://yaml.org/spec/1.2/spec.html#id2803231
	TypeBoolean TypeName = "ard.boolean"
	TypeInteger TypeName = "ard.integer"
	TypeFloat   TypeName = "ard.float"

	// Other schemas: https://yaml.org/spec/1.2/spec.html#id2805770
	TypeNull      TypeName = "ard.null"
	TypeBytes     TypeName = "ard.bytes"
	TypeTimestamp TypeName = "ard.timestamp"
)

func GetTypeName 

func GetTypeName(value Value) TypeName

type TypeValidator 

type TypeValidator = func(Value) bool

type Value 

type Value = any

func Canonicalize added in v0.1.21 

func Canonicalize(value Value) (Value, error)

func CopyToARD added in v0.1.61 

func CopyToARD(value Value) (Value, error)

func DecodeYAMLTemplate added in v0.1.19 

func DecodeYAMLTemplate(code string, data any) (Value, error)

func EnsureCompatibleJSON added in v0.1.21 

func EnsureCompatibleJSON(value Value) (Value, error)

func EnsureCompatibleXML added in v0.1.62 

func EnsureCompatibleXML(value Value) (Value, error)

func FromCompatibleJSON added in v0.1.15 

func FromCompatibleJSON(value Value) (Value, bool)

func NormalizeMaps added in v0.1.60 

func NormalizeMaps(value Value) (Value, bool)

Ensure data adheres to the ARD map type (JSON decoding uses map[string]any instead of map[any]any)

func NormalizeMapsCopyToARD added in v0.1.61 

func NormalizeMapsCopyToARD(value Value) (Value, error)

func NormalizeMapsSimpleCopy added in v0.1.60 

func NormalizeMapsSimpleCopy(value Value) Value

func NormalizeStringMaps added in v0.1.60 

func NormalizeStringMaps(value Value) (Value, bool)

func NormalizeStringMapsCopyToARD added in v0.1.61 

func NormalizeStringMapsCopyToARD(value Value) (Value, error)

func NormalizeStringMapsSimpleCopy added in v0.1.60 

func NormalizeStringMapsSimpleCopy(value Value) Value

func Roundtrip added in v0.1.62 

func Roundtrip(value Value, format string) (Value, error)

Default is CBOR

func RoundtripCBOR added in v0.1.21 

func RoundtripCBOR(value Value) (Value, error)

func RoundtripCompatibleJSON added in v0.1.21 

func RoundtripCompatibleJSON(value Value) (Value, error)

func RoundtripCompatibleXML added in v0.1.62 

func RoundtripCompatibleXML(value Value) (Value, error)

func RoundtripJSON added in v0.1.62 

func RoundtripJSON(value Value) (Value, error)

func RoundtripMessagePack added in v0.1.62 

func RoundtripMessagePack(value Value) (Value, error)

func RoundtripYAML added in v0.1.21 

func RoundtripYAML(value Value) (Value, error)

func SimpleCopy added in v0.1.60 

func SimpleCopy(value Value) Value

Will leave primitive and non-ARD types as is

func ToCompatibleJSON added in v0.1.15 

func ToCompatibleJSON(value Value) (Value, bool)

type YAMLLocator 

type YAMLLocator struct {
	RootNode *yaml.Node
}

func NewYAMLLocator 

func NewYAMLLocator(rootNode *yaml.Node) *YAMLLocator

func (*YAMLLocator) Locate 

func (self *YAMLLocator) Locate(path ...PathElement) (int, int, bool)

Locator interface

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.