ard

package
v0.1.45 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 4, 2021 License: Apache-2.0 Imports: 17 Imported by: 0

README

Agnostic Raw Data (ARD)

This library is also implemented in Python.

What is "agnostic raw data"?

Agnostic

ARD comprises data types that are "agnostic", meaning that they can be trivially used by practically any programming language, stored in practically any database, and can also be transmitted in a wide variety of formats.

The following data types are supported:

  • strings (Unicode)
  • byte arrays
  • signed integers
  • unsigned integers
  • floats
  • booleans
  • nulls

As well as two nestable structures:

  • lists
  • maps (unordered)

Note that map keys do not have to be strings and indeed can be arbitrarily complex. Such keys might be impossible to use in hashtable implementations in some programming languages. In such cases maps can be stored as lists of key/value tuples.

Raw

Data validation is out of scope for ARD. There's no schema. The idea is to support arbitrary data of any structure and size. Once the ARD is made available other layers can validate its structure and otherwise process the values.

Data

This is about data as opposed to the representation of data. What's the difference? ARD does not define how the data is stored or transmitted. Thus ARD in itself is not concerned with the endiannes or precision of integers and floats, and also not concerned with character encodings (compare the Unicode standard for data vs. the UTF-8 standard for encoding that data).

ARD and Representation Formats

CBOR

CBOR supports everything! The only caveat is that it is not human-readable.

YAML

YAML supports a rich set of primitive types (when it includes the common JSON schema), so most ARD will survive a round trip to YAML.

YAML, however, does not distinguish between signed and unsigned integers.

Byte arrays can also be problematic. Some parsers support the optional !!binary type, but others may not. Encoded strings (e.g. using Base64) can be used instead to ensure portability.

Also note that some YAML 1.1 implementations support ordered maps (!!omap vs. !!map). These will lose their order when converted to ARD, so it's best to standardize on arbitrary order (!!map). YAML 1.2 does not support !!omap by default, so this use case may become less and less common.

JSON

JSON can be read into ARD. However, because JSON has fewer types and more limitations than YAML (no integers, only floats; map keys can only be strings), ARD will lose quite a bit of type information when translated into JSON.

We can overcome this challenge by extending JSON with some conventions for encoding extra types. See our conventions here or in the Python ARD library.

XML

XML does not have a type system. Arbitrary XML cannot be parsed into ARD.

However, we support certain conventions that enforce such compatibility.

ARD and Programming Languages

Go

Unfortunately, the most popular Go YAML parser does not easily support arbitrarily complex keys (see this issue). We provide an independent library, yamlkeys, to make this easier.

Python

Likewise, the Python ruamel.yaml parser does not easily support arbitrarily complex keys. We solve this by extending ruamel.yaml in our Python ARD library.

JavaScript

See the discussion of JSON, above (JSON stands for "JavaScript Object Notation"). A straightforward way to work with ARD in JavaScript is via our ARD-compatible JSON conventions. However, it may also be possible to create a library of classes to support ARD features.

Documentation

Index

Constants

View Source 
const (
	FieldPathType = iota
	MapPathType
	ListPathType
	SequencedListPathType
)

Variables

View Source 
var CompatibleJSONBytesCode = "$ard.bytes"
View Source 
var CompatibleJSONIntegerCode = "$ard.integer"
View Source 
var CompatibleJSONMapCode = "$ard.map"
View Source 
var CompatibleJSONUIntegerCode = "$ard.uinteger"
View Source 
var CompatibleXMLListTag = "list"
View Source 
var CompatibleXMLMapEntryKeyTag = "key"
View Source 
var CompatibleXMLMapEntryTag = "entry"
View Source 
var CompatibleXMLMapEntryValueTag = "value"
View Source 
var CompatibleXMLMapTag = "map"
View Source 
var NoNode = &Node{nil, nil, ""}
View Source 
var TypeValidators = map[TypeName]TypeValidator{
	TypeMap:       IsMap,
	TypeList:      IsList,
	TypeString:    IsString,
	TypeBoolean:   IsBoolean,
	TypeInteger:   IsInteger,
	TypeFloat:     IsFloat,
	TypeNull:      IsNull,
	TypeTimestamp: IsTimestamp,
}
View Source 
var TypeZeroes = map[TypeName]Value{
	TypeMap:       make(Map),
	TypeList:      List{},
	TypeString:    "",
	TypeBoolean:   false,
	TypeInteger:   int(0),
	TypeFloat:     float64(0.0),
	TypeNull:      nil,
	TypeTimestamp: time.Time{},
}

Functions

func Decode added in v0.1.19 

func Decode(code string, format string, locate bool) (Value, Locator, error)

func DecodeCBOR added in v0.1.20 

func DecodeCBOR(code string, locate bool) (Value, Locator, error)

The code should be in Base64

func DecodeCompatibleJSON added in v0.1.19 

func DecodeCompatibleJSON(code string, locate bool) (Value, Locator, error)

func DecodeCompatibleJSONBytes added in v0.1.15 

func DecodeCompatibleJSONBytes(code StringMap) ([]byte, bool)

func DecodeCompatibleJSONInteger added in v0.1.15 

func DecodeCompatibleJSONInteger(code StringMap) (int64, bool)

func DecodeCompatibleJSONUInteger added in v0.1.15 

func DecodeCompatibleJSONUInteger(code StringMap) (uint64, bool)

func DecodeCompatibleXML added in v0.1.19 

func DecodeCompatibleXML(code string, locate bool) (Value, Locator, error)

func DecodeJSON added in v0.1.19 

func DecodeJSON(code string, locate bool) (Value, Locator, error)

func DecodeYAML added in v0.1.19 

func DecodeYAML(code string, locate bool) (Value, Locator, error)

func Equals 

func Equals(a Value, b Value) bool

func FindYAMLNode 

func FindYAMLNode(node *yaml.Node, path ...PathElement) *yaml.Node

func FromCompatibleXML added in v0.1.18 

func FromCompatibleXML(element *etree.Element) (interface{}, error)

func IsBoolean 

func IsBoolean(value Value) bool

bool

func IsFloat 

func IsFloat(value Value) bool

float64, float32

func IsInteger 

func IsInteger(value Value) bool

int64, int32, int16, int8, int, uint64, uint32, uint16, uint8, uint

func IsList 

func IsList(value Value) bool

List = []interface{}

func IsMap 

func IsMap(value Value) bool

Map = map[interface{}]interface{}

func IsNull 

func IsNull(value Value) bool

func IsString 

func IsString(value Value) bool

string

func IsTimestamp 

func IsTimestamp(value Value) bool

time.Time

func IsValid added in v0.1.20 

func IsValid(value Value) bool

func MergeMaps 

func MergeMaps(target Map, source Map, mergeLists bool)

func MergeStringMaps 

func MergeStringMaps(target StringMap, source StringMap, mergeLists bool)

func Read 

func Read(reader io.Reader, format string, locate bool) (Value, Locator, error)

func ReadCBOR added in v0.1.20 

func ReadCBOR(reader io.Reader, locate bool) (Value, Locator, error)

func ReadCompatibleJSON added in v0.1.16 

func ReadCompatibleJSON(reader io.Reader, locate bool) (Value, Locator, error)

func ReadCompatibleXML added in v0.1.18 

func ReadCompatibleXML(reader io.Reader, locate bool) (Value, Locator, error)

func ReadJSON 

func ReadJSON(reader io.Reader, locate bool) (Value, Locator, error)

func ReadYAML 

func ReadYAML(reader io.Reader, locate bool) (Value, Locator, error)

func StringMapPutNested 

func StringMapPutNested(map_ StringMap, key string, value string) error

TODO: use Node instead

func ToCompatibleXML added in v0.1.18 

func ToCompatibleXML(data interface{}) interface{}

func ToYAMLDocumentNode 

func ToYAMLDocumentNode(value Value, verbose bool) *yaml.Node

func ToYAMLNode 

func ToYAMLNode(value Value, verbose bool) (*yaml.Node, bool)

func ValueToString 

func ValueToString(data Value) string

Types

type CompatibleJSONBytes added in v0.1.15 

type CompatibleJSONBytes []byte

func (CompatibleJSONBytes) MarshalJSON added in v0.1.15 

func (self CompatibleJSONBytes) MarshalJSON() ([]byte, error)

json.Marshaler interface

type CompatibleJSONInteger added in v0.1.15 

type CompatibleJSONInteger int64

func (CompatibleJSONInteger) MarshalJSON added in v0.1.15 

func (self CompatibleJSONInteger) MarshalJSON() ([]byte, error)

json.Marshaler interface

type CompatibleJSONMap added in v0.1.15 

type CompatibleJSONMap Map

func (CompatibleJSONMap) MarshalJSON added in v0.1.15 

func (self CompatibleJSONMap) MarshalJSON() ([]byte, error)

json.Marshaler interface

type CompatibleJSONMapEntry added in v0.1.15 

type CompatibleJSONMapEntry struct {
	Key   Value `json:"key"`
	Value Value `json:"value"`
}

func DecodeCompatibleJSONMapEntry added in v0.1.15 

func DecodeCompatibleJSONMapEntry(entry Value) (*CompatibleJSONMapEntry, bool)

type CompatibleJSONUInteger added in v0.1.15 

type CompatibleJSONUInteger uint64

func (CompatibleJSONUInteger) MarshalJSON added in v0.1.15 

func (self CompatibleJSONUInteger) MarshalJSON() ([]byte, error)

json.Marshaler interface

type CompatibleXMLList added in v0.1.18 

type CompatibleXMLList struct {
	Entries []interface{}
}

func (CompatibleXMLList) MarshalXML added in v0.1.18 

func (self CompatibleXMLList) MarshalXML(encoder *xml.Encoder, start xml.StartElement) error

xml.Marshaler interface

type CompatibleXMLMap added in v0.1.18 

type CompatibleXMLMap struct {
	Entries []CompatibleXMLMapEntry
}

func (CompatibleXMLMap) MarshalXML added in v0.1.18 

func (self CompatibleXMLMap) MarshalXML(encoder *xml.Encoder, start xml.StartElement) error

xml.Marshaler interface

type CompatibleXMLMapEntry added in v0.1.18 

type CompatibleXMLMapEntry struct {
	Key   interface{}
	Value interface{}
}

func NewCompatibleXMLMapEntry added in v0.1.20 

func NewCompatibleXMLMapEntry(element *etree.Element) (CompatibleXMLMapEntry, error)

func (CompatibleXMLMapEntry) MarshalXML added in v0.1.18 

func (self CompatibleXMLMapEntry) MarshalXML(encoder *xml.Encoder, start xml.StartElement) error

xml.Marshaler interface

type List 

type List = []Value

func ReadAllYAML added in v0.1.19 

func ReadAllYAML(reader io.Reader) (List, error)

type Locator 

type Locator interface {
	Locate(path ...PathElement) (int, int, bool)
}

type Map 

type Map = map[Value]Value

func DecodeCompatibleJSONMap added in v0.1.15 

func DecodeCompatibleJSONMap(code StringMap) (Map, bool)

func StringMapToMap added in v0.1.20 

func StringMapToMap(stringMap StringMap) Map

Recursive

type Node 

type Node struct {
	Data interface{}
	// contains filtered or unexported fields
}

func NewNode 

func NewNode(data interface{}) *Node

func (*Node) Append 

func (self *Node) Append(value interface{}) bool

func (*Node) Boolean 

func (self *Node) Boolean(allowNil bool) (bool, bool)

func (*Node) Bytes added in v0.1.38 

func (self *Node) Bytes(allowNil bool) ([]byte, bool)

func (*Node) Float 

func (self *Node) Float(allowNil bool) (float64, bool)

func (*Node) Get 

func (self *Node) Get(key string) *Node

func (*Node) Integer 

func (self *Node) Integer(allowNil bool) (int64, bool)

func (*Node) List 

func (self *Node) List(allowNil bool) (List, bool)

func (*Node) Map 

func (self *Node) Map(allowNil bool) (Map, bool)

func (*Node) Put 

func (self *Node) Put(key string, value interface{}) bool

func (*Node) String 

func (self *Node) String(allowNil bool) (string, bool)

func (*Node) StringList added in v0.1.30 

func (self *Node) StringList(allowNil bool) ([]string, bool)

func (*Node) StringMap 

func (self *Node) StringMap(allowNil bool) (StringMap, bool)

func (*Node) UnsignedInteger 

func (self *Node) UnsignedInteger(allowNil bool) (uint64, bool)

type Path 

type Path []PathElement

func (Path) Append 

func (self Path) Append(element PathElement) Path

func (Path) AppendField 

func (self Path) AppendField(name string) Path

func (Path) AppendList 

func (self Path) AppendList(index int) Path

func (Path) AppendMap 

func (self Path) AppendMap(name string) Path

func (Path) AppendSequencedList 

func (self Path) AppendSequencedList(index int) Path

func (Path) String 

func (self Path) String() string

fmt.Stringer interface

type PathElement 

type PathElement struct {
	Type  PathElementType
	Value interface{} // string for FieldPathType and MapPathType, int for ListPathType and SequencedListPathType
}

func NewFieldPathElement 

func NewFieldPathElement(name string) PathElement

func NewListPathElement 

func NewListPathElement(index int) PathElement

func NewMapPathElement 

func NewMapPathElement(name string) PathElement

func NewSequencedListPathElement 

func NewSequencedListPathElement(index int) PathElement

type PathElementType 

type PathElementType uint8

type StringMap 

type StringMap = map[string]Value

func EnsureStringMaps 

func EnsureStringMaps(stringMap StringMap) StringMap

Ensure data adheres to map[string]interface{} (JSON encoding does not support map[interface{}]interface{})

func MapToStringMap added in v0.1.20 

func MapToStringMap(map_ Map) StringMap

Recursive

type TypeName 

type TypeName string
const (
	NoType TypeName = ""

	// Failsafe schema: https://yaml.org/spec/1.2/spec.html#id2802346
	TypeMap    TypeName = "ard.map"
	TypeList   TypeName = "ard.list"
	TypeString TypeName = "ard.string"

	// JSON schema: https://yaml.org/spec/1.2/spec.html#id2803231
	TypeBoolean TypeName = "ard.boolean"
	TypeInteger TypeName = "ard.integer"
	TypeFloat   TypeName = "ard.float"

	// Other schemas: https://yaml.org/spec/1.2/spec.html#id2805770
	TypeNull      TypeName = "ard.null"
	TypeTimestamp TypeName = "ard.timestamp"
)

func GetTypeName 

func GetTypeName(value Value) TypeName

type TypeValidator 

type TypeValidator = func(Value) bool

type Value 

type Value = interface{}

func Canonicalize added in v0.1.21 

func Canonicalize(value Value) (Value, error)

func Copy 

func Copy(value Value) Value

func DecodeYAMLTemplate added in v0.1.19 

func DecodeYAMLTemplate(code string, data interface{}) (Value, error)

func EnsureCompatibleJSON added in v0.1.21 

func EnsureCompatibleJSON(value Value) Value

func FromCompatibleJSON added in v0.1.15 

func FromCompatibleJSON(value Value) (Value, bool)

func MapsToStringMaps added in v0.1.20 

func MapsToStringMaps(value Value) (Value, bool)

func Normalize added in v0.1.20 

func Normalize(value Value) (Value, bool)

Ensure data adheres to the ARD map type (JSON decoding uses map[string]interface{} instead of map[interface{}]interface{})

func RoundtripCBOR added in v0.1.21 

func RoundtripCBOR(value Value) (Value, error)

func RoundtripCompatibleJSON added in v0.1.21 

func RoundtripCompatibleJSON(value Value) (Value, error)

func RoundtripYAML added in v0.1.21 

func RoundtripYAML(value Value) (Value, error)

func ToCompatibleJSON added in v0.1.15 

func ToCompatibleJSON(value Value) (Value, bool)

type YAMLLocator 

type YAMLLocator struct {
	RootNode *yaml.Node
}

func NewYAMLLocator 

func NewYAMLLocator(rootNode *yaml.Node) *YAMLLocator

func (*YAMLLocator) Locate 

func (self *YAMLLocator) Locate(path ...PathElement) (int, int, bool)

Locator interface

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.