unstructured

README

Unstructured - for when you can't fit your data into a struct

Why?

Go is awesome at [de-]serialising structured data. We can do things like:

type MyType struct {
  Key1 string `yaml:"key1"`
  Key2 int    `yaml:"key2"`
}

...

myYaml = `
key1: "Alice"
key2: 65537
`

var myVar MyType
err := yaml.Unmarshal([]byte(myYaml), &myVar)

This is fantastic so long as:

1. You know the exact structure of your data at compile time.
2. That structure can be mapped to a go struct type.

For an example of point (2) failing, consider the following yaml:

top-level-list:
- "this first element is a string - perhaps containing metadata"
- name: first real element
  type: element-type-1
  payload: [1,2,3,4]
- name: second real element
  type: element-type-2
  payload: 
    some: embedded structure

I do not know of a way to define a struct type in go which will accept this YAML when I attempt to deserialize it. If we're designing our systems end-to-end in go, we will almost certainly avoid producing YAML structures like this. However, sometimes we may have to interface with other systems written perhaps in more dynamically typed languages, where this sort of thing is more natural.

For an example of point (1) failing, consider the schema of a bosh manifest. The type of the properties block of an instance group is described as a "hash". In practice, these hashes can be quite complex structures, depending on how configurable the jobs in that particular instance group are. Unfortunately, the schema for each individual properties block is defined by the authors of the jobs in the instance group. If we are writing a tool in go that manages bosh manifests in general, then we have no way to be more specific than the type map[string]interface{}.

What?

When working with data which cannot be described at compile time in the go type language, we have no choice but to either leave go and work in some other language, or to work without the safety net of our type system. This library attempts to make managing unstructured data in an untyped way less unpleasant than it might otherwise be.

How?

This library leans on the excellent gojsonpointer library to allow us to address deep into JSON and YAML structures and:

• retrieve data -- if it exists
• write data -- if the parent we're writing into exists

This allows us to handle the data above with code something like this:

	myData, err := unstructured.ParseYAML(myYaml)
	myPayloadData, err := myData.GetByPointer("/top-level-list/2/payload/some")
	myPayloadString, err := myPayloadData.StringValue()
	fmt.Println(myPayloadString)
	myPayloadMap, err := myData.GetByPointer("/top-level-list/2/payload")
	err := myPayloadMap.SetField("additional-key", []string{"some", "arbitrary", "data"})

Of course, in the real version, all the error values are checked.

We also provide a number of gomega matchers in case you want to inspect semi-structured data in your tests. You can see these used here.

Gotchas

Since we're deliberately working around go's type system, we have to perform a lot of type-like checks ourselves, at runtime. For most of the methods in the library, this means we might return an error, which you should check. However, sometimes it's useful to be able to chain our methods, and this is only possible if those methods return singletons. To facilitate this, we have some "Unsafe" methods, which may panic if their preconditions are not met. For example, d.F("parent-field").F("sub-field") will panic unless d is a map with key "parent-field" which contains a map with key "sub-field". You can check these things with methods like IsOb and HasKey, but you should probably just use GetByPointer instead. The unsafe accessors are only really useful for writing terse chained statements in tests, when you're almost entirely certain that the path exists, and a panic would be a correctly failing test anyway.

You can see examples of the usage of these unsafe accessors in the "alternative formulations" of the tests in this example.

Documentation

Overview

Package unstructured provides ways of manipulating unstructured data such as JSON or YAML

Index

• Constants
• type Data
  • func ParseJSON(rawjson string) (Data, error)
  • func ParseYAML(rawjson string) (Data, error)
  • func (j Data) BoolValue() (bool, error)
  • func (j Data) F(key string) Data
  • func (j Data) FindElem(match ElementMatcher) (Data, bool)
  • func (j Data) GetByPointer(p string) (data Data, err error)
  • func (j Data) HasKey(key string) bool
  • func (j Data) HasPointer(p string) (bool, error)
  • func (j Data) IsBool() bool
  • func (j Data) IsList() bool
  • func (j Data) IsNull() bool
  • func (j Data) IsNum() bool
  • func (j Data) IsOb() bool
  • func (j Data) IsOfType(typ string) bool
  • func (j Data) IsString() bool
  • func (j Data) Keys() ([]string, error)
  • func (j Data) ListValue() ([]Data, error)
  • func (j Data) NumValue() (float64, error)
  • func (j Data) ObValue() (map[string]interface{}, error)
  • func (j Data) RawValue() interface{}
  • func (j Data) SetElem(index int, value interface{}) error
  • func (j Data) SetField(fieldName string, val interface{}) error
  • func (j Data) StringValue() (string, error)
  • func (j Data) UnsafeBoolValue() bool
  • func (j Data) UnsafeGetField(key string) Data
  • func (j Data) UnsafeListValue() (list []Data)
  • func (j Data) UnsafeNumValue() float64
  • func (j Data) UnsafeObValue() map[string]interface{}
  • func (j Data) UnsafeStringValue() string
• type ElementMatcher

Constants

const (
	DataString = "string"
	DataNum    = "number"
	DataOb     = "object"
	DataList   = "list"
	DataNull   = "null"
	DataBool   = "bool"
)

Types

type Data

type Data struct {
	// contains filtered or unexported fields
}

Data represents some unstructured data

func ParseJSON

func ParseJSON(rawjson string) (Data, error)

ParseJSON unmarshals json from an input string. Use this for generating a Data struct, whose contents you can examine using the following functions.

func ParseYAML

func ParseYAML(rawjson string) (Data, error)

ParseYAML unmarshals yaml from an input string. Use this for generating a Data struct, whose contents you can examine using the following functions.

func (Data) BoolValue

func (j Data) BoolValue() (bool, error)

BoolValue returns the golang bool representation of the bool represented by this Data struct. If the Data struct does not represent a bool, this method returns an error.

func (Data) F

func (j Data) F(key string) Data

F is a shorthand for `UnsafeGetField`

func (Data) FindElem

func (j Data) FindElem(match ElementMatcher) (Data, bool)

FindElem finds an element in a list, using a provided matcher

func (Data) GetByPointer

func (j Data) GetByPointer(p string) (data Data, err error)

GetByPointer returns a Data struct containing the contents of the original data at the given pointer address `p`. For more information on json pointers, see https://tools.ietf.org/html/rfc6901

func (Data) HasKey

func (j Data) HasKey(key string) bool

HasKey returns true iff the object represented by this Data struct contains `key`

Note: this will panic if the data represented by this Data struct is not an object. If in doubt, check with `IsOb()`

func (Data) HasPointer

func (j Data) HasPointer(p string) (bool, error)

HasPointer returns true iff the object represented by this Data struct contains the pointer `p`

For more information on json pointers, see https://tools.ietf.org/html/rfc6901

func (Data) IsBool

func (j Data) IsBool() bool

IsBool returns true iff the data represented by this Data struct is a boolean.

func (Data) IsList

func (j Data) IsList() bool

IsList returns true iff the data represented by this Data struct is a list.

func (Data) IsNull

func (j Data) IsNull() bool

IsNull returns true iff the data represented by this Data struct is null.

func (Data) IsNum

func (j Data) IsNum() bool

IsNum returns true iff the data represented by this Data struct is a number.

func (Data) IsOb

func (j Data) IsOb() bool

IsOb returns true iff the data represented by this Data struct is an object or map.

func (Data) IsOfType

func (j Data) IsOfType(typ string) bool

IsOfType returns true iff the Data struct represents data of type `typ`. Valid values of `typ` are listed as constants above.

func (Data) IsString

func (j Data) IsString() bool

IsString returns true iff the data represented by this Data struct is a string.

func (Data) Keys

func (j Data) Keys() ([]string, error)

Keys returns a list of the keys on this Data object.

If this is not a Data object, return an error

func (Data) ListValue

func (j Data) ListValue() ([]Data, error)

ListValue returns a golang slice of Data structs representing the unstructured list represented by this Data struct. If the Data struct does not represent a list, this method returns an error.

func (Data) NumValue

func (j Data) NumValue() (float64, error)

NumValue returns the golang float64 representation of the number represented by this Data struct. If the Data struct does not represent a number, this method returns an error.

func (Data) ObValue

func (j Data) ObValue() (map[string]interface{}, error)

ObValue returns a golang map[string]interface{} represenation of the object represented by this Data struct. If the Data struct does not represent an object, this method returns an error

func (Data) RawValue

func (j Data) RawValue() interface{}

RawValue returns the raw go value of the parsed data, without any type checking

func (Data) SetElem

func (j Data) SetElem(index int, value interface{}) error

SetElem sets the element at a given index in this Data list to the given value. If this Data object does not represent a list, return an error

func (Data) SetField

func (j Data) SetField(fieldName string, val interface{}) error

SetField updates the field `fieldName` of this Data object. If the field `fieldName` does not exist on this object, create it.

If this Data does not represent an object, return an error.

func (Data) StringValue

func (j Data) StringValue() (string, error)

StringValue returns the golang string representation of the string represented by this Data struct. If the Data struct does not represent a string, this method returns an error.

func (Data) UnsafeBoolValue

func (j Data) UnsafeBoolValue() bool

UnsafeBoolValue returns the golang bool representation of the bool represented by this Data struct. If the Data struct does not represent a bool, this method panics. If in doubt, check with `IsBool()`

func (Data) UnsafeGetField

func (j Data) UnsafeGetField(key string) Data

UnsafeGetField returns a Data struct containing the contents of the original data at the given `key`. If this method name feels too long, use `F(key)`.

Note: this function panics if the given `key` does not exist. If in doubt, check with `HasKey()`.

func (Data) UnsafeListValue

func (j Data) UnsafeListValue() (list []Data)

UnsafeListValue returns a golang slice of Data structs representing the unstructured list represented by this Data struct. If the Data struct does not represent a list, this method panics. If in doubt, check with `IsList()`

func (Data) UnsafeNumValue

func (j Data) UnsafeNumValue() float64

UnsafeNumValue returns the golang float64 representation of the number represented by this Data struct. If the Data struct does not represent a number, this method panics. If in doubt, check with `IsNum()`

func (Data) UnsafeObValue

func (j Data) UnsafeObValue() map[string]interface{}

UnsafeObValue returns a golang map[string]interface{} represenation of the object represented by this Data struct. If the Data struct does not represent an object, this method panics. If in doubt, check with `IsOb()`

func (Data) UnsafeStringValue

func (j Data) UnsafeStringValue() string

UnsafeStringValue returns the golang string representation of the string represented by this Data struct. If the Data struct does not represent a string, this method panics. If in doubt, check with `IsString()`

type ElementMatcher

type ElementMatcher func(Data) bool

An ElementMatcher can be used with FindElem to find an element in an unstructured list.