conflate

README

CONFLATE

Library providing routines to merge and validate JSON, YAML, TOML files and/or structs (godoc)

Typical use case: Make your application configuration files multi-format, modular, templated, sparse, location-independent and validated

Description

Conflate is a library and cli-tool, that provides the following features :

• merge data from multiple formats (JSON/YAML/TOML/go structs) and multiple locations (filesystem paths and urls)
• validate the merged data against a JSON schema
• apply any default values defined in a JSON schema to the merged data
• expand environment variables inside the data
• marshal merged data to multiple formats (JSON/YAML/TOML/go structs)

It supports draft-04, draft-06 and draft-07 of JSON Schema. If the key $schema is missing, or the draft version is not explicitly set, a hybrid mode is used which merges together functionality of all drafts into one mode. Improvements, ideas and bug fixes are welcomed.

Getting started

Run the following command, which will build and install the latest binary in $GOPATH/bin

go get github.com/zeitlinger/conflate/...

Alternatively, you can install one of the pre-built release binaries from https://github.com/miracl/conflate/releases

Usage of Library

Please refer to the godoc and the example code

Usage of CLI Tool

Help can be obtained in the usual way :

$conflate --help
Usage of conflate:
  -data value
    	The path/url of JSON/YAML/TOML data, or 'stdin' to read from standard input
  -defaults
    	Apply defaults from schema to data
  -expand
    	Expand environment variables in files
  -format string
    	Output format of the data JSON/YAML/TOML
  -includes string
    	Name of includes array. Blank string suppresses expansion of includes arrays (default "includes")
  -noincludes
    	Switches off conflation of includes. Overrides any --includes setting.
  -schema string
    	The path/url of a JSON v4 schema file
  -validate
    	Validate the data against the schema
  -version
    	Display the version number

To conflate the following file ... :

$cat ./testdata/valid_parent.json
{
  "includes": [
    "valid_child.json",
    "valid_sibling.json"
  ],
  "parent_only" : "parent",
  "parent_child" : "parent",
  "parent_sibling" : "parent",
  "all": "parent"
}

...run the following command, which will merge valid_parent.json, valid_child.json, valid_sibling.json :

$conflate -data ./testdata/valid_parent.json -format JSON
{
  "all": "parent",
  "child_only": "child",
  "parent_child": "parent",
  "parent_only": "parent",
  "parent_sibling": "parent",
  "sibling_child": "sibling",
  "sibling_only": "sibling"
}

Note how the includes are loaded remotely as relative paths.

Also, note values in a file override values in any included files, and that an included file overrides values in any included file above it in the includes list.

If you instead host a file somewhere else, then just use a URL :

$conflate -data https://raw.githubusercontent.com/zeitlinger/conflate/master/testdata/valid_parent.json -format JSON
{
  "all": "parent",
  "child_only": "child",
  "parent_child": "parent",
  "parent_only": "parent",
  "parent_sibling": "parent",
  "sibling_child": "sibling",
  "sibling_only": "sibling"
}

The includes here are also loaded as relative urls and follow exactly the same merging rules.

To output in a different format use the -format option, e.g. TOML :

$conflate -data ./testdata/valid_parent.json -format TOML
all = "parent"
child_only = "child"
parent_child = "parent"
parent_only = "parent"
parent_sibling = "parent"
sibling_child = "sibling"
sibling_only = "sibling"

To additionally use defaults from a JSON schema and validate the conflated data against the schema, use -defaults and -validate respectively :

$cat ./testdata/blank.yaml

$conflate -data ./testdata/blank.yaml -schema ./testdata/test.schema.json -validate -format YAML
Schema validation failed : The document is not valid against the schema : Invalid type. Expected: object, given: null (#)

$conflate -data ./testdata/blank.yaml -schema ./testdata/test.schema.json -defaults -validate -format YAML
all: parent
child_only: child
parent_child: parent
parent_only: parent
parent_sibling: parent
sibling_child: sibling
sibling_only: sibling

Note any defaults are applied before validation is performed, as you would expect.

If you don't want to intrusively embed an "includes" array inside your JSON, you can instead provide multiple data files which are processed from left-to-right :

$conflate -data ./testdata/valid_child.json -data ./testdata/valid_sibling.json -format JSON
{
  "all": "sibling",
  "child_only": "child",
  "parent_child": "child",
  "parent_sibling": "sibling",
  "sibling_child": "sibling",
  "sibling_only": "sibling"
}

Or alternatively, you can create a top-level JSON file containing only the includes array. For fun, lets choose to use YAML for the top-level file, and output TOML :

$cat toplevel.yaml
includes:
  - testdata/valid_child.json
  - testdata/valid_sibling.json

$conflate -data toplevel.yaml -format TOML
all = "sibling"
child_only = "child"
parent_child = "child"
parent_sibling = "sibling"
sibling_child = "sibling"
sibling_only = "sibling"

If you want to read a file from stdin you can do the following. Here we pipe in some TOML to override a value to demonstrate :

$echo 'all="MY OVERRIDDEN VALUE"' |  conflate -data ./testdata/valid_parent.json -data stdin  -format JSON
{
  "all": "MY OVERRIDDEN VALUE",
  "child_only": "child",
  "parent_child": "parent",
  "parent_only": "parent",
  "parent_sibling": "parent",
  "sibling_child": "sibling",
  "sibling_only": "sibling"
}

Note that in all cases -data sources are processed from left-to-right, with values in right files overriding values in left files, so the following doesn't work :

$echo 'all="MY OVERRIDDEN VALUE"' |  conflate -data stdin -data ./testdata/valid_parent.json  -format JSON
{
  "all": "parent",
  "child_only": "child",
  "parent_child": "parent",
  "parent_only": "parent",
  "parent_sibling": "parent",
  "sibling_child": "sibling",
  "sibling_only": "sibling"
}

You can optionally expand environment variables in the files like this :

$export echo MYVALUE="some value"
$export echo MYJSONMAP='{ "item1" : "value1" }'
$echo '{ "my_value": "$MYVALUE", "my_map": $MYJSONMAP }' | conflate -data stdin -expand -format JSON
{
  "my_map": {
    "item1": "value1"
  },
  "my_value": "some value"
}

Acknowledgements

Images derived from originals by Renee French https://golang.org/doc/gopher/

Documentation

Overview

Package conflate is a library that helps to merge and validate data from multiple formats (JSON/YAML/TOML), and multiple locations (filesystem paths and urls).

Index

• Variables
• func JSONUnmarshal(data []byte, out interface{}) error
• func TOMLUnmarshal(data []byte, out interface{}) error
• func YAMLUnmarshal(data []byte, out interface{}) error
• type Conflate
  • func FromData(data ...[]byte) (*Conflate, error)
  • func FromFiles(paths ...string) (*Conflate, error)
  • func FromGo(data ...interface{}) (*Conflate, error)
  • func FromURLs(urls ...*url.URL) (*Conflate, error)
  • func New() *Conflate
  • func (c *Conflate) AddData(data ...[]byte) error
  • func (c *Conflate) AddFiles(paths ...string) error
  • func (c *Conflate) AddGo(objs ...interface{}) error
  • func (c *Conflate) AddURLs(urls ...*url.URL) error
  • func (c *Conflate) ApplyDefaults(s *Schema) error
  • func (c *Conflate) Expand(expand bool)
  • func (c *Conflate) MarshalJSON() ([]byte, error)
  • func (c *Conflate) MarshalTOML() ([]byte, error)
  • func (c *Conflate) MarshalYAML() ([]byte, error)
  • func (c *Conflate) Unmarshal(out interface{}) error
  • func (c *Conflate) Validate(s *Schema) error
• type Schema
  • func NewSchemaData(data []byte) (*Schema, error)
  • func NewSchemaFile(path string) (*Schema, error)
  • func NewSchemaGo(s interface{}) (*Schema, error)
  • func NewSchemaURL(u *url.URL) (*Schema, error)
  • func (s *Schema) ApplyDefaults(pData interface{}) error
  • func (s *Schema) Validate(data interface{}) error
• type UnmarshallerFunc
• type UnmarshallerFuncs
• type UnmarshallerMap

Variables

var Includes = "includes"

Includes is used to specify the top level key that holds the includes array.

var Unmarshallers = UnmarshallerMap{
	".json": {JSONUnmarshal},
	".jsn":  {JSONUnmarshal},
	".yaml": {YAMLUnmarshal},
	".yml":  {YAMLUnmarshal},
	".toml": {TOMLUnmarshal},
	".tml":  {TOMLUnmarshal},
	"":      {JSONUnmarshal, YAMLUnmarshal, TOMLUnmarshal},
}

Unmarshallers is a list of unmarshalling functions to be used for given file extensions. The unmarshaller slice for the blank file extension is used when no match is found.

Functions

func JSONUnmarshal

func JSONUnmarshal(data []byte, out interface{}) error

JSONUnmarshal unmarshals the data as JSON.

func TOMLUnmarshal

func TOMLUnmarshal(data []byte, out interface{}) error

TOMLUnmarshal unmarshals the data as TOML.

func YAMLUnmarshal

func YAMLUnmarshal(data []byte, out interface{}) error

YAMLUnmarshal unmarshals the data as YAML.

Types

type Conflate

type Conflate struct {
	// contains filtered or unexported fields
}

Conflate contains a 'working' merged data set and optionally a JSON v4 schema.

func FromData

func FromData(data ...[]byte) (*Conflate, error)

FromData constructs a new Conflate instance populated with the given data.

func FromFiles

func FromFiles(paths ...string) (*Conflate, error)

FromFiles constructs a new Conflate instance populated with the data from the given files.

func FromGo

func FromGo(data ...interface{}) (*Conflate, error)

FromGo constructs a new Conflate instance populated with the given golang objects.

func FromURLs

func FromURLs(urls ...*url.URL) (*Conflate, error)

FromURLs constructs a new Conflate instance populated with the data from the given URLs.

func New

func New() *Conflate

New constructs a new empty Conflate instance.

func (*Conflate) AddData

func (c *Conflate) AddData(data ...[]byte) error

AddData recursively merges the given data into the Conflate instance.

func (*Conflate) AddFiles

func (c *Conflate) AddFiles(paths ...string) error

AddFiles recursively merges the data from the given files into the Conflate instance.

func (*Conflate) AddGo

func (c *Conflate) AddGo(objs ...interface{}) error

AddGo recursively merges the given (json-serializable) golang objects into the Conflate instance.

func (*Conflate) AddURLs

func (c *Conflate) AddURLs(urls ...*url.URL) error

AddURLs recursively merges the data from the given urls into the Conflate instance.

func (*Conflate) ApplyDefaults

func (c *Conflate) ApplyDefaults(s *Schema) error

ApplyDefaults sets any nil or missing values in the data, to the default values defined in the JSON v4 schema.

func (*Conflate) Expand

func (c *Conflate) Expand(expand bool)

Expand is an option to automatically expand environment variables in data files.

func (*Conflate) MarshalJSON

func (c *Conflate) MarshalJSON() ([]byte, error)

MarshalJSON exports the data as JSON.

func (*Conflate) MarshalTOML

func (c *Conflate) MarshalTOML() ([]byte, error)

MarshalTOML exports the data as TOML.

func (*Conflate) MarshalYAML

func (c *Conflate) MarshalYAML() ([]byte, error)

MarshalYAML exports the data as YAML.

func (*Conflate) Unmarshal

func (c *Conflate) Unmarshal(out interface{}) error

Unmarshal extracts the data as a Golang object.

func (*Conflate) Validate

func (c *Conflate) Validate(s *Schema) error

Validate checks the data against the JSON v4 schema.

type Schema

type Schema struct {
	// contains filtered or unexported fields
}

Schema contains a JSON v4 schema.

func NewSchemaData

func NewSchemaData(data []byte) (*Schema, error)

NewSchemaData loads a JSON v4 schema from the given data.

func NewSchemaFile

func NewSchemaFile(path string) (*Schema, error)

NewSchemaFile loads a JSON v4 schema from the given path.

func NewSchemaGo

func NewSchemaGo(s interface{}) (*Schema, error)

NewSchemaGo creates a Schema instance from a schema represented as a golang object.

func NewSchemaURL

func NewSchemaURL(u *url.URL) (*Schema, error)

NewSchemaURL loads a JSON v4 schema from the given URL.

func (*Schema) ApplyDefaults

func (s *Schema) ApplyDefaults(pData interface{}) error

ApplyDefaults adds default values defined in the schema to the data pointed to by pData.

func (*Schema) Validate

func (s *Schema) Validate(data interface{}) error

Validate checks the given golang data against the schema.

type UnmarshallerFunc

type UnmarshallerFunc func([]byte, interface{}) error

UnmarshallerFunc defines the type of function used for unmarshalling data.

type UnmarshallerFuncs

type UnmarshallerFuncs []UnmarshallerFunc

UnmarshallerFuncs defines the type for a slice of UnmarshallerFunc.

type UnmarshallerMap

type UnmarshallerMap map[string]UnmarshallerFuncs

UnmarshallerMap defines the type of a map of string to UnmarshallerFuncs.