neatly

README

Neatly - a neat format for representing nested structured data.

This library is compatible with Go 1.8+

Please refer to CHANGELOG.md if you encounter breaking changes.

• Motivation
• Getting started
• Neatly
• User Defined Function
• Usage
• License
• Credits and Acknowledgements

Motivation

Most of the data-driven application use extensively nested structured data to instruct how the application should behave. There is various data format that comes handy like vanilla CSV, XML, JSON, YAML which are more or less human-friendly. Once the data grows, organizing it with these formats becomes the mundane task. As a matter of fact, in the absence of UI or/and data store (either RDBMS or NoSQL) data organization is challenging. Neatly has been designing in mind to address all these concerns promoting flexibility, reusability, data cohesion, delegation and data organization

Neatly

Neatly is a neat format for representing nested structured data, with a simple tabular approach.

Neatly document uses tabular format thus can be easily stored in CSV or other delimited formats,

The first column in a row represents an object tag, followed by columns defining the object fields. Next row/s would define object tag values, in this case, the first column would be left empty. You can think of an object tag as it is an object definition, whereas object values would create one or more object instances. An object tag can represent a single instance or an instance element within an array. In the latter case, object tag would be created with [] prefix.

The very first object tag in the document represents a root object, it can be mapped to either to user-defined struct type or a simple map.

The field defines a path to the leaf object of an object tag.

Field name should start with an uppercase letter to map to the underlying data structure.

It can be prefixed with

1. square bracket '[]' to denote that field is an array, all rows below will be elements for the array, unless there is empty line
2. slash '/' to denote that field belongs to root object rather then preceding tag object
3. colon ':' or expression stating with lowercase character denotes a virtual object field that can be used as data substitution source referenced with dollar($) sign expression.

On top of that field definition can use dot (.) to define nested object of object or array type. For instance

1. field1 defines 'field1' field.
2. []field2.attr2 defines 'field2' filed of an array of an object type, which has 'attr2' field.
3. field3.[]a.attr3 defines 'field3' field of an object type, which has another 'a' field of an array of object type with 'attr3' field.
4. /data.[]tables defines in the root object 'data' field of an object type which has another 'tables' field of an array type.
5. :v.attr5 defines a virtual object which has 'v' field of an object type which has another 'attr5' field .

This keyword used in field expression expands corresponding key/value directly to the current object tag.

Neatly capabilities

Use cases

Basic data structure with repeated fields.

Take as example the following data structure:

{
  "UseCase": "case 1",
  "Requests": [
    {
      "URL": "http://127.0.0.1/test1",
      "Method": "GET",
      "Cookies": {
        "Cookie1": "value1",
        "Cookie2": "value2"
      }
    },
    {
      "URL": "http://127.0.0.1/test2",
      "Method": "GET",
      "Cookies": {
        "Cookie1": "value1",
        "Cookie2": "value2"
      }
    }
  ],
  "Expect": [
    {
      "StatusCode": 200,
      "Body": "test1Content"
    },
    {
      "StatusCode": 404
    }
  ]
}
 

Neatly tabular representation.

Root	UseCase	[]Requests.Method	[]Requests.URL	[]Requests.Cookies	[]Expect.StatusCode	[]Expect.Body	Comments
	case 1	GET	http://127.0.0.1/test1	{"Cookie1":"value1", "Cookie1":"value2"}	200	testContent	{!123!}
		GET	http://127.0.0.1/test2	{"Cookie1":"value1", "Cookie1":"value2"}	404

In this case we have only one object tag called root which has simple and repeated fields.
Note that {} or [] prefix, sufix in an object value converts value to an object or array respectively. You can escape { ... } with {! ... !} quote or [ ... ] with [! ... !] to represent value as text instead.

One to many with forward object tag reference use case

Take as example the following data structure:

{
"CreateTime": "2017-10-23 10:00",
"Orders": [
 {
   "Id": 1,
   "Name": "Order 1",
   "SubTotal": 100.0,
   "LineItems": [
     {
       "Product": "Magic Mouse",
       "Quantity": 5,
       "Price": 10.0
     },
     {
       "Position": 1,
       "Product": "TrackPad",
       "Quantity": 5,
       "Price": 10.0
     }
   ]
 },
 {
   "Id": 2,
   "Name": "Order 2",
   "SubTotal": 150.0,
   "LineItems": [
     {
       "Product": "Keyboard",
       "Quantity": 10,
       "Price": 10.0
     },
     {
       "Product": "TrackPad",
       "Quantity": 5,
       "Price": 10.0
     }
   ]
 }
]
}

Neatly tabular representation.

Root	CreateTime	Orders
	2017-10-23 10:00	%Orders
[]Orders	Id	Name	LineItems	SubTotal
	1	Order 1	%LineItems1	100
[]LineItems1	Product	Quantity	Price
	Magic Mouse	5	10.0
	TrackPad	5	10.0
[]Orders	Id	Name	LineItems	SubTotal
	1	Order 2	%LineItems2	150
[]LineItems2	Product	Quantity	Price	Note
	Keyboard	10	10.0	Requested extended keyboard
	TrackPad	5	10.0

In this case we have 5 tags each defining its own objects, Note that percentage (%) prefixed object's value will be substitute with the object matching tag's value. Percentage (%) denotes forward reference, which means that referencing tag definition takes places in the following rows.

You can escape % ... %%.

Root object tag field with data cohesion use case.

{
"Bonus": [
 {
   "EmpNo": 1,
   "Name": "Smith",
   "Amount": 10000
 },
 {
   "EmpNo": 2,
   "Name": "Kowalczyk",
   "Amount": 8000
 },
 {
   "EmpNo": 3,
   "Name": "Schmidt",
   "Amount": 4000
 }
],
"Merits": [
 {
   "EmpNo": 1,
   "Description": "Increase sales by 400%"
 },
 {
   "EmpNo": 2,
   "Description": "Reduced cost by 30%"
 },
 {
   "EmpNo": 3,
   "Description": "Improve resource reusibility by 40%"
 }
]
}

Neatly tabular representation.

SomeName	Merits	Created
	%Merits	2017-10-10
[]Merits	Empno	Description	/[]Bonus
	1	Increased sales by 400%	{"EmpNo":1, "Name":"Smith", "Amount":10000}
	2	Reduced cost by 30%	{"EmpNo":2, "Name":"Kowalczyk", "Amount":8000}
	3	Improved resource reusibility by 40%	{"EmpNo":3, "Name":"Schmidt", "Amount":4000}

In this case "Bonus filed" on Merits tag is actual root object field reference. Cohesion has been achieved by placing data related the same employee in the same row. Note that root object uses slash(/) in the field name

Virtual Objects for data sharing and delegation use case.

The previous example address cohesion somehow, however we can see that empNo is repeated twice in the same row, Op top of that bonus value uses json notation, which may not be too elegant.

Root	Merits	Created
	%Merits	2017-10-10
[]Merits	Empno	Description	/[]Bonus	emp.EmpNo	emp.Name	:emp.Amount
	$emp.EmpNo	Increased sales by 400%	$emp	1	Smith	10000
	$emp.EmpNo	Reduced cost by 30%	$emp	2	Kowalczyk	8000
	$emp.EmpNo	Improved resource reusibility by 40%	$emp	3	Schmidt	4000

In this case the virtual object emp was defined by 3 fields either lowercase or prefixed with colon (:) sing. In order to reference virtual object dollar sign expression is being used.

Loading repeated data with tags template.

Take as example the following data structure:

{
"Repeated": [
 {
   "Id": 1,
   "Name": "Name 01"
 },
 {
   "Id": 2,
   "Name": "Name 02"
 },
 {
   "Id": 3,
   "Name": "Name 03"
 },
 {
   "Id": 4,
   "Name": "Name 04"
 },
 {
   "Id": 5,
   "Name": "Name 05"
 }
]
}

Neatly tabular representation.

Root	Repeated
	%Repeated
[]Repeated{1 .. 05}	Id	Name
	$index	Name $index

In this case Repeted tag uses Tag Range expression { }, which repeats all value rows as (max - min) from the range.

Variable substitution

Variable substitution rules

1. '$' path expression will be only substituted if path is present in the current context
2. UDF wil be substituted only if it returns no error and in case it takes $ expression as parameter, expression path is present in context.

$index / ${index} is a special/reserved keyword in this context it would expand to 01 in the firs iteration followed by 02, 03, 04 and 05. If there are more than one value row, all can be expanded to the same index value within the same iteration.

Note that number of digits in the upper bound range value creates a padding template. 1 .. 010 -> would '0' left pad an index to 3 digits 1 .. 00100 -> would '0' left pad an index to 5 digits.

$tag expands to the current object tag.

$tagId expands to current root object Name if specified, followed by object tag, object tag index, and subpath if specified.

$subPath expands to current subPath if specified for the tag

$path expands to current path if sub path is specified for the tag

$pathMatch expands to resolved wildcard subPath fragment

Data delegation and loading external resources use case

Take as example the following data structure:

{
"Students": [
 {
   "Id": 1,
   "Name": "Smith",
   "Scores": [
     {
       "Subject":"Math",
       "Score": 3.2
     },
     {
       "Subject":"English",
       "Score":3.5
     }
   ]
 },
 {
   "Id": 2,
   "Name": "Kowalczyk",
   "Scores": [
     {
       "Subject":"Math",
       "Score": 3.7
     },
     {
       "Subject":"English",
       "Score": 3.2
     }
   ]
 }
]
}

Neatly tabular representation.

Root	Students
	%Students
[]Students	Id	Name	Scores
	1	Smith	@scores1.json
	2	Kowalczyk	@scores2.json

In this case scores are loaded from local json file.

External resource starts with (@) or # sing, resource URL can be relative, absolute path or a valid URL to any content. In case of json or yaml files, the content is treated as data structure. To escape '@' use '@@', or '#' use '##' .

Where @scores1.json

  [
        {
           "Subject":"Math",
           "Score": 3.2
        },
        {
          "Subject":"English",
           "Score":3.5
        }
  ]

@scores2.json

  [
        {
           "Subject":"Math",
           "Score": 3.7
        },
        {
           "Subject":"English",
           "Score":3.2
        }
  ]

Data delegation and loading external resources with subpath use case.

Take as example the following data structure:

{
"Setup": {
 "MyDb": {
   "Customer": [
     {
       "ID": 1,
       "NAME": "Smith",
       "DAILY_CAP": 100,
       "OVERALL_CAP": 1000
     },
     {
       "ID": 2,
       "NAME": "Kowalczyk",
       "DAILY_CAP": 400,
       "OVERALL_CAP": 8000
     }
   ]
 }
},
"UseCases": [
 {
   "Id": "1",
   "Description": "use case 1"
 },
 {
   "Id": "2",
   "Description": "use case 2"
 }
]
}

Neatly tabular representation.

Root	UseCases
	%UseCases
[]UseCases{1..002}	Subpath	Id	Description	/Setup.MyDb.[]Customer
	usecase7/${index} /	$index	@use_case.txt	@customer.json

Where

@usecase7/001/use_case.txt

use case 1

@usecase7/001/customer.json

{
  "ID": 1,
  "NAME": "Smith",
  "DAILY_CAP": 100,
  "OVERALL_CAP": 1000
}

@usecase7/002/use_case.txt

use case 2

@usecase7/002/customer.json

{
  "ID": 2,
  "NAME": "Kowalczyk",
  "DAILY_CAP": 400,
  "OVERALL_CAP": 8000
}

Note that external lookup first check if resource exists in subpath if not it will check if it exists in the same directory as a master file (source of the tabular data).

<a name="external_resource_substitution>

External resources loading with content substitution and user defined function (udf) use case.

Take as example the following data structure:

{
"Setup": {
 "MyDb": {
   "Customer": [
     {
       "ID": 1,
       "NAME": "Smith",
       "DAILY_CAP": 100,
       "OVERALL_CAP": 1000
     },
     {
       "ID": 2,
       "NAME": "Kowalczyk",
       "DAILY_CAP": 100,
       "OVERALL_CAP": 1000
     }
   ]
 }
},
"UseCases": [
 {
   "Id": "1",
   "Description": "use case 1"
 },
 {
   "Id": "2",
   "Description": "use case 2"
 }
]
}

Neatly tabular representation.

Root	UseCases
	%UseCases
[]UseCases{1..002}	Subpath	Id	Description	/Setup.MyDb.[]Customer
	usecase8/${index} /	$index	@use_case.txt	@customer.json| {"dailyCap":100, "overallCap":1000}

Where

@usecase8/001/use_case.txt

use case 1

@usecase8/001/customer.json

{
   "ID": 1,
   "NAME": "Smith",
   "DAILY_CAP": "$AsInt($dailyCap)",
   "OVERALL_CAP": "$AsInt($overallCap)"
 
}

@usecase8/002/use_case.txt

use case 2

@usecase8/002/customer.json

{
  "ID": 2,
  "NAME": "Kowalczyk",
   "DAILY_CAP": "$AsInt($dailyCap)",
   "OVERALL_CAP": "$AsInt($overallCap)"
}

In this case content of the @customer.json were substituted with dailyCap and overallCap values. Since customer and cap data represents valid data structure, dailyCap and overallCap are substituted.

The following construct in a cell

@customer.json | {"dailyCap":100, "overallCap":1000}

can be replaced with two or more external JSON or YAML resources (with space or pipe) providing desired data structure.

@customer.json @cap.json

where @cap.json

 {"dailyCap":100, "overallCap":1000}

Note that xxx_CAP values were expanded within double quote thus they became of text type. In order to convert then to int data type, AsInt user defined function is being used.

External resources arguments inlining

In case where the whole content of subsequent arguments needs to be inlined, you can use $argsX reference.

Neatly tabular alternative representation.

Root	UseCases
	%UseCases
[]UseCases{1..2}	Subpath	Id	Description	/Setup.MyDb.[]Customer
	usecase9/${index} /	$index	@use_case.txt	@customer.json| {"DAILY_CAP":100, "OVERALL_CAP":1000}

Where

@usecase9/001/customer.json

{
   "ID": 1,
   "NAME": "Smith",
   $args0
}

@usecase9/002/customer.json

{
  "ID": 2,
  "NAME": "Kowalczyk",
   $args0
}

The following special variables are available for substitution:

1. $args{index} - piping content stripped from first and last characters.
2. $arg{index} - full piping content.

Where index corresponds to piping number starting with 0

Dynamic subpath discovery

Subpath can use '*' as suffix to dynamically discover actual supath directory

Take as example the following

Neatly tabular alternative representation.

Root	UseCases
	%UseCases
[]UseCases{1..2}	Subpath	Id	Description	/Setup.MyDb.[]Customer
	usecase10/${index}* /	$index	@use_case.txt	@customer.json| {"DAILY_CAP":100, "OVERALL_CAP":1000}

Where

@usecase10/001_description1/customer.json

{
   "ID": 1,
   "NAME": "Smith",
   $args0
}

@usecase10/002_description2/customer.json

{
  "ID": 2,
  "NAME": "Kowalczyk",
   $args0
}

User defined functions (udf)

The user defined system allowed to dynamically convert value from one form to another. To invoke udf value of data structure has to start with $FUNC_NAME(ARGS) for instance $AsFloat("123"), $AsFloat($key1)

In order to define udf please use the following function signature:

    type Udf func(interface{}, Map) (interface{}, error)

Build-in udf'

• HasResource returns true if external resource exists
• Md5 generates md5 for provided parameter
• WorkingDirectory returns working directory joined with supplied sub path, '../' is supported.
• LoadNeatly loads neatly document as data structure.
• Zip, takes []byte or string to compress it.
• Unzip, takes []byte to uncompress it into []byte.
• UnzipText takes []byte to uncompress it into string.
• Markdown generate HTML for suppied markdown
• Cat returns content of supplied filename

UDF Defined in toolbox/data/udf

External resources loading with virtual object value substitution use case.

Take as example the following data structure:

{
"Setup": {
 "MyDb": {
   "Customer": [
     {
       "ID": 1,
       "NAME": "Smith",
       "DAILY_CAP": "200",
       "OVERALL_CAP": "3000"
     },
     {
       "ID": 2,
       "NAME": "Kowalczyk",
       "DAILY_CAP": "100",
       "OVERALL_CAP": "1000"
     }
   ]
 }
},
"UseCases": [
 {
   "Id": "1",
   "Description": "use case 1"
 },
 {
   "Id": "2",
   "Description": "use case 2"
 }
]
}

Neatly tabular representation.

Root	UseCases
	%UseCases
[]UseCases	Subpath	Id	Description	/Setup.MyDb.[]Customer	:data.DAILY_CAP	:data.OVERALL_CAP
	usecase10/001 /	1	@useCase.json	@customer.json| $data	200	3000
	usecase10/002 /	2	@useCase.json	@customer.json| $data	100	1000

Loading external resources lookups

1. For valid URL, new resource if returned with owner resource credential
2. For asset starting with /, a new file resource if returned with owner resource credential
3. For asset is a relative path then the following lookup are being used with the first successful to be used. a) owner resource path with subpath if provided and asset name b) owner resource path without subpath and asset name c) Local/remoteResourceRepo and asset name

Inline array use case

Take as example the following data structure:

{
"Actions": [
 {
   "Send": {
     "Request": [
       {
         "Method": "GET",
         "URL": "http://127.0.0.1/path1"
       },
       {
         "Method": "GET",
         "URL": "http://127.0.0.1/path2"
       }
     ],
     "Udf": "MyUdf"
   },
   "Expect": [
     {
       "Code": 200
     },
     {
       "Code": 404
     }
   ]
 },
 {
   "Send": {
     "Request": [
       {
         "Method": "GET",
         "URL": "http://127.0.0.1/path3"
       },
       {
         "Method": "GET",
         "URL": "http://127.0.0.1/path4"
       }
     ],
     "Udf": "MyUdf"
   },
   "Expect": [
     {
       "Code": 404
     },
     {
       "Code": 200
     }
   ]
 },
 {
       "Send": {
         "Request": [
           {
             "Method": "GET",
             "URL": "http://127.0.0.1/path5"
           },
           {
             "Method": "GET",
             "URL": "http://127.0.0.1/path6"
           }
         ],
         "Udf": "MyUdf"
       },
       "Expect": [
         {
           "Code": 200
         },
         {
           "Code": 200
         }
       ]
     }
]
}

Document	Actions
	%Actions
[]Actions	Send.Udf	Send.[]Requests.Method	Send.[]Requests.URL	[]Expect.Code
	MyUdf	GET	http://127.0.0.1/path1	200
		GET	http://127.0.0.1/path2	404
[]Actions	Send.Udf	Send.[]Requests.Method	Send.[]Requests.URL	[]Expect.Code
	MyUdf	GET	http://127.0.0.1/path3	404
		GET	http://127.0.0.1/path4	200
-	MyUdf	GET	http://127.0.0.1/path5	200
		GET	http://127.0.0.1/path6	200

If tag object field uses [] neatly will scan all rows down to add values to the array unless encounter next tag object or hyphen '-' in the first column.

Accessing meta data

For every object the following attributes will be set as object values, thus they should be treated as reserved keyword, unless object needs to expose them.

1. Tag name of currently processing tag.
2. TagId tag id of currently processing tag (concatenation of Tag name, Group if specified, Index if within Tag Range, Subpath is specified).
3. TagIndex index value if within tag range.
4. Subpath defines subpath.
5. PathMatch defines matched subpath.

Comments

To prevent line loading into document , // can be used at the beginning of the line, optionally followed by some comments.

Usage

Download or build latest neatly.

neatly -i=test/use_case1.csv -f=yaml

 $ neatly -h
Usage of neatly:
  -f string
    	<output format> json or yaml (default "json")
  -i string
    	<neatly document path> 
  -v	neatly version
  -f string
    	<output format> json or yaml (default "json")
  -m	include meta data -m=true

To convert neatly document into go data structure.

    import (
    		"github.com/viant/neatly"
        	"github.com/viant/toolbox/data"
        	"github.com/viant/toolbox/url"
    )


    var localAssetRepo, remoteAssetRepo string
	dao := neatly.NewDao(localAssetRepo, remoteAssetRepo, "yyyy-MM-dd h:mm:ss", nil)
	
	//user defined function
	var nilUdf = fun(source interface{}, state data.Map) (interface{}, error) {
		return nil, nil
	}
	
	var context = data.NewMap() //data is toolbox/data package
	//optionally register UDF
	context.Put("Nil", nilUdf)
    
	var targetObject = &MyStruct{} // or map[string]interface{}
	err := dao.Load(context, url.NewResource("mystruct.csv"), targetObject)
    if err != nil {
    	log.Fatal(err)
    }

License

The source code is made available under the terms of the Apache License, Version 2, as stated in the file LICENSE.

Individual files may be made available under their own specific license, all compatible with Apache License, Version 2. Please see individual files for details.

Credits and Acknowledgements

Library Author: Adrian Witas

Documentation

Index

• Constants
• func AddStandardUdf(aMap data.Map)
• func AssetsToMap(source interface{}, state data.Map) (interface{}, error)
• func BinaryAssetsToMap(source interface{}, state data.Map) (interface{}, error)
• func Cat(source interface{}, state data.Map) (interface{}, error)
• func GetOwnerDirectory(state data.Map) (string, error)
• func HasResource(source interface{}, state data.Map) (interface{}, error)
• func IsJSON(fileName interface{}, state data.Map) (interface{}, error)
• func LoadBinary(source interface{}, state data.Map) (interface{}, error)
• func LoadNeatly(source interface{}, state data.Map) (interface{}, error)
• func Markdown(source interface{}, state data.Map) (interface{}, error)
• func Md5(source interface{}, state data.Map) (interface{}, error)
• func Unzip(source interface{}, state data.Map) (interface{}, error)
• func UnzipText(source interface{}, state data.Map) (interface{}, error)
• func WorkingDirectory(source interface{}, state data.Map) (interface{}, error)
• func Zip(source interface{}, state data.Map) (interface{}, error)
• type Dao
  • func NewDao(includeMeta bool, localResourceRepo, remoteResourceRepo, dataFormat string, ...) *Dao
  • func (d *Dao) AddStandardUdf(context data.Map)
  • func (d *Dao) Load(context data.Map, source *url.Resource, target interface{}) error
  • func (d *Dao) NewRepoResource(context data.Map, URI string) (*url.Resource, error)
• type Field
  • func NewField(expression string) *Field
  • func (f *Field) ArrayPath() string
  • func (f *Field) GetArraySize(value data.Map) int
  • func (f *Field) Set(value interface{}, target data.Map, indexes ...int)
• type Tag
  • func NewTag(ownerName string, ownerSource *url.Resource, key string, lineNumber int) *Tag
  • func (t *Tag) Expand(text string) string
  • func (t *Tag) HasActiveIterator() bool
  • func (t *Tag) SetSubPath(subpath string)
  • func (t *Tag) TagID() string
• type TagIterator
  • func (i *TagIterator) Has() bool
  • func (i *TagIterator) Index() string
  • func (i *TagIterator) Next() bool

Constants

const (
	//OwnerURL represewnt currently loading neatly URL
	OwnerURL = "ownerURL"
	//NeatlyDao nearly dao key
	NeatlyDao = "nearlyDAO"
)

const AppName = "neatly"

const AppVersion = "0.3.0"

AppVersion represent neatly version

Functions

func AddStandardUdf

func AddStandardUdf(aMap data.Map)

AddStandardUdf register building udf to the context

func AssetsToMap

func AssetsToMap(source interface{}, state data.Map) (interface{}, error)

AssetsToMap loads assets into map[string]string, it takes url, with optional list of extension as filter

func BinaryAssetsToMap

func BinaryAssetsToMap(source interface{}, state data.Map) (interface{}, error)

BinaryAssetsToMap loads binary assets into map[string]string, it takes url, with optional list of extension as filter

func Cat

func Cat(source interface{}, state data.Map) (interface{}, error)

Cat returns content of supplied file name

func GetOwnerDirectory

func GetOwnerDirectory(state data.Map) (string, error)

GetOwnerDirectory returns owner neatly document directory

func HasResource

func HasResource(source interface{}, state data.Map) (interface{}, error)

HasResource check if patg/url to external resource exists

func IsJSON

func IsJSON(fileName interface{}, state data.Map) (interface{}, error)

Validate if JSON file is a well-formed JSON Returns true if file content is valid JSON

func LoadBinary

func LoadBinary(source interface{}, state data.Map) (interface{}, error)

LoadBinary returns []byte content of supplied file name

func LoadNeatly

func LoadNeatly(source interface{}, state data.Map) (interface{}, error)

LoadNeatly loads neatly document as data structure, source represents path to nearly document

func Markdown

func Markdown(source interface{}, state data.Map) (interface{}, error)

Markdown returns html fot supplied markdown

func Md5

func Md5(source interface{}, state data.Map) (interface{}, error)

Md5 computes source md5

func Unzip

func Unzip(source interface{}, state data.Map) (interface{}, error)

Unzip uncompress supplied []byte or error

func UnzipText

func UnzipText(source interface{}, state data.Map) (interface{}, error)

UnzipText uncompress supplied []byte into text or error

func WorkingDirectory

func WorkingDirectory(source interface{}, state data.Map) (interface{}, error)

WorkingDirectory return joined path with current directory, ../ is supported as subpath

func Zip

func Zip(source interface{}, state data.Map) (interface{}, error)

Zip compresses supplied []byte or test or error

Types

type Dao

type Dao struct {
	// contains filtered or unexported fields
}

Dao represents neatly data access object

func NewDao

func NewDao(includeMeta bool, localResourceRepo, remoteResourceRepo, dataFormat string, delimiterDecoderFactory toolbox.DecoderFactory) *Dao

NewDao creates a new neatly format compatible format data access object. It takes localResourceRepo, remoteResourceRepo, dataFormat and optionally delimiterDecoderFactory

func (*Dao) AddStandardUdf

func (d *Dao) AddStandardUdf(context data.Map)

AddStandardUdf register building udf to the context

func (*Dao) Load

func (d *Dao) Load(context data.Map, source *url.Resource, target interface{}) error

Load reads data from provided resource into the target pointer

func (*Dao) NewRepoResource

func (d *Dao) NewRepoResource(context data.Map, URI string) (*url.Resource, error)

NewRepoResource returns resource build as localResourceURL/remoteResourceURL and URI If Local resource does not exist but remote does it copy it over to Local to avoid remote round trips in the future.

type Field

type Field struct {
	Field             string //actual expression field
	Child             *Field //child expression if expression contains .
	IsArray           bool   //is this filed an array type
	HasSubPath        bool   //flag indicating if this field has sub fields
	HasArrayComponent bool   //flag indicating if this or child fieds have an array component
	IsRoot            bool   //flag indicating if this field is Root
	IsVirtual         bool   //flag indicating if this field belong to virtual object
	IsIndex           bool   //flag indicating if this filed is actual array index, as opposed to sub field name
	Leaf              *Field //leaf field
	// contains filtered or unexported fields
}

Field represent field of object

func NewField

func NewField(expression string) *Field

NewField return a new Field for provided expression.

func (*Field) ArrayPath

func (f *Field) ArrayPath() string

ArrayPath returns a field array path

func (*Field) GetArraySize

func (f *Field) GetArraySize(value data.Map) int

GetArraySize returns field array size

func (*Field) Set

func (f *Field) Set(value interface{}, target data.Map, indexes ...int)

Set sets value into target map, if indexes are provided value will be pushed into a slice

type Tag

type Tag struct {
	OwnerSource *url.Resource
	OwnerName   string
	Name        string
	Group       string
	IsArray     bool
	Iterator    *TagIterator
	LineNumber  int
	Subpath     string
	PathMatch   string
	// contains filtered or unexported fields
}

Tag represents a nearly tag

func NewTag

func NewTag(ownerName string, ownerSource *url.Resource, key string, lineNumber int) *Tag

NewTag creates a new neatly tag

func (*Tag) Expand

func (t *Tag) Expand(text string) string

func (*Tag) HasActiveIterator

func (t *Tag) HasActiveIterator() bool

HasActiveIterator returns true if tag has active iterator

func (*Tag) SetSubPath

func (t *Tag) SetSubPath(subpath string)

SetSubPath set subpath for the tag

func (*Tag) TagID

func (t *Tag) TagID() string

TagID returns tag ID

type TagIterator

type TagIterator struct {
	Template string
	Min      int
	Max      int
	// contains filtered or unexported fields
}

TagIterator represents tag iterator to produce TagIndex

func (*TagIterator) Has

func (i *TagIterator) Has() bool

Has return true if iterator has not been exhausted

func (*TagIterator) Index

func (i *TagIterator) Index() string

Index returns an index of the iterator

func (*TagIterator) Next

func (i *TagIterator) Next() bool

Next eturns increment counter and checks if it is has next.