jsonnet

README

go-jsonnet

This an implementation of Jsonnet in pure Go. It is a feature complete, production-ready implementation. It is compatible with the original Jsonnet C++ implementation. Bindings to C and Python are available (but not battle-tested yet).

This code is known to work on Go 1.12 and above. We recommend always using the newest stable release of Go.

Installation instructions

# go >= 1.17
# Using `go get` to install binaries is deprecated.
# The version suffix is mandatory.
go install github.com/google/go-jsonnet/cmd/jsonnet@latest

# Or other tools in the 'cmd' directory
go install github.com/google/go-jsonnet/cmd/jsonnet-lint@latest

# go < 1.17
go get github.com/google/go-jsonnet/cmd/jsonnet

It's also available on Homebrew:

brew install go-jsonnet

jsonnetfmt and jsonnet-lint are also available as pre-commit hooks. Example .pre-commit-config.yaml:

- repo: https://github.com/google/go-jsonnet
  rev: # ref you want to point at, e.g. v0.17.0
  hooks:
    - id: jsonnet-format
    - id: jsonnet-lint

It can also be embedded in your own Go programs as a library:

package main

import (
	"fmt"
	"log"

	"github.com/google/go-jsonnet"
)

func main() {
	vm := jsonnet.MakeVM()

	snippet := `{
		person1: {
		    name: "Alice",
		    welcome: "Hello " + self.name + "!",
		},
		person2: self.person1 { name: "Bob" },
	}`

	jsonStr, err := vm.EvaluateAnonymousSnippet("example1.jsonnet", snippet)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(jsonStr)
	/*
	   {
	     "person1": {
	         "name": "Alice",
	         "welcome": "Hello Alice!"
	     },
	     "person2": {
	         "name": "Bob",
	         "welcome": "Hello Bob!"
	     }
	   }
	*/
}

Build instructions (go 1.12+)

git clone git@github.com:google/go-jsonnet.git
cd go-jsonnet
go build ./cmd/jsonnet
go build ./cmd/jsonnetfmt
go build ./cmd/jsonnet-deps

To build with Bazel instead:

git clone git@github.com:google/go-jsonnet.git
cd go-jsonnet
git submodule init
git submodule update
bazel build //cmd/jsonnet
bazel build //cmd/jsonnetfmt
bazel build //cmd/jsonnet-deps

The resulting jsonnet program will then be available at a platform-specific path, such as bazel-bin/cmd/jsonnet/darwin_amd64_stripped/jsonnet for macOS.

Bazel also accommodates cross-compiling the program. To build the jsonnet program for various popular platforms, run the following commands:

Target platform	Build command
Current host	bazel build //cmd/jsonnet
Linux	bazel build --platforms=@io_bazel_rules_go//go/toolchain:linux_amd64 //cmd/jsonnet
macOS	bazel build --platforms=@io_bazel_rules_go//go/toolchain:darwin_amd64 //cmd/jsonnet
Windows	bazel build --platforms=@io_bazel_rules_go//go/toolchain:windows_amd64 //cmd/jsonnet

For additional target platform names, see the per-Go release definitions here in the rules_go Bazel package.

Additionally if any files were moved around, see the section Keeping the Bazel files up to date.

Building libjsonnet.wasm

GOOS=js GOARCH=wasm go build -o libjsonnet.wasm ./cmd/wasm 

Or if using bazel:

bazel build //cmd/wasm:libjsonnet.wasm

Running tests

./tests.sh  # Also runs `go test ./...`

Running Benchmarks

Method 1

go get golang.org/x/tools/cmd/benchcmp

1. Make sure you build a jsonnet binary prior to making changes.

go build -o jsonnet-old ./cmd/jsonnet

1. Make changes (iterate as needed), and rebuild new binary

go build ./cmd/jsonnet

1. Run benchmark:

# e.g. ./benchmark.sh Builtin
./benchmark.sh <TestNameFilter>

Method 2

1. get benchcmp

go get golang.org/x/tools/cmd/benchcmp

1. Make sure you build a jsonnet binary prior to making changes.

make build-old

1. iterate with (which will also automatically rebuild the new binary ./jsonnet)

replace the FILTER with the name of the test you are working on

FILTER=Builtin_manifestJsonEx make benchmark

Implementation Notes

We are generating some helper classes on types by using http://clipperhouse.github.io/gen/. Do the following to regenerate these if necessary:

go get github.com/clipperhouse/gen
go get github.com/clipperhouse/set
export PATH=$PATH:$GOPATH/bin  # If you haven't already
go generate

Update cpp-jsonnet sub-repo

This repo depends on the original Jsonnet repo. Shared parts include the standard library, headers files for C API and some tests.

You can update the submodule and regenerate dependent files with one command:

./update_cpp_jsonnet.sh

Note: It needs to be run from repo root.

Updating and modifying the standard library

Standard library source code is kept in cpp-jsonnet submodule, because it is shared with Jsonnet C++ implementation.

For performance reasons we perform preprocessing on the standard library, so for the changes to be visible, regeneration is necessary:

go run cmd/dumpstdlibast/dumpstdlibast.go cpp-jsonnet/stdlib/std.jsonnet > astgen/stdast.go

**The

The above command creates the astgen/stdast.go file which puts the desugared standard library into the right data structures, which lets us avoid the parsing overhead during execution. Note that this step is not necessary to perform manually when building with Bazel; the Bazel target regenerates the astgen/stdast.go (writing it into Bazel's build sandbox directory tree) file when necessary.

Keeping the Bazel files up to date

Note that we maintain the Go-related Bazel targets with the Gazelle tool. The Go module (go.mod in the root directory) remains the primary source of truth. Gazelle analyzes both that file and the rest of the Go files in the repository to create and adjust appropriate Bazel targets for building Go packages and executable programs.

After changing any dependencies within the files covered by this Go module, it is helpful to run go mod tidy to ensure that the module declarations match the state of the Go source code. In order to synchronize the Bazel rules with material changes to the Go module, run the following command to invoke Gazelle's update-repos command:

bazel run //:gazelle -- update-repos -from_file=go.mod -to_macro=bazel/deps.bzl%jsonnet_go_dependencies

Similarly, after adding or removing Go source files, it may be necessary to synchronize the Bazel rules by running the following command:

bazel run //:gazelle

Documentation

Overview

Package jsonnet implements a parser and evaluator for jsonnet.

Jsonnet is a domain specific configuration language that helps you define JSON data. Jsonnet lets you compute fragments of JSON within the structure, bringing the same benefit to structured data that templating languages bring to plain text.

See http://jsonnet.org/ for a full language description and tutorial.

Index

• func SnippetToAST(filename string, snippet string) (ast.Node, error)
• func Version() string
• type ColorFormatter
• type Contents
  • func MakeContents(s string) Contents
  • func MakeContentsRaw(bytes []byte) Contents
  • func (c Contents) Data() []byte
  • func (c Contents) String() string
• type ErrorFormatter
• type FileImporter
  • func (importer *FileImporter) Import(importedFrom, importedPath string) (contents Contents, foundAt string, err error)
• type Importer
• type LineReader
  • func (r *LineReader) Read() ([]byte, error)
• type MemoryImporter
  • func (importer *MemoryImporter) Import(importedFrom, importedPath string) (contents Contents, foundAt string, err error)
• type NativeFunction
• type Reader
• type RuntimeError
  • func (err RuntimeError) Error() string
• type VM
  • func MakeVM() *VM
  • func (vm *VM) Evaluate(node ast.Node) (val string, err error)
  • func (vm *VM) EvaluateAnonymousSnippet(filename string, snippet string) (json string, formattedErr error)
  • func (vm *VM) EvaluateAnonymousSnippetMulti(filename string, snippet string) (files map[string]string, formattedErr error)
  • func (vm *VM) EvaluateAnonymousSnippetStream(filename string, snippet string) (docs []string, formattedErr error)
  • func (vm *VM) EvaluateFile(filename string) (json string, formattedErr error)
  • func (vm *VM) EvaluateFileMulti(filename string) (files map[string]string, formattedErr error)
  • func (vm *VM) EvaluateFileStream(filename string) (docs []string, formattedErr error)
  • func (vm *VM) EvaluateMulti(node ast.Node) (output map[string]string, err error)
  • func (vm *VM) EvaluateSnippet(filename string, snippet string) (json string, formattedErr error)
  • func (vm *VM) EvaluateSnippetMulti(filename string, snippet string) (files map[string]string, formattedErr error)
  • func (vm *VM) EvaluateSnippetStream(filename string, snippet string) (docs []string, formattedErr error)
  • func (vm *VM) EvaluateStream(node ast.Node) (output []string, err error)
  • func (vm *VM) ExtCode(key string, val string)
  • func (vm *VM) ExtNode(key string, node ast.Node)
  • func (vm *VM) ExtReset()
  • func (vm *VM) ExtVar(key string, val string)
  • func (vm *VM) FindDependencies(importedFrom string, importedPaths []string) ([]string, error)
  • func (vm *VM) ImportAST(importedFrom, importedPath string) (contents ast.Node, foundAt string, err error)
  • func (vm *VM) ImportData(importedFrom, importedPath string) (contents string, foundAt string, err error)
  • func (vm *VM) Importer(i Importer)
  • func (vm *VM) NativeFunction(f *NativeFunction)
  • func (vm *VM) ResolveImport(importedFrom, importedPath string) (foundAt string, err error)
  • func (vm *VM) SetTraceOut(traceOut io.Writer)
  • func (vm *VM) TLACode(key string, val string)
  • func (vm *VM) TLANode(key string, node ast.Node)
  • func (vm *VM) TLAReset()
  • func (vm *VM) TLAVar(key string, val string)
• type YAMLReader
  • func NewYAMLReader(r *bufio.Reader) *YAMLReader
  • func (r *YAMLReader) Read() ([]byte, error)
• type YAMLToJSONDecoder
  • func NewYAMLToJSONDecoder(r io.Reader) *YAMLToJSONDecoder
  • func (d *YAMLToJSONDecoder) Decode(into interface{}) error

Functions

func SnippetToAST

func SnippetToAST(filename string, snippet string) (ast.Node, error)

SnippetToAST parses a snippet and returns the resulting AST.

func Version

func Version() string

Version returns the Jsonnet version number.

Types

type ColorFormatter

type ColorFormatter func(w io.Writer, f string, a ...interface{}) (n int, err error)

ColorFormatter represents a function that writes to the terminal using color.

type Contents

type Contents struct {
	// contains filtered or unexported fields
}

Contents is a representation of imported data. It is a simple byte wrapper, which makes it easier to enforce the caching policy.

func MakeContents

func MakeContents(s string) Contents

MakeContents creates Contents from a string.

func MakeContentsRaw

func MakeContentsRaw(bytes []byte) Contents

MakeContentsRaw creates Contents from (possibly non-utf8) []byte data.

func (Contents) Data

func (c Contents) Data() []byte

Data returns content bytes

func (Contents) String

func (c Contents) String() string

type ErrorFormatter

type ErrorFormatter interface {
	// Format static, runtime, and unexpected errors prior to printing them.
	Format(err error) string

	// Set the the maximum length of stack trace before cropping.
	SetMaxStackTraceSize(size int)

	// Set the color formatter for the location color.
	SetColorFormatter(color ColorFormatter)
}

An ErrorFormatter formats errors with stacktraces and color.

type FileImporter

type FileImporter struct {
	JPaths []string
	// contains filtered or unexported fields
}

FileImporter imports data from the filesystem.

func (*FileImporter) Import

func (importer *FileImporter) Import(importedFrom, importedPath string) (contents Contents, foundAt string, err error)

Import imports file from the filesystem.

type Importer

type Importer interface {
	// Import fetches data from a given path. It may be relative
	// to the file where we do the import. What "relative path"
	// means depends on the importer.
	//
	// It is required that:
	// a) for given (importedFrom, importedPath) the same
	//    (contents, foundAt) are returned on subsequent calls.
	// b) for given foundAt, the contents are always the same
	//
	// It is recommended that if there are multiple locations that
	// need to be probed (e.g. relative + multiple library paths)
	// then all results of all attempts will be cached separately,
	// both nonexistence and contents of existing ones.
	// FileImporter may serve as an example.
	//
	// IMPORTANT: The passed importedFrom might be "" (an empty string).
	// It means that the import is coming from an ad-hoc snippet, e.g.
	// code passed on the command line, read from stdin or passed as
	// a snippet to execute. Importer may have a "default" path to use
	// in such case or it may only allow absolute imports from such
	// "anonymous locations".
	//
	// Importing the same file multiple times must be a cheap operation
	// and shouldn't involve copying the whole file - the same buffer
	// should be returned.
	Import(importedFrom, importedPath string) (contents Contents, foundAt string, err error)
}

An Importer imports data from a path. TODO(sbarzowski) caching of errors (may require breaking changes)

type LineReader

type LineReader struct {
	// contains filtered or unexported fields
}

LineReader reads single lines.

func (*LineReader) Read

func (r *LineReader) Read() ([]byte, error)

Read returns a single line (with '\n' ended) from the underlying reader. An error is returned iff there is an error with the underlying reader.

type MemoryImporter

type MemoryImporter struct {
	Data map[string]Contents
}

MemoryImporter "imports" data from an in-memory map.

func (*MemoryImporter) Import

func (importer *MemoryImporter) Import(importedFrom, importedPath string) (contents Contents, foundAt string, err error)

Import fetches data from a map entry. All paths are treated as absolute keys.

type NativeFunction

type NativeFunction struct {
	Name   string
	Func   func([]interface{}) (interface{}, error)
	Params ast.Identifiers
}

NativeFunction represents a function implemented in Go.

type Reader

type Reader interface {
	Read() ([]byte, error)
}

Reader reads bytes

type RuntimeError

type RuntimeError struct {
	Msg        string
	StackTrace []traceFrame
}

RuntimeError is an error discovered during evaluation of the program

func (RuntimeError) Error

func (err RuntimeError) Error() string

type VM

type VM struct {
	MaxStack int

	ErrorFormatter ErrorFormatter
	StringOutput   bool
	// contains filtered or unexported fields
}

VM is the core interpreter and is the touchpoint used to parse and execute Jsonnet.

func MakeVM

func MakeVM() *VM

MakeVM creates a new VM with default parameters.

func (*VM) Evaluate

func (vm *VM) Evaluate(node ast.Node) (val string, err error)

Evaluate evaluates a Jsonnet program given by an Abstract Syntax Tree and returns serialized JSON as string. TODO(sbarzowski) perhaps is should return JSON in standard Go representation

func (*VM) EvaluateAnonymousSnippet

func (vm *VM) EvaluateAnonymousSnippet(filename string, snippet string) (json string, formattedErr error)

EvaluateAnonymousSnippet evaluates a string containing Jsonnet code, return a JSON string.

The filename parameter is only used for error messages.

func (*VM) EvaluateAnonymousSnippetMulti

func (vm *VM) EvaluateAnonymousSnippetMulti(filename string, snippet string) (files map[string]string, formattedErr error)

EvaluateAnonymousSnippetMulti evaluates a string containing Jsonnet code to key-value pairs. The keys are field name strings and the values are JSON strings.

The filename parameter is only used for error messages.

func (*VM) EvaluateAnonymousSnippetStream

func (vm *VM) EvaluateAnonymousSnippetStream(filename string, snippet string) (docs []string, formattedErr error)

EvaluateAnonymousSnippetStream evaluates a string containing Jsonnet code to an array. The array is returned as an array of JSON strings.

The filename parameter is only used for error messages.

func (*VM) EvaluateFile

func (vm *VM) EvaluateFile(filename string) (json string, formattedErr error)

EvaluateFile evaluates Jsonnet code in a file and returns a JSON string.

The importer is used to fetch the contents of the file.

func (*VM) EvaluateFileMulti

func (vm *VM) EvaluateFileMulti(filename string) (files map[string]string, formattedErr error)

EvaluateFileMulti evaluates Jsonnet code in a file to key-value pairs. The keys are field name strings and the values are JSON strings.

The importer is used to fetch the contents of the file.

func (*VM) EvaluateFileStream

func (vm *VM) EvaluateFileStream(filename string) (docs []string, formattedErr error)

EvaluateFileStream evaluates Jsonnet code in a file to an array. The array is returned as an array of JSON strings.

The importer is used to fetch the contents of the file.

func (*VM) EvaluateMulti

func (vm *VM) EvaluateMulti(node ast.Node) (output map[string]string, err error)

EvaluateMulti evaluates a Jsonnet program given by an Abstract Syntax Tree and returns key-value pairs. The keys are strings and the values are JSON strigns (serialized JSON).

func (*VM) EvaluateSnippet

func (vm *VM) EvaluateSnippet(filename string, snippet string) (json string, formattedErr error)

EvaluateSnippet evaluates a string containing Jsonnet code, return a JSON string.

The filename parameter is used for resolving relative imports and for errors messages.

Deprecated: Use EvaluateFile or EvaluateAnonymousSnippet instead.

func (*VM) EvaluateSnippetMulti

func (vm *VM) EvaluateSnippetMulti(filename string, snippet string) (files map[string]string, formattedErr error)

EvaluateSnippetMulti evaluates a string containing Jsonnet code to key-value pairs. The keys are field name strings and the values are JSON strings.

The filename parameter is used for resolving relative imports and for errors messages.

Deprecated: Use EvaluateFileMulti or EvaluateAnonymousSnippetMulti instead.

func (*VM) EvaluateSnippetStream

func (vm *VM) EvaluateSnippetStream(filename string, snippet string) (docs []string, formattedErr error)

EvaluateSnippetStream evaluates a string containing Jsonnet code to an array. The array is returned as an array of JSON strings.

The filename parameter is used for resolving relative imports and for errors messages.

Deprecated: Use EvaluateFileStream or EvaluateAnonymousSnippetStream instead.

func (*VM) EvaluateStream

func (vm *VM) EvaluateStream(node ast.Node) (output []string, err error)

EvaluateStream evaluates a Jsonnet program given by an Abstract Syntax Tree and returns an array of JSON strings.

func (*VM) ExtCode

func (vm *VM) ExtCode(key string, val string)

ExtCode binds a Jsonnet external code var to the given code.

func (*VM) ExtNode

func (vm *VM) ExtNode(key string, node ast.Node)

ExtNode binds a Jsonnet external code var to the given AST node.

func (*VM) ExtReset

func (vm *VM) ExtReset()

ExtReset rests all external variables registered for this VM.

func (*VM) ExtVar

func (vm *VM) ExtVar(key string, val string)

ExtVar binds a Jsonnet external var to the given value.

func (*VM) FindDependencies

func (vm *VM) FindDependencies(importedFrom string, importedPaths []string) ([]string, error)

FindDependencies returns a sorted array of unique transitive dependencies (via import/importstr/importbin) from all the given `importedPaths` which are themselves excluded from the returned array. The `importedPaths` are parsed as if they were imported from a Jsonnet file located at `importedFrom`.

func (*VM) ImportAST

func (vm *VM) ImportAST(importedFrom, importedPath string) (contents ast.Node, foundAt string, err error)

ImportAST fetches the Jsonnet AST just as if it was imported from a Jsonnet file located at `importedFrom`. It shares the cache with the actual evaluation.

func (*VM) ImportData

func (vm *VM) ImportData(importedFrom, importedPath string) (contents string, foundAt string, err error)

ImportData fetches the data just as if it was imported from a Jsonnet file located at `importedFrom`. It shares the cache with the actual evaluation.

func (*VM) Importer

func (vm *VM) Importer(i Importer)

Importer sets Importer to use during evaluation (import callback).

func (*VM) NativeFunction

func (vm *VM) NativeFunction(f *NativeFunction)

NativeFunction registers a native function.

func (*VM) ResolveImport

func (vm *VM) ResolveImport(importedFrom, importedPath string) (foundAt string, err error)

ResolveImport finds the actual path where the imported file can be found. It will cache the contents of the file immediately as well, to avoid the possibility of the file disappearing after being checked.

func (*VM) SetTraceOut

func (vm *VM) SetTraceOut(traceOut io.Writer)

SetTraceOut sets the output stream for the builtin function std.trace().

func (*VM) TLACode

func (vm *VM) TLACode(key string, val string)

TLACode binds a Jsonnet top level argument to the given code.

func (*VM) TLANode

func (vm *VM) TLANode(key string, node ast.Node)

TLANode binds a Jsonnet top level argument to the given AST node.

func (*VM) TLAReset

func (vm *VM) TLAReset()

TLAReset resets all TLAs registered for this VM.

func (*VM) TLAVar

func (vm *VM) TLAVar(key string, val string)

TLAVar binds a Jsonnet top level argument to the given value.

type YAMLReader

type YAMLReader struct {
	// contains filtered or unexported fields
}

YAMLReader reads YAML

func NewYAMLReader

func NewYAMLReader(r *bufio.Reader) *YAMLReader

NewYAMLReader creates a new YAMLReader

func (*YAMLReader) Read

func (r *YAMLReader) Read() ([]byte, error)

Read returns a full YAML document.

type YAMLToJSONDecoder

type YAMLToJSONDecoder struct {
	// contains filtered or unexported fields
}

YAMLToJSONDecoder decodes YAML documents from an io.Reader by separating individual documents. It first converts the YAML body to JSON, then unmarshals the JSON.

func NewYAMLToJSONDecoder

func NewYAMLToJSONDecoder(r io.Reader) *YAMLToJSONDecoder

NewYAMLToJSONDecoder decodes YAML documents from the provided stream in chunks by converting each document (as defined by the YAML spec) into its own chunk, converting it to JSON via yaml.YAMLToJSON, and then passing it to json.Decoder.

func (*YAMLToJSONDecoder) Decode

func (d *YAMLToJSONDecoder) Decode(into interface{}) error

Decode reads a YAML document as JSON from the stream or returns an error. The decoding rules match json.Unmarshal, not yaml.Unmarshal.