datalark

README

go-datalark

Bindings for IPLD in Starlark.

Starlark is a dynamic interpreted language that is easy to embed in golang programs (and also has interpreters in Rust, and in Java -- it can probably go anywhere). The syntax is python-like and easily approachable for developers from many backgrounds. (Remarkably, it's also non-turing complete and inherently sandboxed -- it has no access to a host enviroment (unless you intentionally give the interpreter sandbox custom functions that can do that!).)

IPLD is a data interchange standard, with emphasis on utility in the decentralized web.

Marrying Starlark and IPLD gives you a data interchange format that can talk to many kinds of APIs and is great for building new protocols... and gives you a full-fledged (yet sandboxed, safe) langauge you can use for anything from basic templating (e.g. for config files) all the way to being a generic data transform language.

Features

• Wrap ipld.Node values from golang and hand them to a Starlark interpreter to read.

• Get constructor functions you can hand to a Starlark interpreter so it can make basic ipld.Nodes (maps, strings, lists, etc etc).

• Combined with bindnode.Wrap: easily expose arbitrary golang structs and types to a Starlark interpreter environment to read as if they were dicts and lists and regular starlark values.

• Combined with IPLD Schemas: get constructor functions you can hand to a Starlark interpreter to make those types.

  • ... so that you can access all of IPLD Schema's validation powers from within Starlark!
  • ... so that you can use kwargs initialization to create struct types with syntactic grace.
  • ... so that you can have dotted access to struct fields in the Starlark syntax, just like you'd expect.
  • ... so that, together with bindnode, you can fill in golang native structs from Starlark with ease!

• Does it support ADLs? Of course it does!

  • ADLs are just ipld.Nodes, so they work with datalark just like any other nodes do. No fuss.

• Use Starlark's regular print function to get the IPLD debug printout format for data, for rapid development and easy debugging.

• Well-behaved as a library: go-datalark doesn't wrap the Starlark interpreter or do any other weird non-composable hacks: it just gives you functions and variables that you hand into the Starlark environment (and, a few helper functions to make that even easier to do). That's it. It's totally composable.

Status

Datalark should currently be considered in an alpha or beta status. It's working, but there is currently no promise of API stability. Some features may also be missing.

The best way to increase the stability and completeness of datalark is to start using it! We're adding features as the demand for them is found.

Naming Convention

Variables used in the engine of datalark occasionally use prefixes, for example names such as "skey" and "starVal" and "hostItem". The prefixes in total are "host", "star", "node", and the single letter prefixes "h", "s", "n" and "g", "t", "m" respectively.

While we normally avoid things like Hungarian Notion and just let the type system do its job, the prefixes in use here are doing something different. Since datalark exists as a middle layer between two type systems (ipl and starlark), we end up with lots of different kinds of objects that all represent the same abstract data. For example, a string could be a "datalark.String", or a "starlark.String", or a "ipld.datamodel.Node" of type "string", or a native go-lang "string". Often these types all appear at once in the same function, and this is not something a type system alone does not do a good job of differentiating. Trying to use only unprefixed names will lead to either confusing or inconsistent code across functions, and code that is much more difficult to read.

The variable name prefixes in use:

prefix      type system           examples

h/g/host    datalark              hval, hostVal
s/t/star    starlark              skey, starObj
n/m/node    ipld.datamodel.Node   nodeMapIter
(plain)     go-lang native value  name, err

Contributing

If you'd like to dive deep in on code, there's a HACKME guide in the engine package which gives some pointers for how to orient yourself in the code.

If you want to get in touch with folks working on this for a chat: this project is part of the IPLD community, so you can follow any of the suggestions in https://github.com/ipld/ipld#finding-us -- there's a matrix chat, and a discord bridged to it, and you should find some friendly folks there.

License

SPDX-License-Identifier: Apache-2.0 OR MIT

Documentation

Overview

datalark makes IPLD data legible to, and constructable in, starlark.

Given an IPLD Schema (and optionally, a list of types to focus on), datalark can generate a set of starlark constructor functions for those types. These functions should generally DWIM ("do what I mean"): for structs, they accept kwargs corresponding to the field names, etc. Some functions get clever: for example, for structs with stringy representations (stringjoin, etc), the representation form can be used as an argument to the constructor instead of the kwargs form, and the construction will "DWIM" with that information and parse it in the appropriate way.

Standard datamodel data is also always legible, and a set of functions for creating it can also be obtained from the datalark package.

All IPLD data exposed to starlark always acts as if it is "frozen", in starlark parlance. This should be unsurprising, since IPLD is already oriented around immutability.

datalark can be used on natural golang structs by combining it with the go-ipld-prime/node/bindnode package. This may make it an interesting alternative to github.com/starlight-go/starlight (although admittedly more complicated; it's probably only worth it if you also already value some of the features of IPLD Schemas).

Future objectives for this package include the ability to provide a function to starlark which will accept an IPLD Schema document and a type name as parameters, and will return a constructor for that type. (Not yet implemented.)

Example (Hello)

// Prepare things needed by a starlark interpreter.  (This is Starlark boilerplate!)
thread := &starlark.Thread{
	Name: "thethreadname",
	Print: func(thread *starlark.Thread, msg string) {
		fmt.Printf("%s\n", msg)
	},
}

// Use datalark to make IPLD value constructors available to Starlark!
globals := starlark.StringDict{}
globals["datalark"] = datalark.PrimitiveConstructors()

// Now here's our demo script:
script := testutil.Dedent(`
		print(datalark.String("yo"))
	`)

// Invoke the starlark interpreter!
_, err := starlark.ExecFile(thread, "thefilename.star", script, globals)
if err != nil {
	panic(err)
}

Output:

string{"yo"}

Example (HelloGlobals)

// In this example, we do similar things to the other examples,
// except we put our functions directly into the global namespace.
// You may wish to do this to make it even easier to use
// (but remember to weigh it against cluttering the namespace your users will experience!).

// Prepare things needed by a starlark interpreter.  (This is Starlark boilerplate!)
thread := &starlark.Thread{
	Name: "thethreadname",
	Print: func(thread *starlark.Thread, msg string) {
		fmt.Printf("%s\n", msg)
	},
}

// Use datalark to make IPLD value constructors available to Starlark!
// Note the use of 'InjectGlobals' here -- this puts things into scope without any namespace,
// as opposed to what we did in other examples, which let you choose a name in the globals to put everything under.
globals := starlark.StringDict{}
datalark.InjectGlobals(globals, datalark.PrimitiveConstructors())

// Now here's our demo script:
script := testutil.Dedent(`
		print(String("yo")) # look, no 'datalark.' prefix!
	`)

// Invoke the starlark interpreter!
_, err := starlark.ExecFile(thread, "thefilename.star", script, globals)
if err != nil {
	panic(err)
}

Output:

string{"yo"}

Example (HelloTypes)

// In this example we'll use an IPLD Schema!
typesystem, err := ipld.LoadSchema("<noname>", strings.NewReader(`
		type FooBar struct {
			foo String
			bar String
		}
	`))
if err != nil {
	panic(err)
}

// And we'll bind it to this golang native type:
type FooBar struct{ Foo, Bar string }

// Prepare things needed by a starlark interpreter.  (This is Starlark boilerplate!)
thread := &starlark.Thread{
	Name: "thethreadname",
	Print: func(thread *starlark.Thread, msg string) {
		fmt.Printf("%s\n", msg)
	},
}

// Use datalark to make IPLD value constructors available to Starlark!
globals := starlark.StringDict{}
globals["datalark"] = datalark.PrimitiveConstructors()
globals["mytypes"] = datalark.MakeConstructors(
	[]schema.TypedPrototype{
		bindnode.Prototype((*FooBar)(nil), typesystem.TypeByName("FooBar")),
	},
)

// Now here's our demo script:
script := testutil.Dedent(`
		print(mytypes.FooBar)
		print(mytypes.FooBar(foo="helloooo", bar="world!"))
	`)

// Invoke the starlark interpreter!
_, err = starlark.ExecFile(thread, "thefilename.star", script, globals)
if err != nil {
	panic(err)
}

Output:

<built-in function datalark.Prototype<FooBar>>
struct<FooBar>{
	foo: string<String>{"helloooo"}
	bar: string<String>{"world!"}
}

Index

• func InjectGlobals(globals starlark.StringDict, obj *datalarkengine.Object)
• func MakeConstructors(prototypes []schema.TypedPrototype) *datalarkengine.Object
• func PrimitiveConstructors() *datalarkengine.Object

Examples

• Package (Hello)
• Package (HelloGlobals)
• Package (HelloTypes)

Functions

func InjectGlobals

func InjectGlobals(globals starlark.StringDict, obj *datalarkengine.Object)

InjectGlobals mutates a starlark.StringDict to contain the values in the given Object. It will panic if keys that aren't starlark.String are encountered, if iterators error, etc.

func MakeConstructors

func MakeConstructors(prototypes []schema.TypedPrototype) *datalarkengine.Object

MakeConstructors returns an Object containing constructor functions for IPLD typed nodes, based on the list of schema.TypedPrototype provided, and using the names of each of those prototype's types as the keys.

func PrimitiveConstructors

func PrimitiveConstructors() *datalarkengine.Object

PrimitiveConstrutors returns an Object containing constructor functions for all the IPLD Data Model kinds -- strings, maps, etc -- as those names, in TitleCase.