skytf

README

Qri Skylark Transformation Syntax

Qri ("query") is about datasets. Transformions are repeatable scripts for generating a dataset. Skylark is a scripting langauge from Google that feels a lot like python. This package implements skylark as a transformation syntax. Skylark tranformations are about as close as one can get to the full power of a programming language as a transformation syntax. Often you need this degree of control to generate a dataset.

Typical examples of a skylark transformation include:

• combining paginated calls to an API into a single dataset
• downloading unstructured structured data from the internet to extract
• re-shaping raw input data before saving a dataset

We're excited about skylark for a few reasons:

• python syntax - many people working in data science these days write python, we like that, skylark likes that. dope.
• deterministic subset of python - unlike python, skylark removes properties that reduce introspection into code behaviour. things like while loops and recursive functions are ommitted, making it possible for qri to infer how a given transformation will behave.
• parallel execution - thanks to this deterministic requirement (and lack of global interpreter lock) skylark functions can be executed in parallel. Combined with peer-2-peer networking, we're hoping to advance tranformations toward peer-driven distribed computing. More on that in the coming months.

Getting started

If you're mainly interested in learning how to write skylark transformations, our documentation is a better place to start. If you're interested in contributing to the way skylark transformations work, this is the place!

The easiest way to see skylark transformations in action is to use qri. This skytf package powers all the skylark stuff in qri. Assuming you have the go programming language the following should work from a terminal:

# get this package
$ go get github.com/qri-io/skytf

# navigate to package
$ cd $GOPATH/src/github.com/qri-io/skytf

# run tests
$ go test ./...

Often the next steps are to install qri, mess with this skytf package, then rebuild qri with your changes to see them in action within qri itself.

Skylark Data Functions

Data Functions are the core of a skylark transform script. Here's an example of a very simple data function:

def transform(qri):
  return(["hello","world"])

Skylark transformations have a few rules on top of skylark itself:

• Data functions always return data
• When you define a data function, qri calls it for you
• All tranform functions are optional (you don't need to define them), but
• A transformation must have at least one data function
• Data functions are always called in the same order
• Data functions often get a qri parameter that lets them do special things

---

Documentation

Overview

Package skytf implements dataset transformations using the skylark programming dialect For more info on skylark check github.com/google/skylark

Index

• Constants
• Variables
• func DefaultExecOpts(o *ExecOpts)
• func Error(thread *skylark.Thread, _ *skylark.Builtin, args skylark.Tuple, ...) (skylark.Value, error)
• func ExecFile(ds *dataset.Dataset, filename string, infile cafs.File, ...) (dsio.EntryReader, error)
• type EntryReader
  • func NewEntryReader(st *dataset.Structure, data skylark.Iterable) *EntryReader
  • func (r *EntryReader) ReadEntry() (e dsio.Entry, err error)
  • func (r *EntryReader) Structure() *dataset.Structure
• type ExecOpts

Constants

const Version = "0.0.2"

Version is the current version of this skytf, this version number will be written with each transformation exectution

Variables

var ErrNotDefined = fmt.Errorf("not defined")

ErrNotDefined is for when a skylark value is not defined or does not exist

Functions

func DefaultExecOpts

func DefaultExecOpts(o *ExecOpts)

DefaultExecOpts applies default options to an ExecOpts pointer

func Error

func Error(thread *skylark.Thread, _ *skylark.Builtin, args skylark.Tuple, kwargs []skylark.Tuple) (skylark.Value, error)

Error halts program execution with an error

func ExecFile

func ExecFile(ds *dataset.Dataset, filename string, infile cafs.File, opts ...func(o *ExecOpts)) (dsio.EntryReader, error)

ExecFile executes a transformation against a skylark file located at filepath, giving back an EntryReader of resulting data ExecFile modifies the given dataset pointer. At bare minimum it will set transformation details, but skylark scripts can modify many parts of the dataset pointer, including meta, structure, and transform

Types

type EntryReader

type EntryReader struct {
	// contains filtered or unexported fields
}

EntryReader implements the dsio.EntryReader interface for skylark.Iterable's

func NewEntryReader

func NewEntryReader(st *dataset.Structure, data skylark.Iterable) *EntryReader

NewEntryReader creates a new Entry Reader

func (*EntryReader) ReadEntry

func (r *EntryReader) ReadEntry() (e dsio.Entry, err error)

ReadEntry reads one entry from the reader

func (*EntryReader) Structure

func (r *EntryReader) Structure() *dataset.Structure

Structure gives this reader's structure

type ExecOpts

type ExecOpts struct {
	AllowFloat     bool                   // allow floating-point numbers
	AllowSet       bool                   // allow set data type
	AllowLambda    bool                   // allow lambda expressions
	AllowNestedDef bool                   // allow nested def statements
	Secrets        map[string]interface{} // passed-in secrets (eg: API keys)
	Globals        skylark.StringDict
}

ExecOpts defines options for exection