interpreter

Documentation

Overview

Package interpreter contains customized Starlark interpreter.

It is an opinionated wrapper around the basic single-file interpreter provided by go.starlark.net library. It implements 'load' and a new built-in 'exec' in a specific way to support running Starlark programs that consist of many files that reference each other, thus allowing decomposing code into small logical chunks and enabling code reuse.

Modules and packages

Two main new concepts are modules and packages. A package is a collection of Starlark files living under the same root. A module is just one such Starlark file. Furthermore, modules can either be "library-like" (executed via 'load' statement) or "script-like" (executed via 'exec' function).

Library-like modules can load other library-like modules via 'load', but may not call 'exec'. Script-like modules may use both 'load' and 'exec'.

Modules within a single package can refer to each other (in 'load' and 'exec') using their relative or absolute (if start with "//") paths.

Packages also have identifiers, though they are local to the interpreter (e.g. not defined anywhere in the package itself). For that reason they are called "package aliases".

Modules from one package may refer to modules from another package by using the following syntax:

load("@<package alias>//<path within the package>", ...)

Presently the mapping between a package alias (e.g. 'stdlib') and package's source code (a collection of *.star files under a single root) is static and supplied by Go code that uses the interpreter. This may change in the future to allow loading packages dynamically, e.g. over the network.

Special packages

There are two special package aliases recognized natively by the interpreter: '__main__' and 'stdlib'.

'__main__' is a conventional name of the package with top level user-supplied code. When printing module paths in stack traces and error message, "@__main__//<path>" are shown as simply "//<path>" to avoid confusing users who don't know what "@<alias>//" might mean. There's no other special handling.

Module '@stdlib//builtins.star' (if present) is loaded before any other code. All its exported symbols that do not start with '_' are made available in the global namespace of all other modules (except ones loaded by '@stdlib//builtins.star' itself). This allows to implement in Starlark built-in global functions exposed by the interpreter.

Exec'ing modules

The built-in function 'exec' can be used to execute a module for its side effects and get its global dict as a return value. It works similarly to 'load', except it doesn't import any symbols into the caller's namespace, and it is not idempotent: calling 'exec' on already executed module is an error.

Each module is executed with its own instance of starlark.Thread. Modules are either loaded as libraries (via 'load' call or LoadModule) or executed as scripts (via 'exec' or ExecModule), but not both. Attempting to load a module that was previous exec'ed (and vice versa) is an error.

Consequently, each Starlark thread created by the nterpreter is either executing some 'load' or some 'exec'. This distinction is available to builtins through GetThreadKind function. Some builtins may check it. For example, a builtin that mutates a global state may return an error when it is called from a 'load' thread.

Dicts of modules loaded via 'load' are reused, e.g. if two different scripts load the exact same module, they'll get the exact same symbols as a result. The loaded code always executes only once. The interpreter MAY load modules in parallel in the future, libraries must not rely on their loading order and must not have side effects.

On the other hand, modules executed via 'exec' are guaranteed to be executed sequentially, and only once. Thus 'exec'-ed scripts essentially form a tree, traversed exactly once in the depth first order.

Built-in symbols

In addition to 'exec' builtin implemented by the interpreter itself, users of the interpreter can also supply arbitrary "predeclared" symbols they want to be available in the global scope of all modules. Predeclared symbols and '@stdlib//builtins.star' module explained above, are the primary mechanisms of making the interpreter do something useful.

Index

• Constants
• Variables
• func Context(th *starlark.Thread) context.Context
• type Interpreter
  • func GetThreadInterpreter(th *starlark.Thread) *Interpreter
  • func (intr *Interpreter) ExecModule(ctx context.Context, pkg, path string) (starlark.StringDict, error)
  • func (intr *Interpreter) Init(ctx context.Context) error
  • func (intr *Interpreter) LoadModule(ctx context.Context, pkg, path string) (starlark.StringDict, error)
  • func (intr *Interpreter) LoadSource(th *starlark.Thread, ref string) (string, error)
  • func (intr *Interpreter) Thread(ctx context.Context) *starlark.Thread
  • func (intr *Interpreter) Visited() []ModuleKey
• type Loader
  • func FSLoader(fsys fs.FS) Loader
  • func FileSystemLoader(root string) Loader
  • func MemoryLoader(files map[string]string) Loader
• type ModuleKey
  • func GetThreadModuleKey(th *starlark.Thread) *ModuleKey
  • func MakeModuleKey(th *starlark.Thread, ref string) (key ModuleKey, err error)
  • func (key ModuleKey) String() string
• type ThreadKind
  • func GetThreadKind(th *starlark.Thread) ThreadKind

Constants

const (
	// MainPkg is an alias of the package with user-supplied code.
	MainPkg = "__main__"
	// StdlibPkg is an alias of the package with the standard library.
	StdlibPkg = "stdlib"
)

Variables

var (
	// ErrNoModule is returned by loaders when they can't find a source code of
	// the requested module. It is also returned by LoadModule when it can't find
	// a module within an existing package.
	ErrNoModule = errors.New("no such module")

	// ErrNoPackage is returned by LoadModule when it can't find a package.
	ErrNoPackage = errors.New("no such package")
)

Functions

func Context

func Context(th *starlark.Thread) context.Context

Context returns a context of the thread created through Interpreter.

Panics if the Starlark thread wasn't created through Interpreter.Thread().

Types

type Interpreter

type Interpreter struct {
	// Predeclared is a dict with predeclared symbols that are available globally.
	//
	// They are available when loading stdlib and when executing user modules. Can
	// be used to extend the interpreter with native Go functions.
	Predeclared starlark.StringDict

	// Packages is a mapping from a package alias to a loader with package code.
	//
	// Users of Interpreter are expected to supply a loader for at least __main__
	// package.
	Packages map[string]Loader

	// Logger is called by Starlark's print(...) function.
	//
	// The callback takes the position in the starlark source code where
	// print(...) was called and the message it received.
	//
	// The default implementation just prints the message to stderr.
	Logger func(file string, line int, message string)

	// ThreadModifier is called whenever interpreter makes a new starlark.Thread.
	//
	// It can inject additional state into thread locals. Useful when hooking up
	// a thread to starlarktest's reporter in unit tests.
	ThreadModifier func(th *starlark.Thread)

	// PreExec is called before launching code through some 'exec' or ExecModule.
	//
	// It may modify the thread or some other global state in preparation for
	// executing a script.
	//
	// 'load' calls do not trigger PreExec/PostExec hooks.
	PreExec func(th *starlark.Thread, module ModuleKey)

	// PostExec is called after finishing running code through some 'exec' or
	// ExecModule.
	//
	// It is always called, even if the 'exec' failed. May restore the state
	// modified by PreExec. Note that PreExec/PostExec calls can nest (if an
	// 'exec'-ed script calls 'exec' itself).
	//
	// 'load' calls do not trigger PreExec/PostExec hooks.
	PostExec func(th *starlark.Thread, module ModuleKey)
	// contains filtered or unexported fields
}

Interpreter knows how to execute starlark modules that can load or execute other starlark modules.

func GetThreadInterpreter

func GetThreadInterpreter(th *starlark.Thread) *Interpreter

GetThreadInterpreter returns Interpreter that created the Starlark thread.

Panics if the Starlark thread wasn't created through Interpreter.Thread().

func (*Interpreter) ExecModule

func (intr *Interpreter) ExecModule(ctx context.Context, pkg, path string) (starlark.StringDict, error)

ExecModule finds and executes a starlark module, returning its dict.

This is similar to exec(...) statement: always executes the module.

'pkg' is a package alias, it will be used to lookup the package loader in intr.Packages. 'path' is a module path (without leading '//') within the package.

The context ends up available to builtins through Context(...).

func (*Interpreter) Init

func (intr *Interpreter) Init(ctx context.Context) error

Init initializes the interpreter and loads '@stdlib//builtins.star'.

Registers whatever was passed via Predeclared plus 'exec'. Then loads '@stdlib//builtins.star', which may define more symbols or override already defined ones. Whatever symbols not starting with '_' end up in the global dict of '@stdlib//builtins.star' module will become available as global symbols in all modules.

The context ends up available to builtins through Context(...).

func (*Interpreter) LoadModule

func (intr *Interpreter) LoadModule(ctx context.Context, pkg, path string) (starlark.StringDict, error)

LoadModule finds and loads a starlark module, returning its dict.

This is similar to load(...) statement: caches the result of the execution. Modules are always loaded only once.

'pkg' is a package alias, it will be used to lookup the package loader in intr.Packages. 'path' is a module path (without leading '//') within the package.

The context ends up available to builtins through Context(...).

func (*Interpreter) LoadSource

func (intr *Interpreter) LoadSource(th *starlark.Thread, ref string) (string, error)

LoadSource returns a body of a file inside a package.

It doesn't have to be a Starlark file, can be any text file as long as the corresponding package Loader can find it.

'ref' is either an absolute reference to the file, in the same format as accepted by 'load' and 'exec' (i.e. "[@pkg]//path"), or a path relative to the currently executing module (i.e. just "path").

Only Starlark threads started via LoadModule or ExecModule can be used with this function. Other threads don't have enough context to resolve paths correctly.

func (*Interpreter) Thread

func (intr *Interpreter) Thread(ctx context.Context) *starlark.Thread

Thread creates a new Starlark thread associated with the given context.

Thread() can be used, for example, to invoke callbacks registered by the loaded Starlark code.

The context ends up available to builtins through Context(...).

The returned thread has no implementation of load(...) or exec(...). Use LoadModule or ExecModule to load top-level Starlark code instead. Note that load(...) statements are forbidden inside Starlark functions anyway.

func (*Interpreter) Visited

func (intr *Interpreter) Visited() []ModuleKey

Visited returns a list of modules visited by the interpreter.

Includes both loaded and execed modules, successfully or not.

type Loader

type Loader func(path string) (dict starlark.StringDict, src string, err error)

Loader knows how to load modules of some concrete package.

It takes a module path relative to the package and returns either module's dict (e.g. for go native modules) or module's source code, to be interpreted.

Returns ErrNoModule if there's no such module in the package.

The source code is returned as 'string' to guarantee immutability and allowing efficient use of string Go constants.

func FSLoader

func FSLoader(fsys fs.FS) Loader

FSLoader returns a loader that loads files from a fs.FS implementation.

func FileSystemLoader

func FileSystemLoader(root string) Loader

FileSystemLoader returns a loader that loads files from the file system.

func MemoryLoader

func MemoryLoader(files map[string]string) Loader

MemoryLoader returns a loader that loads files from the given map.

Useful together with 'assets' package to embed Starlark code into the interpreter binary.

type ModuleKey

type ModuleKey struct {
	Package string // a package name, e.g. "stdlib"
	Path    string // path within the package, e.g. "abc/script.star"
}

ModuleKey is a key of a module within a cache of loaded modules.

It identifies a package and a file within the package.

func GetThreadModuleKey

func GetThreadModuleKey(th *starlark.Thread) *ModuleKey

GetThreadModuleKey returns a ModuleKey with the location of the module being processed by a current load(...) or exec(...) statement.

It has no relation to the module that holds the top-level stack frame. For example, if a currently loading module 'A' calls a function in module 'B' and this function calls GetThreadModuleKey, it will see module 'A' as the result, even though the call goes through code in module 'B'.

Returns nil if the current thread is not executing any load(...) or exec(...), i.e. it has ThreadUnknown kind.

func MakeModuleKey

func MakeModuleKey(th *starlark.Thread, ref string) (key ModuleKey, err error)

MakeModuleKey takes '[@pkg]//<path>' or '<path>', parses and normalizes it.

Converts the path to be relative to the package root. Does some light validation, in particular checking the resulting path doesn't start with '../'. Module loaders are expected to validate module paths more rigorously (since they interpret them anyway).

'th' is used to get the name of the currently executing package and a path to the currently executing module within it. It is required if 'ref' is not given as an absolute path (i.e. does NOT look like '@pkg//path').

func (ModuleKey) String

func (key ModuleKey) String() string

String returns a fully-qualified module name to use in error messages.

If is either "@pkg//path" or just "//path" if pkg is "__main__". We omit the name of the top-level package with user-supplied code ("__main__") to avoid confusing users who are oblivious of packages.

type ThreadKind

type ThreadKind int

ThreadKind is enumeration describing possible uses of a thread.

const (
	// ThreadUnknown indicates a thread used in some custom way, not via
	// LoadModule or ExecModule.
	ThreadUnknown ThreadKind = iota

	// ThreadLoading indicates a thread that is evaluating a module referred to by
	// some load(...) statement.
	ThreadLoading

	// ThreadExecing indicates a thread that is evaluating a module referred to by
	// some exec(...) statement.
	ThreadExecing
)

func GetThreadKind

func GetThreadKind(th *starlark.Thread) ThreadKind

GetThreadKind tells what sort of thread 'th' is: it is either inside some load(...), some exec(...) or some custom call (probably a callback called from native code).

Panics if the Starlark thread wasn't created through Interpreter.Thread().