skycfg

README

Skycfg

Skycfg is an extension library for the Starlark language that adds support for constructing Protocol Buffer messages. It was developed by Stripe to simplify configuration of Kubernetes services, Envoy routes, Terraform resources, and other complex configuration data.

At present, only the Go implementation of Starlark is supported.

Getting Started

The entry point to Skycfg is the skycfg.Load() function, which reads a configuration file from local disk. As the implementation stabilizes we expect to expand the public API surface so that Skycfg can be combined with other Starlark extensions.

Lets start with a simple main function that prints out every Protobuf message created by the config file hello.sky:

package main

import (
    "context"
    "fmt"
    "github.com/stripe/skycfg"
    _ "github.com/golang/protobuf/ptypes/wrappers"
)

func main() {
    ctx := context.Background()
    config, err := skycfg.Load(ctx, "hello.sky")
    if err != nil { panic(err) }
    messages, err := config.Main(ctx)
    if err != nil { panic(err) }
    for _, msg := range messages {
        fmt.Printf("%s\n", msg.String())
    }
}

# hello.sky
pb = proto.package("google.protobuf")

def main(ctx):
  return [pb.StringValue(value = "Hello, world!")]

Now we can build a small test driver and see what values are returned:

$ go get github.com/stripe/skycfg
$ go build -o test-skycfg
$ ./test-skycfg
value:"Hello, world!"
$

Success!

For more in-depth examples covering specific topics, see the _examples/ directory:

• _examples/repl: Interactive evaluation of a Skycfg file
• _examples/k8s: Basic Kubernetes integration
• _examples/wasm: Skycfg in the browser with WebAssembly

Why use Skycfg?

Compared to bare YAML or TOML, the Python-like syntax of Skycfg might not seem like a win. Why would we want configuration files with all those quotes and colons and braces?

There are four important benefits to using Skycfg over YAML:

Type Safety

Protobuf has a statically-typed data model, which means the type of every field is known to Skycfg when it's building your configuration. There is no risk of accidentally assigning a string to a number, a struct to a different struct, or forgetting to quote a YAML value.

pb = proto.package("google.protobuf")

def main(ctx):
  return [pb.StringValue(value = 123)]

$ ./test-skycfg
panic: TypeError: value 123 (type `int') can't be assigned to type `string'.

Functions

As in standard Python, you can define helper functions to reduce duplicated typing and share logic.

pb = proto.package("google.protobuf")

def greet(lang):
  greeting = {
    "en": "Hello, world!",
    "fr": "Bonjour, monde!",
  }[lang]
  return pb.StringValue(value = greeting)

def main(ctx):
  return [greet("en"), greet("fr")]

$ ./test-skycfg
value:"Hello world!"
value:"Bonjour, monde!"

Modules

Starlark supports importing modules from other files, which you can use to share common code between configurations. By default the paths to load are resolved on the local filesystem, but you can also override the load() handler to support syntaxes such as Bazel-style target labels.

Modules can protect service owners from complex Kubernetes logic:

load("//config/common/kubernetes.sky", "kubernetes")

def my_service(ctx):
  return kubernetes.pod(
      name = "my-namespace/my-service",
      containers = [
          kubernetes.container(name = "main", image = "index.docker.io/hello-world"),
      ]
  )

When combined with VCS hooks like GitHub CODEOWNERS, you can use modules to provide an API surface for third-party tools deployed in your infrastructure:

load("//config/common/constants.sky",  "CLUSTERS")
load("//config/production/k8s_dashboard/v1.10.0/main.sky",
     "kubernetes_dashboard")

def main(ctx):
  return [
    kubernetes_dashboard(ctx, cluster = CLUSTERS['ord1']),
    kubernetes_dashboard(ctx, cluster = CLUSTERS['ord1-canary']),
  ]

Context Variables

Skycfg supports limited dynamic behavior through the use of context variables, which let the Go caller pass arbitrary key:value pairs in the ctx parameter.

func main() {
    // ...
    messages, err := config.Main(ctx, skycfg.WithVars(starlark.StringDict{
      "revision": starlark.String("master/12345"),
    }))

pb = proto.package("google.protobuf")

def main(ctx):
  print("ctx.vars:", ctx.vars)
  return []

$ ./test-skycfg
[hello.sky:4] ctx.vars: {"revision": "master/12345"}

Contributing

We welcome contributions from the community. For small simple changes, go ahead and open a pull request. Larger changes should start out in the issue tracker, so we can make sure they fit into the roadmap. Changes to the Starlark language itself (such as new primitive types or syntax) should be applied to https://github.com/google/starlark-go.

Stability

Skycfg depends on internal details of the go-protobuf generated code, and as such may need to be updated to work with future versions of go-protobuf. We will release Skycfg v1.0 after all dependencies on go-protobuf implementation details have been fixed, which will be after the "api-v2" branch lands in a stable release of go-protobuf.

Our existing public APIs are expected to be stable even before the v1.0 release. Symbols that will change before v1.0 are hidden from the public docs and named Unstable*.

Known issues

gogo/protobuf support

Skycfg has dropped support for gogo/protobuf, an alternative protobuf go code generator, when upgrading to using the go-protobuf v2 api. For using gogo support, see Skycfg@v0.1.0 (Skycfg prior to switching to v2).

Documentation

Overview

Package skycfg is an extension library for the Starlark language that adds support for constructing Protocol Buffer messages.

Index

• func AsProtoMessage(v starlark.Value) (proto.Message, bool)
• func NewProtoMessage(msg proto.Message) (starlark.Value, error)
• func NewProtoPackage(r unstableProtoRegistryV2, name string) starlark.Value
• func NewUnstableProtobufRegistryV2(r *protoregistry.Types) unstableProtoRegistryV2
• func UnstablePredeclaredModules(r unstableProtoRegistryV2) starlark.StringDict
• func UnstableProtoModule(r unstableProtoRegistryV2) starlark.Value
• type Config
  • func Load(ctx context.Context, filename string, opts ...LoadOption) (*Config, error)
  • func (c *Config) Filename() string
  • func (c *Config) Globals() starlark.StringDict
  • func (c *Config) Locals() starlark.StringDict
  • func (c *Config) Main(ctx context.Context, opts ...ExecOption) ([]proto.Message, error)
  • func (c *Config) Tests() []*Test
• type ExecOption
  • func WithEntryPoint(name string) ExecOption
  • func WithFlattenLists() ExecOption
  • func WithVars(vars starlark.StringDict) ExecOption
• type FileReader
  • func LocalFileReader(root string) FileReader
• type LoadOption
  • func WithFileReader(r FileReader) LoadOption
  • func WithGlobals(globals starlark.StringDict) LoadOption
  • func WithProtoRegistry(r unstableProtoRegistryV2) LoadOption
• type Test
  • func (t *Test) Name() string
  • func (t *Test) Run(ctx context.Context, opts ...TestOption) (*TestResult, error)
• type TestOption
  • func WithTestVars(vars starlark.StringDict) TestOption
• type TestResult

Functions

func AsProtoMessage

func AsProtoMessage(v starlark.Value) (proto.Message, bool)

AsProtoMessage returns a Protobuf message underlying the given Starlark value, which must have been created by NewProtoMessage(). Returns (_, false) if the value is not a valid message.

func NewProtoMessage

func NewProtoMessage(msg proto.Message) (starlark.Value, error)

NewProtoMessage returns a Starlark value representing the given Protobuf message. It can be returned back to a proto.Message() via AsProtoMessage().

func NewProtoPackage

func NewProtoPackage(r unstableProtoRegistryV2, name string) starlark.Value

NewProtoPackage returns a Starlark value representing the given Protobuf package. It can be added to global symbols when loading a Skycfg config file.

func NewUnstableProtobufRegistryV2

func NewUnstableProtobufRegistryV2(r *protoregistry.Types) unstableProtoRegistryV2

func UnstablePredeclaredModules

func UnstablePredeclaredModules(r unstableProtoRegistryV2) starlark.StringDict

UnstablePredeclaredModules returns a Starlark string dictionary with predeclared Skycfg modules which can be used in starlark.ExecFile.

Takes in unstableProtoRegistry as param (if nil will use standard proto registry).

Currently provides these modules (see REAMDE for more detailed description):

• fail - interrupts execution and prints a stacktrace.
• hash - supports md5, sha1 and sha245 functions.
• json - marshals plain values (dicts, lists, etc) to JSON.
• proto - package for constructing Protobuf messages.
• struct - experimental Starlark struct support.
• yaml - same as "json" package but for YAML.
• url - utility package for encoding URL query string.

func UnstableProtoModule

func UnstableProtoModule(r unstableProtoRegistryV2) starlark.Value

Types

type Config

type Config struct {
	// contains filtered or unexported fields
}

A Config is a Skycfg config file that has been fully loaded and is ready for execution.

func Load

func Load(ctx context.Context, filename string, opts ...LoadOption) (*Config, error)

Load reads a Skycfg config file from the filesystem.

func (*Config) Filename

func (c *Config) Filename() string

Filename returns the original filename passed to Load().

func (*Config) Globals

func (c *Config) Globals() starlark.StringDict

Globals returns the set of variables in the Starlark global namespace, including any added to the config loader by WithGlobals().

func (*Config) Locals

func (c *Config) Locals() starlark.StringDict

Locals returns the set of variables in the Starlark local namespace for the top-level module.

func (*Config) Main

func (c *Config) Main(ctx context.Context, opts ...ExecOption) ([]proto.Message, error)

Main executes main() or a custom entry point function from the top-level Skycfg config module, which is expected to return either None or a list of Protobuf messages.

func (*Config) Tests

func (c *Config) Tests() []*Test

Tests returns all tests defined in the config

type ExecOption

type ExecOption interface {
	// contains filtered or unexported methods
}

An ExecOption adjusts details of how a Skycfg config's main function is executed.

func WithEntryPoint

func WithEntryPoint(name string) ExecOption

WithEntryPoint changes the name of the Skycfg function to execute.

func WithFlattenLists

func WithFlattenLists() ExecOption

WithFlattenLists flatten lists one layer deep ([1, [2,3]] -> [1, 2, 3])

func WithVars

func WithVars(vars starlark.StringDict) ExecOption

WithVars adds key:value pairs to the ctx.vars dict passed to main().

type FileReader

type FileReader interface {
	// Resolve parses the "name" part of load("name", "symbol") to a path. This
	// is not required to correspond to a true path on the filesystem, but should
	// be "absolute" within the semantics of this FileReader.
	//
	// fromPath will be empty when loading the root module passed to Load().
	Resolve(ctx context.Context, name, fromPath string) (path string, err error)

	// ReadFile reads the content of the file at the given path, which was
	// returned from Resolve().
	ReadFile(ctx context.Context, path string) ([]byte, error)
}

A FileReader controls how load() calls resolve and read other modules.

func LocalFileReader

func LocalFileReader(root string) FileReader

LocalFileReader returns a FileReader that resolves and loads files from within a given filesystem directory.

type LoadOption

type LoadOption interface {
	// contains filtered or unexported methods
}

A LoadOption adjusts details of how Skycfg configs are loaded.

func WithFileReader

func WithFileReader(r FileReader) LoadOption

WithFileReader changes the implementation of load() when loading a Skycfg config.

func WithGlobals

func WithGlobals(globals starlark.StringDict) LoadOption

WithGlobals adds additional global symbols to the Starlark environment when loading a Skycfg config.

func WithProtoRegistry

func WithProtoRegistry(r unstableProtoRegistryV2) LoadOption

WithProtoRegistry is an EXPERIMENTAL and UNSTABLE option to override how Protobuf message type names are mapped to Go types.

type Test

type Test struct {
	// contains filtered or unexported fields
}

A Test is a test case, which is a skycfg function whose name starts with `test_`.

func (*Test) Name

func (t *Test) Name() string

Name returns the name of the test (the name of the function)

func (*Test) Run

func (t *Test) Run(ctx context.Context, opts ...TestOption) (*TestResult, error)

Run actually executes a test. It returns a TestResult if the test completes (even if it fails) The error return value will only be non-nil if the test execution itself errors.

type TestOption

type TestOption interface {
	// contains filtered or unexported methods
}

An TestOption adjusts details of how a Skycfg config's test functions are executed.

func WithTestVars

func WithTestVars(vars starlark.StringDict) TestOption

WithTestVars adds key:value pairs to the ctx.vars dict passed to tests

type TestResult

type TestResult struct {
	TestName string
	Failure  error
	Duration time.Duration
}

A TestResult is the result of a test run