scribe

package module
v0.9.10 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 17, 2022 License: Apache-2.0 Imports: 23 Imported by: 0

README

scribe

Scribe is a framework to write flexible CI pipelines in Go that have consistent behavior when ran locally or in a CI server.

Status

This is still in beta. Expect breaking changes.

Why?

Continuous integration and delivery are built on flaky foundations.

Configuration-based pipeline tooling has been prevalent all over open source software. It works in all languages, it's mostly platform agnostic, and offers flexibility with bash. However, it does not scale with several people, and its problems are seen quickly.

  • Development and testing strictly involves trial and error.
  • Automated testing is non-existent.
  • Development tools are limited to defining configuration schemas, which provide a shallow understanding of what each key and value will do.
  • Configuration languages have nuanced syntax which is typically a lot different than standard programming languages.
  • Dependencies / dependency management is often not supported as it's just not a typical configuration language (yaml, json) construct.
  • Lack of debugging means large / extensive pipelines are flaky and make it difficult to diagnose issues.
    • Problems like these are demoralizing and often lead to neglect; no one wants to address issues like these because they're so difficult to debug.
  • Providers are incentivized to keep their YAML from being ran on other platforms. Many providers probably not like it if I could run a pipeline made for their platform inside a competitor's.

These problems lead to the development of this framework, scribe.

The idea behind scribe is that it is not an application, but a library. There is no server. Users should, instead of defining an amalgamation of yaml/json/toml/whatever and bash, define their build, package, and release processes programmatically. This opens up a whole world of possibilities, like:

  • Writing unit and integration tests for your build pipeline.
  • Reusing and sharing build, package, and deployment definitions.
  • Creating reasonable packages and libraries for developing new pipelines.
  • Improved visualization by allowing pipelines to define metrics and traces.
  • Improved validation by having steps define what they expect in order to run, and what they provide to other steps.

Glossary

  • Pipeline: A pipeline is a generic sequence of steps. A pipeline can be a set of steps to build an application, or it can define how to take an artifact, package it, and push it to a package repository.
    • Action: A pipeline action is a single reusable component in a pipeline. Actions can have arguments and define outputs.
    • Source: A pipeline source defines what causes a pipeline to begin. For typical continuous integration builds, this source would be a commit or a tag. For a release pipeline, this could be a NATS message, a Google Cloud Pub/Sub message that says an artifact is available, or it could be another pipeline.
  • Artifact: The tangible, end-result of a pipeline or step. Not all pipelines produce artifacts.
    • An example of an artifact would be a compiled binary or a docker image.
  • Target: A target is a software release destination. It is the final place that an artifact is sent before it is used to serve user requests.

Running Locally / testing

  • Compile the Scribe utility: mage build
  • Run the local pipeline in the current shell: ./bin/scribe-mode=cli ./ci
  • Run the local pipeline in docker: ./bin/scribe -mode=docker ./ci
  • Generate the drone: ./bin/scribe -mode=drone ./ci
  • Generate the drone and write it to a file: ./bin/scribe -mode=drone ./ci > .drone.yml

How does it work?

The main idea behind scribe is that it defers what is typically considered server logic into the client / pipeline definitions and library.

Clients

Scribe clients are used in the pipelines themselves. All pipelines are programs, and thus have a func main().

There are currently three planned Clients, which are selected using the -mode CLI argument.

  1. cli - Runs the pipeline locally, attempting to emulate what would normally be executed in a standard CI pipeline.
  2. drone - Generates a Drone configuration that matches the specified pipeline.
  3. docker - Similar to run, but instead runs the pipeline using the Docker CLI
Run mode docker

The docker run mode will run each pipeline in a Docker image the same way that it would likely run in a CI platform. It's almost a combination of the cli and drone run modes.

Each step defined must have an image. For steps without defined images, the scribe will be used.

When running in docker mode, the pipeline is compiled and then mounted as volume in the docker container. The compiled pipeline is used as the docker command for that step.

Writing a Pipeline

  1. Every pipeline is a program and should have a package main and a func main.

  2. Every pipeline must have a form of sw.New(...) or sw.NewMulti(...) to produce the scribe object.

    • Steps are then added to that object to create a pipeline.

  3. It is recommended to create a Go workspace for your CI pipeline with go work init {directory}.

    • This will keep the larger and irrelevant modules like docker out of your project.
Examples

To view examples of pipelines, visit the demo folder. These demos are used in our automated tests.

FAQ

  • Why use Go and not JavaScript/TypeScript/Python/Java?

We use Go pretty ubiquitously at Grafana, especially in our server code. Go also allows you to easily compile a static binary for Linux from any platform which helps a lot with the portability of Scribe, especially in Docker mode.

  • Will there be support for any other languages?

Given the current design, it would be very difficult and there are no concrete plans to do that yet.

  • What clients are available?

  • cli, which runs the pipeline in the current shell.

  • docker, which runs the pipeline using the docker daemon (configured via the Docker environment variables).

  • drone, which produces a .drone.yml file in the standard output stream (stdout) that will run the pipeline in Drone.

The current list of clients can always be obtained using the scribe -help command.

  • How can I use unsupported clients or make my own?

Because Scribe is simply a package and your pipeline is a program, you can add a client you have made yourself in your pipeline.

In the init function of the pipeline, simply register your client and it should be available for use. For a demonstration, see ./demo/custom-client.

Documentation

Overview

Package scribe provides the primary library / client functions, types, and methods for creating Scribe pipelines.

Index

Constants

View Source 
const DefaultPipelineID int64 = 1

Variables

View Source 
var (
	// ClientCLI is set when a pipeline is ran from the Scribe CLI, typically for local development, but can also be set when running Scribe within a third-party service like CircleCI or Drone
	ClientCLI string = "cli"

	// ClientDrone is set when a pipeline is ran in Drone mode, which is used to generate a Drone config from a Scribe pipeline
	ClientDrone         = "drone"
	ClientDroneStarlark = "drone-starlark"

	// RunModeDocker runs the pipeline using the Docker CLI for each step
	ClientDocker = "docker"
)
View Source 
var ClientInitializers = map[string]InitializerFunc{
	ClientCLI:           NewCLIClient,
	ClientDrone:         NewDroneClient,
	ClientDroneStarlark: NewDroneStarlarkClient,
	ClientDocker:        NewDockerClient,
}

The ClientInitializers define how different RunModes initialize the Scribe client

View Source 
var ErrorCancelled = errors.New("cancelled")

Functions

func GetState 

func GetState(val string, log logrus.FieldLogger, args *plumbing.PipelineArgs) (*pipeline.State, error)

func NewCLIClient 

func NewCLIClient(opts pipeline.CommonOpts) pipeline.Client

func NewDefaultCollection 

func NewDefaultCollection(opts pipeline.CommonOpts) *pipeline.Collection

func NewDockerClient 

func NewDockerClient(opts pipeline.CommonOpts) pipeline.Client

func NewDroneClient 

func NewDroneClient(opts pipeline.CommonOpts) pipeline.Client

func NewDroneStarlarkClient 

func NewDroneStarlarkClient(opts pipeline.CommonOpts) pipeline.Client

func NewMultiCollection 

func NewMultiCollection() *pipeline.Collection

func RegisterClient 

func RegisterClient(name string, initializer InitializerFunc)

Types

type GitConfig 

type GitConfig struct{}

type InitializerFunc 

type InitializerFunc func(pipeline.CommonOpts) pipeline.Client

type MultiFunc 

type MultiFunc func(*Scribe)

func MultiFuncWithLogging 

func MultiFuncWithLogging(logger logrus.FieldLogger, mf MultiFunc) MultiFunc

type MultiSubFunc 

type MultiSubFunc func(*ScribeMulti)

type PipelineConfig 

type PipelineConfig struct{}

Config defines the typical options that are provided in a pipeline. These options can be provided many ways, and often depend on the execution environment. They are retrieved at pipeline-time.

type Scribe 

type Scribe struct {
	Client     pipeline.Client
	Collection *pipeline.Collection

	// Opts are the options that are provided to the pipeline from outside sources. This includes mostly command-line arguments and environment variables
	Opts    pipeline.CommonOpts
	Log     logrus.FieldLogger
	Version string
	// contains filtered or unexported fields
}

Scribe is the client that is used in every pipeline to declare the steps that make up a pipeline. The Scribe type is not thread safe. Running any of the functions from this type concurrently may have unexpected results.

func New 

func New(name string) *Scribe

New creates a new Scribe client which is used to create pipeline a single pipeline with many steps. This function will panic if the arguments in os.Args do not match what's expected. This function, and the type it returns, are only ran inside of a Scribe pipeline, and so it is okay to treat this like it is the entrypoint of a command. Watching for signals, parsing command line arguments, and panics are all things that are OK in this function. New is used when creating a single pipeline. In order to create multiple pipelines, use the NewMulti function.

func NewClient 

func NewClient(c pipeline.CommonOpts, collection *pipeline.Collection) *Scribe

NewClient creates a new Scribe client based on the commonopts (mostly the mode). It does not check for a non-nil "Args" field.

func NewWithClient 

func NewWithClient(opts pipeline.CommonOpts, client pipeline.Client) *Scribe

NewWithClient creates a new Scribe object with a specific client implementation. This function is intended to be used in very specific environments, like in tests.

func (*Scribe) Background 

func (s *Scribe) Background(steps ...pipeline.Step)

Background allows users to define steps that run in the background. In some environments this is referred to as a "Service" or "Background service". In many scenarios, users would like to simply use a docker image with the default command. In order to accomplish that, simply provide a step without an action.

func (*Scribe) Cache 

func (s *Scribe) Cache(action pipeline.Action, c pipeline.Cacher) pipeline.Action

func (*Scribe) Done 

func (s *Scribe) Done()

func (*Scribe) Execute 

func (s *Scribe) Execute(ctx context.Context, collection *pipeline.Collection) error

Execute is the equivalent of Done, but returns an error. Done should be preferred in Scribe pipelines as it includes sub-process handling and logging.

func (*Scribe) Parallel 

func (s *Scribe) Parallel(steps ...pipeline.Step)

Parallel will run the listed steps at the same time. This function blocks the pipeline execution until all of the steps have completed.

func (*Scribe) Pipeline 

func (s *Scribe) Pipeline() int64

Pipeline returns the current Pipeline ID used in the collection.

func (*Scribe) Run 

func (s *Scribe) Run(steps ...pipeline.Step)

Run allows users to define steps that are ran sequentially. For example, the second step will not run until the first step has completed. This function blocks the pipeline execution until all of the steps provided (step) have completed sequentially.

func (*Scribe) Sub 

func (s *Scribe) Sub(sf SubFunc)

Sub creates a sub-pipeline. The sub-pipeline is equivalent to creating a new coroutine made of multiple steps. This sub-pipeline will run concurrently with the rest of the pipeline at the time of definition. Under the hood, the Scribe client creates a new Scribe object with a clean Collection, then calles the SubFunc (sf) with the new Scribe object. The collection is then populated by the SubFunc, and then appended to the existing collection.

func (*Scribe) When 

func (s *Scribe) When(events ...pipeline.Event)

When allows users to define when this pipeline is executed, especially in the remote environment.

type ScribeMulti 

type ScribeMulti struct {
	Client     pipeline.Client
	Collection *pipeline.Collection

	// Opts are the options that are provided to the pipeline from outside sources. This includes mostly command-line arguments and environment variables
	Opts    pipeline.CommonOpts
	Log     logrus.FieldLogger
	Version string
	// contains filtered or unexported fields
}

func NewMulti 

func NewMulti() *ScribeMulti

NewMulti is the equivalent of `scribe.New`, but for building a pipeline made of multiple pipelines. Pipelines can behave in the same way that a step does. They can be ran in parallel using the Parallel function, or ran in a series using the Run function. To add new pipelines to execution, use the `(*scribe.ScribeMulti).New(...)` function.

func NewMultiWithClient 

func NewMultiWithClient(opts pipeline.CommonOpts, client pipeline.Client) *ScribeMulti

func (*ScribeMulti) Done 

func (s *ScribeMulti) Done()

func (*ScribeMulti) Execute 

func (s *ScribeMulti) Execute(ctx context.Context, collection *pipeline.Collection) error

Execute is the equivalent of Done, but returns an error. Done should be preferred in Scribe pipelines as it includes sub-process handling and logging.

func (*ScribeMulti) New 

func (s *ScribeMulti) New(name string, mf MultiFunc) pipeline.Pipeline

New creates a new Pipeline step that executes the provided MultiFunc onto a new `*Scribe` type, creating a DAG. Because this function returns a pipeline.Step[T], it can be used with the normal Scribe functions like `Run` and `Parallel`.

func (*ScribeMulti) Parallel 

func (s *ScribeMulti) Parallel(steps ...pipeline.Pipeline)

func (*ScribeMulti) Run 

func (s *ScribeMulti) Run(steps ...pipeline.Pipeline)

func (*ScribeMulti) Sub 

func (s *ScribeMulti) Sub(sf MultiSubFunc)

type SubFunc 

type SubFunc func(*Scribe)

SubFunc should use the provided Scribe object to populate a pipeline that runs independently.

Source Files

View all Source files

Directories

Path Synopsis
ci
docker
Package docker contains docker images and metadata for the suite of scribe docker images and utilities
 Package docker contains docker images and metadata for the suite of scribe docker images and utilities
demo
basic
complex
custom-client
multi
multi-sub
state
sub
Package exec provides helper functions and Actions for executing shell commands.
 Package exec provides helper functions and Actions for executing shell commands.
git
x
x
Package plumbing contains the internal scribe command utilities.
 Package plumbing contains the internal scribe command utilities.
cmd
Package main contains the logic for the `scribe` CLI Package main contains the CLI logic for the `scribe` command The scribe command's main responsibility is to run a pipeline.
 Package main contains the logic for the `scribe` CLI Package main contains the CLI logic for the `scribe` command The scribe command's main responsibility is to run a pipeline.
cmd/commands
cmdutil
Package cmdutil provides utility functions and types for working with the 'scribe' CLI.
 Package cmdutil provides utility functions and types for working with the 'scribe' CLI.
pipeline
Package pipeline contains the meta types and interfaces that define a Scribe pipeline.
 Package pipeline contains the meta types and interfaces that define a Scribe pipeline.
pipeline/clients/cli
pipeline/clients/common
pipeline/clients/docker
pipeline/clients/drone
Package drone contians the drone client implementation for generating a Drone pipeline config.
 Package drone contians the drone client implementation for generating a Drone pipeline config.
pipeline/clients/drone/starlark
pipeline/dag
pipelineutil
Package pipelineutil defines utilities for working with Pipelines and is separated as it may also import packages that import pipeline.
 Package pipelineutil defines utilities for working with Pipelines and is separated as it may also import packages that import pipeline.
plog
Package plog (or plumbig log) provides a logging initializer and utility functions for working with a logging library.
 Package plog (or plumbig log) provides a logging initializer and utility functions for working with a logging library.
stringutil
Package stringutil contains general string utilities used throughout this project.
 Package stringutil contains general string utilities used throughout this project.
syncutil
Package syncutil provides utilities for working with asynchronous tasks and provides wrappers around the "sync" package.
 Package syncutil provides utilities for working with asynchronous tasks and provides wrappers around the "sync" package.
tarfs
testutil
wrappers

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.