flow

package module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 12, 2023 License: MIT Imports: 27 Imported by: 0

README

flow

Flow is a framework for Dagger for writing flexible CI pipelines in Go that have consistent behavior when ran locally or in a CI server.

Write your pipeline once, run it locally and produce the config for your CI provider from the same code.

Status

This is still in beta. Expect breaking changes and incomplete features.

Why?

With Flow you can:

  • Run pipelines locally for testing using Dagger.
  • Generate configurations for existing CI providers.
  • Use tools like delve to debug your pipelines.
  • Use Go features to make complex pipelines easier to develop and maintain.

Why not only use Dagger?

Flow internally uses Dagger to accomplish consistency. The promise of running the same thing locally that's ran in a CI service could not happen without it.

Flow adds a few features on top of Dagger, like:

  • Executing an anonymous function instead of a command.
  • Generating a CI configuration from your pipeline code.

Running Locally / testing

Note: For examples, please see the demo folder.

With the flow CLI
Compile the Flow utility mage build
Run the local pipeline with Dagger ./bin/flow ./ci
Generate the drone ./bin/flow -client=gitlab ./ci
Generate the drone and write it to a file ./bin/flow -client=gitlab ./ci > .gitlab.yml
Without the flow CLI
Run the local pipeline with Dagger go run ./ci
Generate the drone go run ./ci -client=drone
Generate the drone and write it to a file go run ./ci -client=gitlab > .gitlab.yml

How?

flow does not create pipelines using templating. It uses pipeline definitions as a compilation target. Rather than templating a YAML file, flow will create one that best represents the pipeline you've defined.

Tips for writing a Pipeline

  1. Every pipeline is a program and must have a package main and a func main.
  2. Every pipeline must have a form of pipeline := flow.New(...) or pipeline := flow.NewMulti(...) to produce the flow object.
    • Steps are then added to that object to create a pipeline.
  3. Every pipeline must conclude with pipeline.Done()
  4. It is recommended to create a Go workspace for your CI pipeline with go work init {directory}.
    • This will keep the larger and irrelevant modules out of your project.
Examples

To view examples of pipelines, visit the demo folder. These demos are used in our automated tests.

FAQ

  • Why use Go and not JavaScript/TypeScript/Python/Java?

We use Go pretty ubiquitously at Grafana, especially in our server code. Go also allows you to easily compile a static binary for Linux from any platform which helps a lot with the portability of Flow, especially with the Dagger client.

  • Will there be support for any other languages?

Given the current design, it would be very difficult and there are no concrete plans to do that yet.

  • What clients are available?

  • dagger, which runs the pipeline using Dagger. Dagger allows us to reproducibly run the pipeline using Docker BuildKit and Docker containers. This is the recommended way to run pipelines locally.

  • drone, which produces a .drone.yml file in the standard output stream (stdout) that will run the pipeline in Drone.

  • cli, which runs the pipeline in the current shell. This mode is not recommended to be used outside of a docker container.

The current list of clients can always be obtained using the flow --help command.

  • How can I use unsupported clients or make my own?

Because Flow is simply a package and your pipeline is a program, you can add a client you have made yourself in your pipeline.

In the init function of the pipeline, simply register your client and it should be available for use. For a demonstration, see ./demo/custom-client.

  • What features are currently available and what's planned for the future?

Take a look at the issues and milestones to get an idea. The demo folder is a good place to see what's currently available for you to use.

Documentation

Overview

Package flow provides the primary library / client functions, types, and methods for creating Flow pipelines.

Index

Constants

View Source 
const DefaultPipelineID int64 = 1

Variables

View Source 
var (
	ClientCLI      string = "cli"
	ClientDrone           = "drone"
	ClientDagger          = "dagger"
	ClientGraphviz        = "graphviz"
)
View Source 
var ArgDefaults = map[string]func(context.Context) *exec.Cmd{
	pipeline.ArgumentCommitRef.Key: func(ctx context.Context) *exec.Cmd {
		return exec.CommandContext(ctx, "git", "rev-parse", "HEAD")
	},
	pipeline.ArgumentCommitSHA.Key: func(ctx context.Context) *exec.Cmd {
		return exec.CommandContext(ctx, "git", "rev-parse", "HEAD")
	},
	pipeline.ArgumentRemoteURL.Key: func(ctx context.Context) *exec.Cmd {
		return exec.CommandContext(ctx, "git", "remote", "get-url", "origin")
	},
	pipeline.ArgumentBranch.Key: func(ctx context.Context) *exec.Cmd {
		return exec.CommandContext(ctx, "git", "rev-parse", "--abbrev-ref", "HEAD")
	},
	pipeline.ArgumentTagName.Key: func(ctx context.Context) *exec.Cmd {
		return exec.CommandContext(ctx, "git", "describe", "--tags")
	},
}
View Source 
var ClientInitializers = map[string]InitializerFunc{
	ClientCLI:      cli.New,
	ClientDrone:    drone.New,
	ClientDagger:   dagger.New,
	ClientGraphviz: graphviz.New,
}

The ClientInitializers define how different RunModes initialize the Flow client

View Source 
var ErrorCancelled = errors.New("cancelled")
View Source 
var LocalModes = []string{"dagger"}

LocalClients define modes that are intended to run a pipeline "locally". These local clients will do things like filter the pipeline based on the selected event with the '-e' flag.

Functions

func NewDefaultCollection 

func NewDefaultCollection(opts clients.CommonOpts) *pipeline.Collection

func NewMultiCollection 

func NewMultiCollection() *pipeline.Collection

func RegisterClient 

func RegisterClient(name string, initializer InitializerFunc)

Types

type Flow 

type Flow struct {
	Client     pipeline.Client
	Collection *pipeline.Collection

	// Opts are the options that are provided to the pipeline from outside sources. This includes mostly command-line arguments and environment variables
	Opts    clients.CommonOpts
	Log     logrus.FieldLogger
	Version string
	// contains filtered or unexported fields
}

Flow is the client that is used in every pipeline to declare the steps that make up a pipeline. The Flow type is not thread safe. Running any of the functions from this type concurrently may have unexpected results.

func New 

func New(name string) *Flow

New creates a new Flow client which is used to create pipeline a single pipeline with many steps. This function will panic if the arguments in os.Args do not match what's expected. This function, and the type it returns, are only ran inside of a Flow pipeline, and so it is okay to treat this like it is the entrypoint of a command. Watching for signals, parsing command line arguments, and panics are all things that are OK in this function. New is used when creating a single pipeline. In order to create multiple pipelines, use the NewMulti function.

func NewClient 

func NewClient(ctx context.Context, c clients.CommonOpts, collection *pipeline.Collection) *Flow

NewClient creates a new Flow client based on the commonopts. It does not check for a non-nil "Args" field.

func NewWithClient 

func NewWithClient(opts clients.CommonOpts, client pipeline.Client) *Flow

NewWithClient creates a new Flow object with a specific client implementation. This function is intended to be used in very specific environments, like in tests.

func (*Flow) Add 

func (s *Flow) Add(steps ...pipeline.Step)

Add allows users to define steps. The order in which steps are ran is defined by what they provide / require. Some steps do not produce anything, like for example running a suite of tests for a pass/fail result.

func (*Flow) Background 

func (s *Flow) Background(steps ...pipeline.Step)

Background allows users to define steps that run in the background. In some environments this is referred to as a "Service" or "Background service". In many scenarios, users would like to simply use a docker image with the default command. In order to accomplish that, simply provide a step without an action.

func (*Flow) Cache 

func (s *Flow) Cache(action pipeline.Action, c pipeline.Cacher) pipeline.Action

func (*Flow) Done 

func (s *Flow) Done()

func (*Flow) Execute 

func (s *Flow) Execute(ctx context.Context, collection *pipeline.Collection) error

Execute is the equivalent of Done, but returns an error. Done should be preferred in Flow pipelines as it includes sub-process handling and logging.

func (*Flow) Pipeline 

func (s *Flow) Pipeline() int64

Pipeline returns the current Pipeline ID used in the collection.

func (*Flow) When 

func (s *Flow) When(events ...pipeline.Event)

When allows users to define when this pipeline is executed, especially in the remote environment. Users can execute the pipeline as if it was triggered from the event by supplying the `-e` or `--event` argument. This function will overwrite any other events that were added to the pipeline.

type FlowMulti 

type FlowMulti struct {
	Client     pipeline.Client
	Collection *pipeline.Collection

	// Opts are the options that are provided to the pipeline from outside sources. This includes mostly command-line arguments and environment variables
	Opts    clients.CommonOpts
	Log     logrus.FieldLogger
	Version string
	// contains filtered or unexported fields
}

func NewMulti 

func NewMulti() *FlowMulti

NewMulti is the equivalent of `Flow.New`, but for building a pipeline made of multiple pipelines. Pipelines can behave in the same way that a step does. They can be ran in parallel using the Parallel function, or ran in a series using the Run function. To add new pipelines to execution, use the `(*flow.FlowMulti).New(...)` function.

func NewMultiWithClient 

func NewMultiWithClient(opts clients.CommonOpts, client pipeline.Client) *FlowMulti

func (*FlowMulti) Add 

func (s *FlowMulti) Add(pipelines ...pipeline.Pipeline)

Add adds new pipelines to the Flow DAG to be processed by the Client.

func (*FlowMulti) AddPipelines 

func (s *FlowMulti) AddPipelines(pipelines ...Pipeline)

AddPipelines adds a list of pipelines into the DAG. The order in which they are defined or added is not important; the order in which they run depends on what they require and what they provide. This function can be ran multiple times; every new item added with 'AddPipelines' will be appended to the dag.

func (*FlowMulti) Done 

func (s *FlowMulti) Done()

func (*FlowMulti) Execute 

func (s *FlowMulti) Execute(ctx context.Context, collection *pipeline.Collection) error

Execute is the equivalent of Done, but returns an error. Done should be preferred in Flow pipelines as it includes sub-process handling and logging.

func (*FlowMulti) New 

func (s *FlowMulti) New(name string, mf MultiFunc) pipeline.Pipeline

New creates a new Pipeline step that executes the provided MultiFunc onto a new `*Flow` type, creating a DAG. Because this function returns a pipeline.Step[T], it can be used with the normal Flow functions like `Run` and `Parallel`.

func (*FlowMulti) PrintGraph 

func (s *FlowMulti) PrintGraph(msg string)

type GitConfig 

type GitConfig struct{}

type InitializerFunc 

type InitializerFunc func(context.Context, clients.CommonOpts) (pipeline.Client, error)

type MultiFunc 

type MultiFunc func(*Flow)

func MultiFuncWithLogging 

func MultiFuncWithLogging(logger logrus.FieldLogger, mf MultiFunc) MultiFunc

type Pipeline 

type Pipeline struct {
	Name     string
	Requires []state.Argument
	Steps    []pipeline.Step
	Provides []state.Argument
	When     []pipeline.Event
}

Pipeline is a more user-friendly, declarative representation of a Pipeline in the 'pipeline' package. This is only used when defining a pipeline in a declarative manner using the 'AddPipelines' function.

type PipelineConfig 

type PipelineConfig struct{}

Config defines the typical options that are provided in a pipeline. These options can be provided many ways, and often depend on the execution environment. They are retrieved at pipeline-time.

Source Files

View all Source files

Directories

Path Synopsis
Package args just holds the type definitions for pipeline / cmd arguments to avoid cyclic imports
 Package args just holds the type definitions for pipeline / cmd arguments to avoid cyclic imports
cmd
Package main contains the logic for the `flow` CLI
 Package main contains the logic for the `flow` CLI
commands
Package cmdutil provides utility functions and types for working with the 'flow' CLI.
 Package cmdutil provides utility functions and types for working with the 'flow' CLI.
demo
basic
complex
custom-client
multi
multi-sub
state
Package exec provides helper functions and Actions for executing shell commands.
 Package exec provides helper functions and Actions for executing shell commands.
git
x
x
Package pipeline contains the meta types and interfaces that define a Flow pipeline.
 Package pipeline contains the meta types and interfaces that define a Flow pipeline.
clients
clients/cli
clients/common
clients/dagger
clients/drone
Package drone contians the drone client implementation for generating a Drone pipeline config.
 Package drone contians the drone client implementation for generating a Drone pipeline config.
clients/graphviz
dag
Package pipelineutil defines utilities for working with Pipelines and is separated as it may also import packages that import pipeline.
 Package pipelineutil defines utilities for working with Pipelines and is separated as it may also import packages that import pipeline.
Package plog (or plumbig log) provides a logging initializer and utility functions for working with a logging library.
 Package plog (or plumbig log) provides a logging initializer and utility functions for working with a logging library.
Package stringutil contains general string utilities used throughout this project.
 Package stringutil contains general string utilities used throughout this project.
Package syncutil provides utilities for working with asynchronous tasks and provides wrappers around the "sync" package.
 Package syncutil provides utilities for working with asynchronous tasks and provides wrappers around the "sync" package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.