pipeline

package
v0.12.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 5, 2023 License: Apache-2.0 Imports: 11 Imported by: 0

Documentation

Overview

Package pipeline contains the meta types and interfaces that define a Scribe pipeline.

Index

Constants

This section is empty.

Variables

View Source 
var (
	// Git arguments
	ArgumentCommitSHA = state.NewStringArgument("git-commit-sha")
	ArgumentCommitRef = state.NewStringArgument("git-commit-ref")
	ArgumentBranch    = state.NewStringArgument("git-branch")
	ArgumentRemoteURL = state.NewStringArgument("remote-url")
	ArgumentTagName   = state.NewStringArgument("git-tag")

	ArgumentWorkingDir = state.NewStringArgument("workdir")
	// ArgumentSourceFS is the path to the root of the source code for this project.
	ArgumentSourceFS        = state.NewUnpackagedDirectoryArgument("source")
	ArgumentPipelineGoModFS = state.NewUnpackagedDirectoryArgument("pipeline-go-mod")
	ArgumentDockerSocketFS  = state.NewUnpackagedDirectoryArgument("docker-socket")

	// CI service arguments
	ArgumentBuildID = state.NewStringArgument("build-id")
)

These arguments are the pre-defined ones and are mostly used in events.

View Source 
var (
	ErrorNoPipelineProvider = errors.New("no pipeline in the graph provides a required argument")
	ErrorNoSteps            = errors.New("no steps were provided")
)
View Source 
var (
	ErrorNoStepProvider    = errors.New("no step in the graph provides a required argument")
	ErrorAmbiguousProvider = errors.New("more than one step provides the same argument(s)")
)
View Source 
var ClientProvidedArguments = []state.Argument{ArgumentBuildID, ArgumentSourceFS, ArgumentPipelineGoModFS, ArgumentDockerSocketFS, ArgumentWorkingDir}

ClientProvidedArguments are argumnets that must be provided by the Client and not another step.

View Source 
var ErrorStepNotFound = errors.New("step not found")
View Source 
var GitCommitEventArgs = []state.Argument{
	ArgumentCommitSHA,
	ArgumentBranch,
	ArgumentRemoteURL,
}

GitCommitEventArgs are arguments that should provide in the pipeline state when a pipeline was created from a git commit event.

View Source 
var GitTagEventArgs = []state.Argument{
	ArgumentCommitSHA,
	ArgumentCommitRef,
	ArgumentRemoteURL,
}

GitTagEventArgs are arguments that should provide in the pipeline state when a pipeline was created from a git tag event.

View Source 
var NoOpStep = Step{
	Name: "no op",
	Action: func(context.Context, ActionOpts) error {
		return nil
	},
}

NoOpStep is used to represent a step which only exists to form uncommon relationships or for testing. Most clients should completely ignore NoOpSteps.

View Source 
var PullRequestEventArgs = []state.Argument{}

PullRequestEventArgs are arguments that should provide in the pipeline state when a pipeline was created from a pull request.

Functions

func PipelineNames 

func PipelineNames(s []Pipeline) []string

func PrintCollection added in v0.12.0 

func PrintCollection(col *Collection)

func StepNames 

func StepNames(steps []Step) []string

Types

type Action 

type Action func(context.Context, ActionOpts) error

Action is the function signature that a step provides when it does something. 

var DefaultAction Action = nil

DefaultAction is a nil action intentionally. In some client implementations, a nil step indicates a specific behavior. In Drone and Docker, for example, a nil step indicates that the docker command or entrypoint should not be supplied, thus using the default command for that image.

type ActionOpts 

type ActionOpts struct {
	State  state.Handler
	Stdout io.Writer
	Stderr io.Writer
	Tracer opentracing.Tracer
	Logger logrus.FieldLogger

	// Path is the path to the pipeline, typically provided via the `-path` argument, but automatically supplied if using the scribe CLI.
	Path string
	// Version refers to the version of Scribe that was used to run the pipeline.
	// This value is set using the `-version` argument when running a pipeline, which is automatically set by the `scribe` command.
	Version string
}

The ActionOpts are provided to every step that is ran. Each step can choose to use these options.

type CacheCondition 

type CacheCondition func() bool

a CacheCondition should return true if the cacher should cache the step. In other words, returning false means that the step must run.

type Cacher 

type Cacher func(Step)

A Cacher defines behavior for caching data. Some behaviors that can happen when dealing with a cacheable step: * The provided Step, using an expensive process, produces some consistent output, possibly on the filesystem. If nothing changes in between runs, then we can re-use the output in the current step and skip this one.

  • The most common example of this is `npm install` producing the `node_modules` folder, which can be re-used if `package-lock.json` is unchanged.

* The provided Step, using an expensive process, calculates a value. If nothing changes in between runs, then this value can be reused.

type Client 

type Client interface {
	// Validate is ran internally before calling Run or Parallel and allows the client to effectively configure per-step requirements
	// For example, Drone steps MUST have an image so the Drone client returns an error in this function when the provided step does not have an image.
	// If the error encountered is not critical but should still be logged, then return a plumbing.ErrorSkipValidation.
	// The error is checked with `errors.Is` so the error can be wrapped with fmt.Errorf.
	Validate(Step) error

	// Done must be ran at the end of the pipeline.
	// This is typically what takes the defined pipeline steps, runs them in the order defined, and produces some kind of output.
	Done(context.Context, *Collection) error
}

type Collection 

type Collection struct {
	Graph     *dag.Graph[Pipeline]
	Providers map[state.Argument]int64
	Root      []int64
}

Collection defines a directed acyclic Graph that stores a collection of Steps. When using Scribe with "scribe.New", this collection contains a graph with one pipeline in it. When using Scribe with "scribe.NewMulti", this collection contains a graph with several pipelines in it.

func NewCollection 

func NewCollection() *Collection

func NewCollectionWithSteps 

func NewCollectionWithSteps(pipelineName string, steps ...Step) (*Collection, error)

NewCollectinoWithSteps creates a new Collection with a single pipeline from a list of Steps.

func (*Collection) AddEvents 

func (c *Collection) AddEvents(pipelineID int64, events ...Event) error

AddEvents adds the list of events to the pipeline with 'pipelineID'. Events are not unique themselves are really only a list of arguments.

func (*Collection) AddPipelines 

func (c *Collection) AddPipelines(pipelines ...Pipeline) error

AppendPipeline adds a populated pipeline of Steps to the Graph.

func (*Collection) AddSteps 

func (c *Collection) AddSteps(pipelineID int64, steps ...Step) error

Add adds a new list of Steps to a pipeline. The order in which steps are added have no particular meaning.

func (*Collection) BuildEdges added in v0.11.1 

func (c *Collection) BuildEdges(log logrus.FieldLogger, rootArgs ...state.Argument) error

BuidlEdges generates the edges in each pipeline based on the required and provided args of each step and pipeline.

func (*Collection) ByID 

func (c *Collection) ByID(ctx context.Context, id int64) (Step, error)

ByID should return the Step that corresponds with a specific ID

func (*Collection) ByName 

func (c *Collection) ByName(ctx context.Context, name string) ([]Step, error)

ByName should return the Step that corresponds with a specific Name

func (*Collection) PipelinesByEvent 

func (c *Collection) PipelinesByEvent(ctx context.Context, name string) ([]Pipeline, error)

func (*Collection) PipelinesByName 

func (c *Collection) PipelinesByName(ctx context.Context, names []string) ([]Pipeline, error)

PipelinesByName should return the Pipelines that corresponds with a specified names

func (*Collection) SetProvider added in v0.11.1 

func (c *Collection) SetProvider(arg state.Argument, id int64) error

SetProvider sets the provider of the argument 'arg' to the pipeline ID 'id'. If there is already a pipeline that provides the argument 'arg', then a 'pipeline.ErrorAmbiguousProvider' is returned. The provider for the argument is used to create an edge between the provider and pipelines that require the argument that is provided by it.

func (*Collection) WalkPipelines 

func (c *Collection) WalkPipelines(ctx context.Context, wf PipelineWalkFunc) error

func (*Collection) WalkSteps 

func (c *Collection) WalkSteps(ctx context.Context, pipelineID int64, wf StepWalkFunc) error

type Configurer 

type Configurer interface {
	// Value returns the implementation-specific pipeline config.
	Value(state.Argument) (string, error)
}

Configurer defines how clients can retrieve configuration values for use in pipelines. For example, a `clone` step might require a remote URL and branch, but how that data is retrieved can change depending on the environment. * In Drone, the value is retrieved from an environment variable at runtime. So the only thing this funcion will likely return is the name of the environment variable. * With the Dagger and CLI clients, the remote URL might be provided as a CLI argument or requested via stdin, or even already available with the `git remote` command.

type EnvVar 

type EnvVar struct {
	Type EnvVarType
	// contains filtered or unexported fields
}

func NewEnvArgument 

func NewEnvArgument(arg state.Argument) EnvVar

NewEnvArgument creates a new EnvVar that will be populated based on an Argument found in the state when the step runs.

func NewEnvString 

func NewEnvString(val string) EnvVar

NewEnvString creates a new EnvVar that will be populated with a static string value.

func (EnvVar) Argument 

func (e EnvVar) Argument() state.Argument

Argument retrieves the argument value set when using the NewEnvArgument function. If the EnvVar's Type property is not "EnvVarString" then it will panic.

func (EnvVar) String 

func (e EnvVar) String() string

String retrieves the static string value set when using the NewEnvString function. If the EnvVar's Type property is not "EnvVarString" then it will panic.

type EnvVarType 

type EnvVarType int
const (
	// EnvVarString should be used whenever the value is known and static.
	EnvVarString EnvVarType = iota

	// EnvVarArgument means that the environment variable will be populated by an argument from the state at run-time.
	// Most (all?) CI services will then leave these values out of the configuration and will be injected when the step runs.
	EnvVarArgument
)

type Event 

type Event struct {
	Name     string
	Filters  map[string]*FilterValue
	Provides []state.Argument
}

Event is provided when defining a Scribe pipeline to define the events that cause the pipeline to be ran. Some example events that might cause pipelines to be created: * Manual events with user input, like 'Promotions' in Drone. In this scenario, the user may have the ability to supply any keys/values as arguments, however, pipeline developers in Scribe should be able to specifically define what fields are accepted. See https://docs.drone.io/promote/. * git and SCM-related events like 'Pull Reuqest', 'Commit', 'Tag'. Each one of these events has a unique set of arguments / filters. `Commit` may allow pipeline developers to filter by branch or message. Tags may allow developers to filter by name. * cron events, which may allow the pipeline in the CI service to be ran on a schedule. The Event type stores both the filters and a list of values that it provides to the pipeline. Client implementations of the pipeline (type Client) are expected to handle events that they are capable of handling. 'Handling' events means that the the arguments in the `Provides` key should be available before any first steps are ran. It will not typically be up to pipeline developers to decide what arguments an event provides. The only case where this may happen is if the event is a manual one, where users are able to submit the event with any arbitrary set of keys/values. The 'Filters' key is provided in the pipeline code and should not be populated when pre-defined in the Scribe package.

func GitCommitEvent 

func GitCommitEvent(filters GitCommitFilters) Event

func GitTagEvent 

func GitTagEvent(filters GitTagFilters) Event

func PullRequestEvent 

func PullRequestEvent(filters PullRequestFilters) Event

type FilterValue 

type FilterValue struct {
	Type  FilterValueType
	Value fmt.Stringer
}

func GlobFilter 

func GlobFilter(v string) *FilterValue

func RegexpFilter 

func RegexpFilter(v *regexp.Regexp) *FilterValue

func StringFilter 

func StringFilter(v string) *FilterValue

func (*FilterValue) String 

func (f *FilterValue) String() string

type FilterValueType 

type FilterValueType int

FilterValueType is the metadata type that identifies the type present event filter. 

const (
	FilterValueString FilterValueType = iota
	FilterValueRegex
	FilterValueGlob
)

type GitCommitFilters 

type GitCommitFilters struct {
	Branch *FilterValue
}

type GitTagFilters 

type GitTagFilters struct {
	Name *FilterValue
}

type Output 

type Output interface{}

type Pipeline 

type Pipeline struct {
	ID   int64
	Name string
	// Graph is a graph where each node is a list of Steps. Each set of steps runs in parallel.
	Graph     *dag.Graph[Step]
	Providers map[state.Argument]int64
	Root      []int64
	Events    []Event
	Type      PipelineType

	RequiredArgs state.Arguments
	ProvidedArgs state.Arguments
}

A Pipeline is really similar to a Step, except that it contains a graph of steps rather than a single action. Just like a Step, it has dependencies, a name, an ID, etc.

func AdjNodesToPipelines 

func AdjNodesToPipelines(nodes []*dag.Node[Pipeline]) []Pipeline

AdjNodesToPipelines converts a list of Nodes (with type Pipeline) to a list of Pipelines

func New 

func New(name string, id int64) Pipeline

New creates a new Step that represents a pipeline.

func (*Pipeline) AddSteps 

func (p *Pipeline) AddSteps(steps ...Step) error

AddStep adds all of the provided steps into the pipeline. The Step's ID field is used as the node ID.

func (Pipeline) BuildEdges added in v0.11.0 

func (p Pipeline) BuildEdges(rootArgs ...state.Argument) error

BuildEdges generates the edges of the step graph based on the required / provided args of each step. It will return an error if there are required arguments that are not satisfied.

func (Pipeline) Provides added in v0.11.1 

func (p Pipeline) Provides(args ...state.Argument) Pipeline

func (Pipeline) Requires added in v0.11.1 

func (p Pipeline) Requires(args ...state.Argument) Pipeline

func (Pipeline) SetProvider added in v0.11.0 

func (p Pipeline) SetProvider(arg state.Argument, id int64) error

type PipelineType 

type PipelineType int
const (
	PipelineTypeDefault PipelineType = iota
	PipelineTypeSub
)

type PipelineWalkFunc 

type PipelineWalkFunc func(context.Context, Pipeline) error

PipelineWalkFunc is implemented by the pipeline 'Clients'. This function is executed for each pipeline. This function follows the same rules for pipelines as the StepWalker func does for pipelines. If multiple pipelines are provided in the steps argument, then those pipelines are intended to be executed in parallel.

type PullRequestFilters 

type PullRequestFilters struct{}

type Step 

type Step struct {
	// ID is the unique number that represents this step.
	// This value is used when calling `scribe -step={serial} [pipeline]`
	ID int64

	// Type represents the how the step is intended to operate. 90% of the time, the default type should be a sufficient descriptor of a step.
	// However in some circumstances, clients may want to handle a step differently based on how it's defined.
	// Background steps, for example, have to have their lifecycles handled differently.
	Type StepType

	// Name is a string that represents or describes the step, essentially the identifier.
	// Not all clients will support using the name for anything beyond logging.
	Name string

	// Image is an optional value that can be assigned to a step.
	// Typically, in docker environments (or drone with a Docker executor), it defines the docker image that is used to run the step.
	Image string

	// Action defines the action this step performs.
	Action Action

	// RequiredArgs are arguments that are must exist in order for this step to run.
	RequiredArgs state.Arguments

	// Provides are arguments that this step provides for other arguments to use in their "Arguments" list.
	ProvidedArgs state.Arguments

	Environment StepEnv
}

A Step stores a Action and a name for use in pipelines. A Step can consist of either a single action or represent a list of actions.

func Combine 

func Combine(step ...Step) Step

Combine combines the list of steps into one step, combining all of their required and provided arguments, as well as their actions. For string values that can not be combined, like Name and Image, the first step's values are chosen. These can be overridden with further chaining.

func NamedStep 

func NamedStep(name string, action Action) Step

NamedStep creates a new step with a name provided

func NewStep 

func NewStep(action Action) Step

NewStep creates a new step with an automatically generated name

func (Step) IsBackground 

func (s Step) IsBackground() bool

func (Step) Provides 

func (s Step) Provides(arg ...state.Argument) Step

func (Step) Requires 

func (s Step) Requires(args ...state.Argument) Step

func (Step) ResetArguments 

func (s Step) ResetArguments() Step

func (Step) WithEnvVar 

func (s Step) WithEnvVar(key string, val EnvVar) Step

WithEnvVar appends a new EnvVar to the Step's environment, replacing existing EnvVars with the provided key. If an EnvVar is provided with a type of EnvVarArgument, then the argument is also added to this step's required arguments.

func (Step) WithEnvironment 

func (s Step) WithEnvironment(env StepEnv) Step

WithEnvironment replaces the entire environment for this step. If an EnvVar is provided with a type of EnvVarArgument, then the argument is also added to this step's required arguments.

func (Step) WithImage 

func (s Step) WithImage(image string) Step

func (Step) WithName 

func (s Step) WithName(name string) Step

type StepEnv 

type StepEnv map[string]EnvVar

type StepType 

type StepType int
const (
	StepTypeDefault StepType = iota
	StepTypeBackground
)

type StepWalkFunc 

type StepWalkFunc func(context.Context, Step) error

WalkFunc is implemented by the pipeline 'Clients'. This function is executed for each Step.

type Stringer 

type Stringer string

func (Stringer) String 

func (s Stringer) String() string

Source Files

View all Source files

Directories

Path Synopsis
cli
common
dagger
drone
Package drone contians the drone client implementation for generating a Drone pipeline config.
 Package drone contians the drone client implementation for generating a Drone pipeline config.
graphviz

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.