pipeline

package
v0.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 4, 2022 License: Apache-2.0 Imports: 10 Imported by: 0

Documentation

Overview

Package pipeline contains the meta types and interfaces that define a Scribe pipeline.

Index

Constants

This section is empty.

Variables

View Source 
var (

	// Git arguments
	ArgumentCommitSHA = state.NewStringArgument("git-commit-sha")
	ArgumentCommitRef = state.NewStringArgument("git-commit-ref")
	ArgumentBranch    = state.NewStringArgument("git-branch")
	ArgumentRemoteURL = state.NewStringArgument("remote-url")
	ArgumentTagName   = state.NewStringArgument("git-tag")

	// Standard pipeline arguments
	ArgumentWorkingDir = state.NewStringArgument("workdir")
	// ArgumentSourceFS is the path to the root of the source code for this project.
	ArgumentSourceFS        = state.NewUnpackagedDirectoryArgument("source")
	ArgumentPipelineGoModFS = state.NewUnpackagedDirectoryArgument("pipeline-go-mod")
	ArgumentDockerSocketFS  = state.NewUnpackagedDirectoryArgument("docker-socket")

	// CI service arguments
	ArgumentBuildID = state.NewStringArgument("build-id")
)

These arguments are the pre-defined ones and are mostly used in events.

View Source 
var (
	ErrorNoSteps = errors.New("no steps were provided")
)
View Source 
var ErrorStepNotFound = errors.New("step not found")
View Source 
var GitCommitEventArgs = []state.Argument{
	ArgumentCommitSHA,
	ArgumentBranch,
	ArgumentRemoteURL,
}

GitCommitEventArgs are arguments that should provide in the pipeline state when a pipeline was created from a git commit event.

View Source 
var GitTagEventArgs = []state.Argument{
	ArgumentCommitSHA,
	ArgumentCommitRef,
	ArgumentRemoteURL,
}

GitTagEventArgs are arguments that should provide in the pipeline state when a pipeline was created from a git tag event.

View Source 
var NoOpStep = Step{
	Name: "no op",
	Action: func(context.Context, ActionOpts) error {
		return nil
	},
}

NoOpStep is used to represent a step which only exists to form uncommon relationships or for testing. Most clients should completely ignore NoOpSteps.

View Source 
var PullRequestEventArgs = []state.Argument{}

PullRequestEventArgs are arguments that should provide in the pipeline state when a pipeline was created from a pull request.

Functions

func PipelineNames 

func PipelineNames(s []Pipeline) []string

func StepIDs 

func StepIDs(steps []Step) []int64

func StepNames 

func StepNames(s []Step) []string

Types

type Action 

type Action func(context.Context, ActionOpts) error

Action is the function signature that a step provides when it does something. 

var DefaultAction Action = nil

DefaultAction is a nil action intentionally. In some client implementations, a nil step indicates a specific behavior. In Drone and Docker, for example, a nil step indicates that the docker command or entrypoint should not be supplied, thus using the default command for that image.

type ActionOpts 

type ActionOpts struct {
	State  *state.State
	Stdout io.Writer
	Stderr io.Writer
	Tracer opentracing.Tracer
	Logger logrus.FieldLogger

	// Path is the path to the pipeline, typically provided via the `-path` argument, but automatically supplied if using the scribe CLI.
	Path string
	// Version refers to the version of Scribe that was used to run the pipeline.
	// This value is set using the `-version` argument when running a pipeline, which is automatically set by the `scribe` command.
	Version string
}

The ActionOpts are provided to every step that is ran. Each step can choose to use these options.

type Artifact 

type Artifact interface{}

type CacheCondition 

type CacheCondition func() bool

a CacheCondition should return true if the cacher should cache the step. In other words, returning false means that the step must run.

type Cacher 

type Cacher func(Step)

A Cacher defines behavior for caching data. Some behaviors that can happen when dealing with a cacheable step: * The provided Step, using an expensive process, produces some consistent output, possibly on the filesystem. If nothing changes in between runs, then we can re-use the output in the current step and skip this one.

  • The most common example of this is `npm install` producing the `node_modules` folder, which can be re-used if `package-lock.json` is unchanged.

* The provided Step, using an expensive process, calculates a value. If nothing changes in between runs, then this value can be reused.

type Client 

type Client interface {
	// Validate is ran internally before calling Run or Parallel and allows the client to effectively configure per-step requirements
	// For example, Drone steps MUST have an image so the Drone client returns an error in this function when the provided step does not have an image.
	// If the error encountered is not critical but should still be logged, then return a plumbing.ErrorSkipValidation.
	// The error is checked with `errors.Is` so the error can be wrapped with fmt.Errorf.
	Validate(Step) error

	// Done must be ran at the end of the pipeline.
	// This is typically what takes the defined pipeline steps, runs them in the order defined, and produces some kind of output.
	Done(context.Context, Walker) error
}

type Collection 

type Collection struct {
	Graph *dag.Graph[Pipeline]
}

Collection defines a directed acyclic Graph that stores a collection of Steps.

func NewCollection 

func NewCollection() *Collection

func NewCollectionWithSteps 

func NewCollectionWithSteps(pipelineName string, steps ...StepList) (*Collection, error)

NewCollectinoWithSteps creates a new Collection with a single pipeline from a list of Steps.

func (*Collection) AddEvents 

func (c *Collection) AddEvents(pipelineID int64, events ...Event) error

AddEvents adds the list of events to the pipeline with 'pipelineID'. Events are not unique themselves are really only a list of arguments.

func (*Collection) AddPipelines 

func (c *Collection) AddPipelines(p ...Pipeline) error

AppendPipeline adds a populated sub-pipeline of Steps to the Graph.

func (*Collection) AddSteps 

func (c *Collection) AddSteps(pipelineID int64, steps StepList) error

Add adds a new list of Steps which are siblings to a pipeline. Because they are siblings, they must all depend on the same step(s).

func (*Collection) ByID 

func (c *Collection) ByID(ctx context.Context, id int64) ([]Step, error)

ByID should return the Step that corresponds with a specific ID

func (*Collection) ByName 

func (c *Collection) ByName(ctx context.Context, name string) ([]Step, error)

ByName should return the Step that corresponds with a specific Name

func (*Collection) PipelinesByEvent 

func (c *Collection) PipelinesByEvent(ctx context.Context, name string) ([]Pipeline, error)

func (*Collection) PipelinesByName 

func (c *Collection) PipelinesByName(ctx context.Context, names []string) ([]Pipeline, error)

PipelinesByName should return the Pipelines that corresponds with a specified names

func (*Collection) WalkPipelines 

func (c *Collection) WalkPipelines(ctx context.Context, wf PipelineWalkFunc) error

func (*Collection) WalkSteps 

func (c *Collection) WalkSteps(ctx context.Context, pipelineID int64, wf StepWalkFunc) error

type Configurer 

type Configurer interface {
	// Value returns the implementation-specific pipeline config.
	Value(state.Argument) (string, error)
}

Configurer defines how clients can retrieve configuration values for use in pipelines. For example, a `clone` step might require a remote URL and branch, but how that data is retrieved can change depending on the environment. * In Drone, the value is retrieved from an environment variable at runtime. So the only thing this funcion will likely return is the name of the environment variable. * With the Dagger and CLI clients, the remote URL might be provided as a CLI argument or requested via stdin, or even already available with the `git remote` command.

type EnvVar 

type EnvVar struct {
	Type EnvVarType
	// contains filtered or unexported fields
}

func NewEnvArgument 

func NewEnvArgument(arg state.Argument) EnvVar

NewEnvArgument creates a new EnvVar that will be populated based on an Argument found in the state when the step runs.

func NewEnvString 

func NewEnvString(val string) EnvVar

NewEnvString creates a new EnvVar that will be populated with a static string value.

func (EnvVar) Argument 

func (e EnvVar) Argument() state.Argument

Argument retrieves the argument value set when using the NewEnvArgument function. If the EnvVar's Type property is not "EnvVarString" then it will panic.

func (EnvVar) String 

func (e EnvVar) String() string

String retrieves the static string value set when using the NewEnvString function. If the EnvVar's Type property is not "EnvVarString" then it will panic.

type EnvVarType 

type EnvVarType int
const (
	// EnvVarString should be used whenever the value is known and static.
	EnvVarString EnvVarType = iota

	// EnvVarArgument means that the environment variable will be populated by an argument from the state at run-time.
	// Most (all?) CI services will then leave these values out of the configuration and will be injected when the step runs.
	EnvVarArgument
)

type Event 

type Event struct {
	Name     string
	Filters  map[string]*FilterValue
	Provides []state.Argument
}

Event is provided when defining a Scribe pipeline to define the events that cause the pipeline to be ran. Some example events that might cause pipelines to be created: * Manual events with user input, like 'Promotions' in Drone. In this scenario, the user may have the ability to supply any keys/values as arguments, however, pipeline developers in Scribe should be able to specifically define what fields are accepted. See https://docs.drone.io/promote/. * git and SCM-related events like 'Pull Reuqest', 'Commit', 'Tag'. Each one of these events has a unique set of arguments / filters. `Commit` may allow pipeline developers to filter by branch or message. Tags may allow developers to filter by name. * cron events, which may allow the pipeline in the CI service to be ran on a schedule. The Event type stores both the filters and a list of values that it provides to the pipeline. Client implementations of the pipeline (type Client) are expected to handle events that they are capable of handling. 'Handling' events means that the the arguments in the `Provides` key should be available before any first steps are ran. It will not typically be up to pipeline developers to decide what arguments an event provides. The only case where this may happen is if the event is a manual one, where users are able to submit the event with any arbitrary set of keys/values. The 'Filters' key is provided in the pipeline code and should not be populated when pre-defined in the Scribe package.

func GitCommitEvent 

func GitCommitEvent(filters GitCommitFilters) Event

func GitTagEvent 

func GitTagEvent(filters GitTagFilters) Event

func PullRequestEvent 

func PullRequestEvent(filters PullRequestFilters) Event

type FilterValue 

type FilterValue struct {
	Type  FilterValueType
	Value fmt.Stringer
}

func GlobFilter 

func GlobFilter(v string) *FilterValue

func RegexpFilter 

func RegexpFilter(v *regexp.Regexp) *FilterValue

func StringFilter 

func StringFilter(v string) *FilterValue

func (*FilterValue) String 

func (f *FilterValue) String() string

type FilterValueType 

type FilterValueType int

FilterValueType is the metadata type that identifies the type present event filter. 

const (
	FilterValueString FilterValueType = iota
	FilterValueRegex
	FilterValueGlob
)

type GitCommitFilters 

type GitCommitFilters struct {
	Branch *FilterValue
}

type GitTagFilters 

type GitTagFilters struct {
	Name *FilterValue
}

type Output 

type Output interface{}

type Pipeline 

type Pipeline struct {
	ID           int64
	Name         string
	Graph        *dag.Graph[StepList]
	Events       []Event
	Type         PipelineType
	Dependencies []Pipeline
}

A Pipeline is really similar to a Step, except that it contains a graph of steps rather than a single action. Just like a Step, it has dependencies, a name, an ID, etc.

func AdjNodesToPipelines 

func AdjNodesToPipelines(nodes []*dag.Node[Pipeline]) []Pipeline

AdjNodesToPipelines converts a list of Nodes (with type Pipeline) to a list of Pipelines

func New 

func New(name string, id int64) Pipeline

New creates a new Step that represents a pipeline.

func NodesToPipelines 

func NodesToPipelines(nodes []dag.Node[Pipeline]) []Pipeline

NodesToPipelines converts a list of Nodes (with type Pipeline) to a list of Pipelines

func (*Pipeline) AddSteps 

func (p *Pipeline) AddSteps(id int64, steps StepList) error

AddStep adds the steps as a single node in the pipeline.

type PipelineType 

type PipelineType int
const (
	PipelineTypeDefault PipelineType = iota
	PipelineTypeSub
)

type PipelineWalkFunc 

type PipelineWalkFunc func(context.Context, ...Pipeline) error

PipelineWalkFunc is implemented by the pipeline 'Clients'. This function is executed for each pipeline. This function follows the same rules for pipelines as the StepWalker func does for pipelines. If multiple pipelines are provided in the steps argument, then those pipelines are intended to be executed in parallel.

type PullRequestFilters 

type PullRequestFilters struct{}

type Step 

type Step struct {
	// ID is the unique number that represents this step.
	// This value is used when calling `scribe -step={serial} [pipeline]`
	ID int64

	// Type represents the how the step is intended to operate. 90% of the time, the default type should be a sufficient descriptor of a step.
	// However in some circumstances, clients may want to handle a step differently based on how it's defined.
	// Background steps, for example, have to have their lifecycles handled differently.
	Type StepType

	// Name is a string that represents or describes the step, essentially the identifier.
	// Not all clients will support using the name for anything beyond logging.
	Name string

	// Image is an optional value that can be assigned to a step.
	// Typically, in docker environments (or drone with a Docker executor), it defines the docker image that is used to run the step.
	Image string

	// Action defines the action this step performs.
	Action Action

	// Dependencies define other steps that are required to run before this one.
	// As far as we're concerned, Steps can only depend on other steps of the same type.
	Dependencies []Step

	// Arguments are arguments that are must exist in order for this step to run.
	Arguments []state.Argument

	// Provides are arguments that this step provides for other arguments to use in their "Arguments" list.
	ProvidesArgs []state.Argument

	Environment StepEnv
}

A Step stores a Action and a name for use in pipelines. A Step can consist of either a single action or represent a list of actions.

func Combine 

func Combine(step ...Step) Step

Combine combines the list of steps into one step, combining all of their required and provided arguments, as well as their actions. For string values that can not be combined, like Name and Image, the first step's values are chosen. These can be overridden with further chaining.

func NamedStep 

func NamedStep(name string, action Action) Step

NamedStep creates a new step with a name provided

func NewStep 

func NewStep(action Action) Step

NewStep creates a new step with an automatically generated name

func NodesToSteps 

func NodesToSteps(nodes []dag.Node[Step]) []Step

NodeListToSteps converts a list of Nodes to a list of Steps

func (Step) After 

func (s Step) After(step Step) Step

func (Step) IsBackground 

func (s Step) IsBackground() bool

func (Step) Provides 

func (s Step) Provides(arg ...state.Argument) Step

func (Step) Requires 

func (s Step) Requires(args ...state.Argument) Step

func (Step) ResetArguments 

func (s Step) ResetArguments() Step

func (Step) WithEnvVar 

func (s Step) WithEnvVar(key string, val EnvVar) Step

WithEnvVar appends a new EnvVar to the Step's environment, replacing existing EnvVars with the provided key. If an EnvVar is provided with a type of EnvVarArgument, then the argument is also added to this step's required arguments.

func (Step) WithEnvironment 

func (s Step) WithEnvironment(env StepEnv) Step

WithEnvironment replaces the entire environment for this step. If an EnvVar is provided with a type of EnvVarArgument, then the argument is also added to this step's required arguments.

func (Step) WithImage 

func (s Step) WithImage(image string) Step

func (Step) WithInput 

func (s Step) WithInput(artifact Artifact) Step

func (Step) WithName 

func (s Step) WithName(name string) Step

func (Step) WithOutput 

func (s Step) WithOutput(artifact Artifact) Step

type StepEnv 

type StepEnv map[string]EnvVar

type StepList 

type StepList struct {
	ID           int64
	Steps        []Step
	Dependencies []StepList
	Type         StepType
}

A StepList is a list of steps that are ran in parallel. This type is only used for intermittent storage and should not be used in the Scribe client library

func NewStepList 

func NewStepList(id int64, steps ...Step) StepList

func NodeListToStepLists 

func NodeListToStepLists(nodes []dag.Node[StepList]) []StepList

NodeListToSteps converts a list of Nodes to a list of Steps

func (*StepList) Names 

func (s *StepList) Names() []string

func (*StepList) String 

func (s *StepList) String() string

type StepType 

type StepType int
const (
	StepTypeDefault StepType = iota
	StepTypeBackground
)

type StepWalkFunc 

type StepWalkFunc func(context.Context, ...Step) error

WalkFunc is implemented by the pipeline 'Clients'. This function is executed for each step. If multiple steps are provided in the argument, then they were provided in "Parallel". If one step in the list of steps is of type "Background", then they all should be.

type Stringer 

type Stringer string

func (Stringer) String 

func (s Stringer) String() string

type Walker 

type Walker interface {
	WalkSteps(context.Context, int64, StepWalkFunc) error
	WalkPipelines(context.Context, PipelineWalkFunc) error
}

Walker is an interface that collections of steps should satisfy.

Source Files

View all Source files

Directories

Path Synopsis
cli
common
dagger
drone
Package drone contians the drone client implementation for generating a Drone pipeline config.
 Package drone contians the drone client implementation for generating a Drone pipeline config.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.