pipeline

package module
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 26, 2024 License: MIT Imports: 15 Imported by: 7

README

go-pipeline

Build status Go Reference

go-pipeline is a Go libary used for building and modifying Buildkite pipelines in golang. It's used internally by the Buildkite Agent to inspect and sign pipelines prior to uploading them, but is also useful for building tools that generate pipelines.

Installation

To install, run

go get -u github.com/buildkite/go-pipeline

This will add go-pipeline to your go.mod file, and make it available for use in your project.

Usage

Loading a pipeline from yaml
const aPipeline = `
env:
  MOUNTAIN: cotopaxi
  COUNTRY: ecuador

steps:
  - command: echo "hello world"
  - wait
  - command: echo "goodbye world"
`

p, err := pipeline.Parse(strings.NewReader(aPipeline))
if err != nil {
  panic(err)
}

pretty.Println(p)
// &pipeline.Pipeline{
//   Env: &ordered.Map[string,string]{
//     items: {
//       {Key:"MOUNTAIN", Value:"cotopaxi", deleted:false},
//       {Key:"COUNTRY", Value:"ecuador", deleted:false},
//     },
//     index: {"MOUNTAIN":0, "COUNTRY":1},
//   },
//   Steps: {
//     &pipeline.CommandStep{
//       Command:         "echo \"hello world\"",
//       Env:             {},
//       RemainingFields: {},
//     },
//     &pipeline.WaitStep{
//       Scalar:  "wait",
//       Contents: {},
//     },
//     &pipeline.CommandStep{
//       Command:         "echo \"goodbye world\"",
//       Env:             {},
//       RemainingFields: {},
//     },
//   },
//   RemainingFields: {},
// }
Marshalling to YAML or JSON
aPipeline := `...`
p, err := pipeline.Parse(strings.NewReader(aPipeline))
if err != nil {
  // ...
}

//... modify the pipeline

// Marshal to YAML
b, err := yaml.Marshal(p)
if err != nil {
  // ...
}

// Marshal to JSON
b, err := json.Marshal(p)
if err != nil {
  // ...
}

Caveats

The pipeline object model (Pipeline, Steps, Plugin, etc) have these caveats:

  • It is incomplete: there may be fields accepted by the API that are not listed. Do not treat Pipeline, CommandStep, etc, as comprehensive reference guides for how to write a pipeline.
  • It normalises: unmarshaling accepts a variety of step forms, but marshaling back out produces more normalised output. An unmarshal/marshal round-trip may produce different output.
  • It is non-canonical: using the object model does not guarantee that a pipeline will be accepted by the pipeline upload API.

Notably, most of the structs defined by this module only contain the elements of a pipeline (and steps) necessary for the agent to understand, and are (at the time of writing) not comprehensive. Where relevant - that is, where there are more fields that are not included in the struct - the RemainingFields field is used to capture the remaining fields as a map[string]any. This allows pipelines to be loaded and modified without losing information, even if the pipeline contains fields that are not yet understood by the agent.

For example, the command step:

command: echo "hello world"
env:
  FOO: bar
  BAZ: qux
artifact_paths:
  - "logs/**/*"
  - "coverage/**/*"
parallelism: 5

would be represented in go as:

&pipeline.CommandStep{
  Command: `echo "hello world"`,
  Env: ordered.MapFromItems(
    ordered.TupleSS("FOO", "bar"),
    ordered.TupleSS("BAZ", "qux"),
  ),
  RemainingFields: map[string]any{
    "artifact_paths": []string{"logs/**/*", "coverage/**/*"},
    "parallelism": 5,
  },
}

This go struct would be marshaled back out to YAML equivalent to the original input.

What's up with the ordered module?

While implementing the pipeline module, we ran into a problem: in some cases, in the buildkite pipeline.yaml, the order of map fields is significant. Because of this, whenever the pipeline gets unmarshaled from YAML or JSON, it needs to be stored in a way that preserves the order of the fields. The ordered module is a simple implementation of an ordered map. In most cases, when the pipeline is dealing with user-input maps, it will store them internally as ordered.Maps. When the pipeline is marshaled back out to YAML or JSON, the ordered.Maps will be marshaled in the correct order.

Contributing

Contributions, bugfixes, issues and PRs are always welcome! Please see CONTRIBUTING.md for more details.

Documentation

Overview

Package pipeline implements the pieces necessary for the agent to work with pipelines (typically in YAML or JSON form).

The pipeline object model (Pipeline, Steps, Plugin, etc) have these caveats:

  • It is incomplete: there may be fields accepted by the API that are not listed. Do not treat Pipeline, CommandStep, etc, as comprehensive reference guides for how to write a pipeline.
  • It normalises: unmarshaling accepts a variety of step forms, but marshaling back out produces more normalised output. An unmarshal/marshal round-trip may produce different output.
  • It is non-canonical: using the object model does not guarantee that a pipeline will be accepted by the pipeline upload API.

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrStepTypeInference = errors.New("cannot infer step type")
	ErrUnknownStepType   = errors.New("unknown step type")
)

Sentinel errors that can appear when falling back to UnknownStep.

Functions

This section is empty.

Types

type Cache added in v0.6.0 

type Cache struct {
	Disabled bool     `yaml:",omitempty"`
	Name     string   `yaml:"name,omitempty"`
	Paths    []string `yaml:"paths,omitempty"`
	Size     string   `yaml:"size,omitempty"`

	RemainingFields map[string]any `yaml:",inline"`
}

Cache models the cache settings for a given step

func (*Cache) MarshalJSON added in v0.7.0 

func (c *Cache) MarshalJSON() ([]byte, error)

MarshalJSON marshals the step to JSON. Special handling is needed because yaml.v3 has "inline" but encoding/json has no concept of it.

func (*Cache) UnmarshalOrdered added in v0.6.0 

func (c *Cache) UnmarshalOrdered(o any) error

UnmarshalOrdered unmarshals from the following types: - string: a single path - []string: multiple paths - ordered.Map: a map containing paths, among potentially other things

type CommandStep 

type CommandStep struct {
	// Fields common to various step types
	Key   string `yaml:"key,omitempty" aliases:"id,identifier"`
	Label string `yaml:"label,omitempty" aliases:"name"`

	// Fields that are meaningful specifically for command steps
	Command   string            `yaml:"command"`
	Plugins   Plugins           `yaml:"plugins,omitempty"`
	Env       map[string]string `yaml:"env,omitempty"`
	Signature *Signature        `yaml:"signature,omitempty"`
	Matrix    *Matrix           `yaml:"matrix,omitempty"`
	Cache     *Cache            `yaml:"cache,omitempty"`

	// RemainingFields stores any other top-level mapping items so they at least
	// survive an unmarshal-marshal round-trip.
	RemainingFields map[string]any `yaml:",inline"`
}

CommandStep models a command step.

Standard caveats apply - see the package comment.

func (*CommandStep) InterpolateMatrixPermutation 

func (c *CommandStep) InterpolateMatrixPermutation(mp MatrixPermutation) error

InterpolateMatrixPermutation validates and then interpolates the choice of matrix values into the step. This should only be used in order to validate a job that's about to be run, and not used before pipeline upload.

func (*CommandStep) MarshalJSON 

func (c *CommandStep) MarshalJSON() ([]byte, error)

MarshalJSON marshals the step to JSON. Special handling is needed because yaml.v3 has "inline" but encoding/json has no concept of it.

func (*CommandStep) UnmarshalJSON 

func (c *CommandStep) UnmarshalJSON(b []byte) error

UnmarshalJSON is used when unmarshalling an individual step directly, e.g. from the Agent API Accept Job.

func (*CommandStep) UnmarshalOrdered 

func (c *CommandStep) UnmarshalOrdered(src any) error

UnmarshalOrdered unmarshals a command step from an ordered map.

type GroupStep 

type GroupStep struct {
	// Fields common to various step types
	Key string `yaml:"key,omitempty" aliases:"id,identifier"`

	// Group must always exist in a group step (so that we know it is a group).
	// If it has a value, it is treated as equivalent to the label or name.
	Group *string `yaml:"group" aliases:"label,name"`

	Steps Steps `yaml:"steps"`

	// RemainingFields stores any other top-level mapping items so they at least
	// survive an unmarshal-marshal round-trip.
	RemainingFields map[string]any `yaml:",inline"`
}

GroupStep models a group step.

Standard caveats apply - see the package comment.

func (*GroupStep) MarshalJSON 

func (g *GroupStep) MarshalJSON() ([]byte, error)

MarshalJSON marshals the step to JSON. Special handling is needed because yaml.v3 has "inline" but encoding/json has no concept of it.

func (*GroupStep) UnmarshalOrdered 

func (g *GroupStep) UnmarshalOrdered(src any) error

UnmarshalOrdered unmarshals a group step from an ordered map.

type InputStep 

type InputStep struct {
	Scalar   string         `yaml:"-"`
	Contents map[string]any `yaml:",inline"`
}

InputStep models a block or input step.

Standard caveats apply - see the package comment.

func (*InputStep) MarshalJSON 

func (s *InputStep) MarshalJSON() ([]byte, error)

MarshalJSON marshals s.Scalar if it's not empty, otherwise s.Contents if that is not empty. If both s.Scalar and s.Contents are empty, it reports an error.

func (*InputStep) MarshalYAML added in v0.4.1 

func (s *InputStep) MarshalYAML() (any, error)

MarshalYAML returns s.Scalar if it's not empty, otherwise s.Contents if that is not empty. If both s.Scalar and s.Contents are empty, it reports an error.

type InterpolationEnv added in v0.4.0 

type InterpolationEnv interface {
	Get(name string) (string, bool)
	Set(name string, value string)
}

InterpolationEnv contains environment variables that may be interpolated into a pipeline. Users may define an equivalence between environment variable name, for example the environment variable names may case-insensitive.

type Matrix 

type Matrix struct {
	Setup       MatrixSetup       `yaml:"setup"`
	Adjustments MatrixAdjustments `yaml:"adjustments,omitempty"`

	RemainingFields map[string]any `yaml:",inline"`
}

Matrix models the matrix specification for command steps.

func (*Matrix) MarshalJSON 

func (m *Matrix) MarshalJSON() ([]byte, error)

MarshalJSON is needed to use inlineFriendlyMarshalJSON, and reduces the representation to a single list if the matrix is simple.

func (*Matrix) MarshalYAML 

func (m *Matrix) MarshalYAML() (any, error)

MarshalYAML is needed to reduce the representation to a single slice if the matrix is simple.

func (*Matrix) UnmarshalOrdered 

func (m *Matrix) UnmarshalOrdered(o any) error

UnmarshalOrdererd unmarshals from either []any or *ordered.MapSA.

type MatrixAdjustment 

type MatrixAdjustment struct {
	With MatrixAdjustmentWith `yaml:"with"`
	Skip any                  `yaml:"skip,omitempty"`

	RemainingFields map[string]any `yaml:",inline"` // NB: soft_fail is in the remaining fields
}

MatrixAdjustment models an adjustment - a combination of (possibly new) matrix values, and skip/soft fail configuration.

func (*MatrixAdjustment) MarshalJSON 

func (ma *MatrixAdjustment) MarshalJSON() ([]byte, error)

MarshalJSON is needed to use inlineFriendlyMarshalJSON.

func (*MatrixAdjustment) ShouldSkip 

func (ma *MatrixAdjustment) ShouldSkip() bool

type MatrixAdjustmentWith 

type MatrixAdjustmentWith map[string]string

MatrixAdjustmentWith is either a map of dimension key -> dimension value, or a single value (for single anonymous dimension matrices).

func (MatrixAdjustmentWith) MarshalJSON 

func (maw MatrixAdjustmentWith) MarshalJSON() ([]byte, error)

MarshalJSON returns either a single scalar or an object.

func (MatrixAdjustmentWith) MarshalYAML 

func (maw MatrixAdjustmentWith) MarshalYAML() (any, error)

MarshalYAML returns either a single scalar or a map.

func (*MatrixAdjustmentWith) UnmarshalOrdered 

func (maw *MatrixAdjustmentWith) UnmarshalOrdered(o any) error

UnmarshalOrdered unmarshals from either a scalar value (string, bool, or int) or *ordered.MapSA.

type MatrixAdjustments 

type MatrixAdjustments []*MatrixAdjustment

MatrixAdjustments is a set of adjustments.

type MatrixPermutation 

type MatrixPermutation map[string]string

MatrixPermutation represents a possible permutation of a matrix.

type MatrixSetup 

type MatrixSetup map[string][]string

MatrixSetup is the main setup of a matrix - one or more dimensions. The cross product of the dimensions in the setup produces the base combinations of matrix values.

func (MatrixSetup) MarshalJSON 

func (ms MatrixSetup) MarshalJSON() ([]byte, error)

MarshalJSON returns either a list (if the setup is a single anonymous dimension) or an object (if it contains one or more (named) dimensions).

func (MatrixSetup) MarshalYAML 

func (ms MatrixSetup) MarshalYAML() (any, error)

MarshalYAML returns either a Scalars (if the setup is a single anonymous dimension) or a map (if it contains one or more (named) dimensions).

func (*MatrixSetup) UnmarshalOrdered 

func (ms *MatrixSetup) UnmarshalOrdered(o any) error

UnmarshalOrdered unmarshals from either []any or *ordered.MapSA.

type Options added in v0.9.0 

type Options func(*Pipeline)

Options are functional options for creating a new Env.

type Pipeline 

type Pipeline struct {
	Steps Steps          `yaml:"steps"`
	Env   *ordered.MapSS `yaml:"env,omitempty"`

	// RemainingFields stores any other top-level mapping items so they at least
	// survive an unmarshal-marshal round-trip.
	RemainingFields map[string]any `yaml:",inline"`
}

Pipeline models a pipeline.

Standard caveats apply - see the package comment.

func Parse 

func Parse(src io.Reader) (*Pipeline, error)

Parse parses a pipeline. It does not apply interpolation. Warnings are passed through the err return: 

p, err := Parse(src)
if w := warning.As(err); w != nil {
	// Here are some warnings that should be shown
	log.Printf("*Warning* - pipeline is not fully parsed:\n%v", w)
} else if err != nil {
    // Parse could not understand src at all
    return err
}
// Use p

func (*Pipeline) Interpolate 

func (p *Pipeline) Interpolate(interpolationEnv InterpolationEnv, preferRuntimeEnv bool) error

Interpolate interpolates variables defined in both interpolationEnv and p.Env into the pipeline. More specifically, it does these things:

  • Interpolate pipeline.Env and copy the results into interpolationEnv, provided they don't conflict, to apply later.
  • Interpolate any string value in the rest of the pipeline.

By default if an environment variable exists in both the runtime and pipeline env we will substitute with the pipeline env IF the pipeline env is defined first. Setting the preferRuntimeEnv option to true instead prefers the runtime environment to pipeline environment variables when both are defined.

func (*Pipeline) MarshalJSON 

func (p *Pipeline) MarshalJSON() ([]byte, error)

MarshalJSON marshals a pipeline to JSON. Special handling is needed because yaml.v3 has "inline" but encoding/json has no concept of it.

func (*Pipeline) UnmarshalOrdered 

func (p *Pipeline) UnmarshalOrdered(o any) error

UnmarshalOrdered unmarshals the pipeline from either []any (a legacy sequence of steps) or *ordered.MapSA (a modern pipeline configuration).

type Plugin 

type Plugin struct {
	Source string
	Config any
}

Plugin models plugin configuration.

Standard caveats apply - see the package comment.

func (*Plugin) FullSource 

func (p *Plugin) FullSource() string

FullSource attempts to canonicalise Source. If it fails, it returns Source unaltered. Otherwise, it resolves sources in a manner described at https://buildkite.com/docs/plugins/using#plugin-sources.

func (*Plugin) MarshalJSON 

func (p *Plugin) MarshalJSON() ([]byte, error)

MarshalJSON returns the plugin in "one-key object" form. Plugin sources are marshalled into "full" form. Plugins originally specified as a single string (no config, only source) are canonicalised into "one-key object" with config null.

func (*Plugin) MarshalYAML 

func (p *Plugin) MarshalYAML() (any, error)

MarshalYAML returns the plugin in either "one-item map" form. Plugin sources are marshalled into "full" form. Plugins originally specified as a single string (no config, only source) are canonicalised into "one-item map" with config nil.

type Plugins 

type Plugins []*Plugin

Plugins is a sequence of plugins. It is useful for unmarshaling.

func (*Plugins) UnmarshalJSON 

func (p *Plugins) UnmarshalJSON(b []byte) error

UnmarshalJSON is used mainly to normalise the BUILDKITE_PLUGINS env var.

func (*Plugins) UnmarshalOrdered 

func (p *Plugins) UnmarshalOrdered(o any) error

UnmarshalOrdered unmarshals Plugins from either

  • []any - originally a sequence of "one-item mappings" (normal form), or
  • *ordered.MapSA - a mapping (where order is important...non-normal form).

"plugins" is supposed to be a sequence of one-item maps, since order matters. But some people (even us) write plugins into one big mapping and rely on order preservation.

type Signature 

type Signature struct {
	Algorithm    string   `json:"algorithm" yaml:"algorithm"`
	SignedFields []string `json:"signed_fields" yaml:"signed_fields"`
	Value        string   `json:"value" yaml:"value"`
}

Signature models a signature (on a step, etc).

type Step 

type Step interface {
	// contains filtered or unexported methods
}

Step models a step in the pipeline. It will be a pointer to one of: - CommandStep - WaitStep - InputStep - TriggerStep - GroupStep

func NewScalarStep 

func NewScalarStep(s string) (Step, error)

NewScalarStep returns a Step that can be represented as a single string. Currently these are "wait", "block", "input", and some deprecated variations ("waiter", "manual"). If it is any other string, NewScalarStep returns an UnknownStep containing an error wrapping ErrUnknownStepType.

type Steps 

type Steps []Step

Steps contains multiple steps. It is useful for unmarshaling step sequences, since it has custom logic for determining the correct step type.

func (*Steps) UnmarshalOrdered 

func (s *Steps) UnmarshalOrdered(o any) error

UnmarshalOrdered unmarshals a slice ([]any) into a slice of steps.

type TriggerStep 

type TriggerStep struct {
	Contents map[string]any `yaml:",inline"`
}

TriggerStep models a trigger step.

Standard caveats apply - see the package comment.

func (TriggerStep) MarshalJSON 

func (t TriggerStep) MarshalJSON() ([]byte, error)

MarshalJSON marshals the contents of the step.

type UnknownStep 

type UnknownStep struct {
	Contents any
}

UnknownStep models any step we don't know how to represent in this version. When future step types are added, they should be parsed with more specific types. UnknownStep is present to allow older parsers to preserve newer pipelines.

func (*UnknownStep) MarshalJSON 

func (u *UnknownStep) MarshalJSON() ([]byte, error)

MarshalJSON marshals the contents of the step.

func (*UnknownStep) MarshalYAML 

func (u *UnknownStep) MarshalYAML() (any, error)

MarshalYAML returns the contents of the step.

func (*UnknownStep) UnmarshalOrdered 

func (u *UnknownStep) UnmarshalOrdered(src any) error

UnmarshalOrdered unmarshals an unknown step.

type WaitStep 

type WaitStep struct {
	Scalar   string         `yaml:"-"`
	Contents map[string]any `yaml:",inline"`
}

WaitStep models a wait step.

Standard caveats apply - see the package comment.

func (*WaitStep) MarshalJSON 

func (s *WaitStep) MarshalJSON() ([]byte, error)

MarshalJSON marshals a wait step as "wait" if the step is empty, or as the s.Scalar if it is not empty, or as s.Contents.

func (*WaitStep) MarshalYAML added in v0.4.1 

func (s *WaitStep) MarshalYAML() (any, error)

MarshalYAML returns a wait step as "wait" if the step is empty, or as the s.Scalar if it is not empty, or as s.Contents.

Source Files

View all Source files

Directories

Path Synopsis
internal
env
Package env contains data structures and methods to assist with managing environment variables.
 Package env contains data structures and methods to assist with managing environment variables.
Package jwkutil provides utilities for working with JSON Web Keys and JSON Web Key Sets as defined in [RFC 7517].
 Package jwkutil provides utilities for working with JSON Web Keys and JSON Web Key Sets as defined in [RFC 7517].
Package ordered implements an ordered map type, suitable for use when decoding yaml documents where mappings must be preserved, such as in certain parts of the buildkite pipeline yaml.
 Package ordered implements an ordered map type, suitable for use when decoding yaml documents where mappings must be preserved, such as in certain parts of the buildkite pipeline yaml.
Package signature implements signing and verification of pipeline steps.
 Package signature implements signing and verification of pipeline steps.
Package warning provides a warning error type (a "soft" multi-error).
 Package warning provides a warning error type (a "soft" multi-error).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.