pipeline

Documentation

Overview

Package pipeline implements the pieces necessary for the agent to work with pipelines (typically in YAML or JSON form).

The pipeline object model (Pipeline, Steps, Plugin, etc) have these caveats:

• It is incomplete: there may be fields accepted by the API that are not listed. Do not treat Pipeline, CommandStep, etc, as comprehensive reference guides for how to write a pipeline.
• It normalises: unmarshaling accepts a variety of step forms, but marshaling back out produces more normalised output. An unmarshal/marshal round-trip may produce different output.
• It is non-canonical: using the object model does not guarantee that a pipeline will be accepted by the pipeline upload API.

Index

• Constants
• type CommandStep
  • func (c *CommandStep) MarshalJSON() ([]byte, error)
  • func (c *CommandStep) SignedFields() (map[string]string, error)
  • func (c *CommandStep) UnmarshalJSON(b []byte) error
  • func (c *CommandStep) UnmarshalOrdered(src any) error
  • func (c *CommandStep) ValuesForFields(fields []string) (map[string]string, error)
• type GroupStep
  • func (g *GroupStep) MarshalJSON() ([]byte, error)
  • func (g *GroupStep) UnmarshalOrdered(src any) error
• type InputStep
  • func (s *InputStep) MarshalJSON() ([]byte, error)
• type Pipeline
  • func Parse(src io.Reader) (*Pipeline, error)
  • func (p *Pipeline) Interpolate(envMap *env.Environment) error
  • func (p *Pipeline) MarshalJSON() ([]byte, error)
  • func (p *Pipeline) Sign(key jwk.Key) error
  • func (p *Pipeline) UnmarshalOrdered(o any) error
• type Plugin
  • func (p *Plugin) FullSource() string
  • func (p *Plugin) MarshalJSON() ([]byte, error)
  • func (p *Plugin) MarshalYAML() (any, error)
• type Plugins
  • func (p *Plugins) UnmarshalJSON(b []byte) error
  • func (p *Plugins) UnmarshalOrdered(o any) error
• type Signature
  • func Sign(env map[string]string, sf SignedFielder, key jwk.Key) (*Signature, error)
  • func (s *Signature) Verify(env map[string]string, sf SignedFielder, keySet jwk.Set) error
• type SignedFielder
• type Step
  • func NewScalarStep(s string) (Step, error)
• type Steps
  • func (s *Steps) UnmarshalOrdered(o any) error
• type TriggerStep
  • func (t TriggerStep) MarshalJSON() ([]byte, error)
• type UnknownStep
  • func (u UnknownStep) MarshalJSON() ([]byte, error)
  • func (u UnknownStep) MarshalYAML() (any, error)
  • func (u *UnknownStep) UnmarshalOrdered(src any) error
• type WaitStep
  • func (s *WaitStep) MarshalJSON() ([]byte, error)

Constants

const EnvNamespacePrefix = "env::"

EnvNamespacePrefix is the string that prefixes all fields in the "env" namespace. This is used to separate signed data that came from the environment from data that came from an object.

Types

type CommandStep

type CommandStep struct {
	Command   string            `yaml:"command"`
	Plugins   Plugins           `yaml:"plugins,omitempty"`
	Env       map[string]string `yaml:"env,omitempty"`
	Signature *Signature        `yaml:"signature,omitempty"`
	Matrix    any               `yaml:"matrix,omitempty"`

	// RemainingFields stores any other top-level mapping items so they at least
	// survive an unmarshal-marshal round-trip.
	RemainingFields map[string]any `yaml:",inline"`
}

CommandStep models a command step.

Standard caveats apply - see the package comment.

func (*CommandStep) MarshalJSON

func (c *CommandStep) MarshalJSON() ([]byte, error)

MarshalJSON marshals the step to JSON. Special handling is needed because yaml.v3 has "inline" but encoding/json has no concept of it.

func (*CommandStep) SignedFields

func (c *CommandStep) SignedFields() (map[string]string, error)

SignedFields returns the default fields for signing.

func (*CommandStep) UnmarshalJSON

func (c *CommandStep) UnmarshalJSON(b []byte) error

UnmarshalJSON is used when unmarshalling an individual step directly, e.g. from the Agent API Accept Job.

func (*CommandStep) UnmarshalOrdered

func (c *CommandStep) UnmarshalOrdered(src any) error

UnmarshalOrdered unmarshals a command step from an ordered map.

func (*CommandStep) ValuesForFields

func (c *CommandStep) ValuesForFields(fields []string) (map[string]string, error)

ValuesForFields returns the contents of fields to sign.

type GroupStep

type GroupStep struct {
	// Group is typically a key with no value. Since it must always exist in
	// a group step, here it is.
	Group *string `yaml:"group"`

	Steps Steps `yaml:"steps"`

	// RemainingFields stores any other top-level mapping items so they at least
	// survive an unmarshal-marshal round-trip.
	RemainingFields map[string]any `yaml:",inline"`
}

GroupStep models a group step.

Standard caveats apply - see the package comment.

func (*GroupStep) MarshalJSON

func (g *GroupStep) MarshalJSON() ([]byte, error)

MarshalJSON marshals the step to JSON. Special handling is needed because yaml.v3 has "inline" but encoding/json has no concept of it.

func (*GroupStep) UnmarshalOrdered

func (g *GroupStep) UnmarshalOrdered(src any) error

UnmarshalOrdered unmarshals a group step from an ordered map.

type InputStep

type InputStep struct {
	Scalar   string         `yaml:"-"`
	Contents map[string]any `yaml:",inline"`
}

InputStep models a block or input step.

Standard caveats apply - see the package comment.

func (*InputStep) MarshalJSON

func (s *InputStep) MarshalJSON() ([]byte, error)

type Pipeline

type Pipeline struct {
	Steps Steps          `yaml:"steps"`
	Env   *ordered.MapSS `yaml:"env,omitempty"`

	// RemainingFields stores any other top-level mapping items so they at least
	// survive an unmarshal-marshal round-trip.
	RemainingFields map[string]any `yaml:",inline"`
}

Pipeline models a pipeline.

Standard caveats apply - see the package comment.

func Parse

func Parse(src io.Reader) (*Pipeline, error)

Parse parses a pipeline. It does not apply interpolation.

func (*Pipeline) Interpolate

func (p *Pipeline) Interpolate(envMap *env.Environment) error

Interpolate interpolates variables defined in both envMap and p.Env into the pipeline. More specifically, it does these things:

• Copy tracing context keys from envMap into pipeline.Env.
• Interpolate pipeline.Env and copy the results into envMap to apply later.
• Interpolate any string value in the rest of the pipeline.

func (*Pipeline) MarshalJSON

func (p *Pipeline) MarshalJSON() ([]byte, error)

MarshalJSON marshals a pipeline to JSON. Special handling is needed because yaml.v3 has "inline" but encoding/json has no concept of it.

func (*Pipeline) Sign

func (p *Pipeline) Sign(key jwk.Key) error

Sign signs each signable part of the pipeline. Currently this is limited to command steps (including command steps within group steps), including all plugin configurations and the pipeline "env". Parts of the pipeline are mutated directly, so an error part-way through may leave some steps un-signed.

func (*Pipeline) UnmarshalOrdered

func (p *Pipeline) UnmarshalOrdered(o any) error

UnmarshalOrdered unmarshals the pipeline from either []any (a legacy sequence of steps) or *ordered.MapSA (a modern pipeline configuration).

type Plugin

type Plugin struct {
	Source string
	Config any
}

Plugin models plugin configuration.

Standard caveats apply - see the package comment.

func (*Plugin) FullSource

func (p *Plugin) FullSource() string

FullSource attempts to canonicalise Source. If it fails, it returns Source unaltered. Otherwise, it resolves sources in a manner described at https://buildkite.com/docs/plugins/using#plugin-sources.

func (*Plugin) MarshalJSON

func (p *Plugin) MarshalJSON() ([]byte, error)

MarshalJSON returns the plugin in "one-key object" form, or "single string" form (no config, only plugin source). Plugin sources are marshalled into "full" form.

func (*Plugin) MarshalYAML

func (p *Plugin) MarshalYAML() (any, error)

MarshalYAML returns the plugin in either "one-item map" form, or "scalar" form (no config, only plugin source). Plugin sources are marshalled into "full" form.

type Plugins

type Plugins []*Plugin

Plugins is a sequence of plugins. It is useful for unmarshaling.

func (*Plugins) UnmarshalJSON

func (p *Plugins) UnmarshalJSON(b []byte) error

UnmarshalJSON is used mainly to normalise the BUILDKITE_PLUGINS env var.

func (*Plugins) UnmarshalOrdered

func (p *Plugins) UnmarshalOrdered(o any) error

UnmarshalOrdered unmarshals Plugins from either

• []any - originally a sequence of "one-item mappings" (normal form), or
• *ordered.MapSA - a mapping (where order is important...non-normal form).

"plugins" is supposed to be a sequence of one-item maps, since order matters. But some people (even us) write plugins into one big mapping and rely on order preservation.

type Signature

type Signature struct {
	Algorithm    string   `json:"algorithm" yaml:"algorithm"`
	SignedFields []string `json:"signed_fields" yaml:"signed_fields"`
	Value        string   `json:"value" yaml:"value"`
}

Signature models a signature (on a step, etc).

func Sign

func Sign(env map[string]string, sf SignedFielder, key jwk.Key) (*Signature, error)

Sign computes a new signature for an environment (env) combined with an object containing values (sf) using a given key.

func (*Signature) Verify

func (s *Signature) Verify(env map[string]string, sf SignedFielder, keySet jwk.Set) error

Verify verifies an existing signature against environment (env) combined with an object containing values (sf) using keys from a keySet.

type SignedFielder

type SignedFielder interface {
	// SignedFields returns the default set of fields to sign, and their values.
	// This is called by Sign.
	SignedFields() (map[string]string, error)

	// ValuesForFields looks up each field and produces a map of values. This is
	// called by Verify. The set of fields might differ from the default, e.g.
	// when verifying older signatures computed with fewer fields or deprecated
	// field names. signedFielder implementations should reject requests for
	// values if "mandatory" fields are missing (e.g. signing a command step
	// should always sign the command).
	ValuesForFields([]string) (map[string]string, error)
}

SignedFielder describes types that can be signed and have signatures verified. Converting non-string fields into strings (in a stable, canonical way) is an exercise left to the implementer.

type Step

type Step interface {
	// contains filtered or unexported methods
}

Step models a step in the pipeline. It will be a pointer to one of: - CommandStep - WaitStep - InputStep - TriggerStep - GroupStep

func NewScalarStep

func NewScalarStep(s string) (Step, error)

type Steps

type Steps []Step

Steps contains multiple steps. It is useful for unmarshaling step sequences, since it has custom logic for determining the correct step type.

func (*Steps) UnmarshalOrdered

func (s *Steps) UnmarshalOrdered(o any) error

UnmarshalOrdered unmarshals a slice ([]any) into a slice of steps.

type TriggerStep

type TriggerStep struct {
	Contents map[string]any `yaml:",inline"`
}

TriggerStep models a trigger step.

Standard caveats apply - see the package comment.

func (TriggerStep) MarshalJSON

func (t TriggerStep) MarshalJSON() ([]byte, error)

MarshalJSON marshals the contents of the step.

type UnknownStep

type UnknownStep struct {
	Contents any
}

UnknownStep models any step we don't know how to represent in this version. When future step types are added, they should be parsed with more specific types. UnknownStep is present to allow older parsers to preserve newer pipelines.

func (UnknownStep) MarshalJSON

func (u UnknownStep) MarshalJSON() ([]byte, error)

MarshalJSON marshals the contents of the step.

func (UnknownStep) MarshalYAML

func (u UnknownStep) MarshalYAML() (any, error)

MarshalYAML returns the contents of the step.

func (*UnknownStep) UnmarshalOrdered

func (u *UnknownStep) UnmarshalOrdered(src any) error

UnmarshalOrdered unmarshals an unknown step.

type WaitStep

type WaitStep struct {
	Scalar   string         `yaml:"-"`
	Contents map[string]any `yaml:",inline"`
}

WaitStep models a wait step.

Standard caveats apply - see the package comment.

func (*WaitStep) MarshalJSON

func (s *WaitStep) MarshalJSON() ([]byte, error)

MarshalJSON marshals a wait step as "wait" if w is empty, or as the step's scalar if it's set. If scalar is empty, it marshals as the remaining fields