build

package
v0.0.0-...-d0ced46 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 22, 2024 License: Apache-2.0 Imports: 39 Imported by: 4

Documentation

Overview

Package build implements a minimal, safe, easy-to-use, hard-to-use-wrong, 'luciexe binary' implementation in Go.

See `Start` for the entrypoint to this API.

Features:

  • Can be used in library code which is used in both luciexe and non-luciexe contexts. You won't need "two versions" of your library just to give it the appearance of steps in a LUCI build context, or to have it read or write build properties.
  • Goroutine-safe API.
  • Handles all interactions with the log sink (e.g. LogDog), and automatically reroutes the logging library if a log sink is configured (so all log messages will be preserved in a build context).
  • Handles coalescing all build mutations from multiple goroutines. Efficient throttling for outgoing updates.
  • Automatically upholds all luciexe protocol invariants; Avoids invalid Build states by construction.
  • Automatically maps errors and panics to their obvious visual representation in the build (and allows manual control in case you want something else).
  • Fully decoupled from LUCI service implementation; You can use this API with no external services (e.g. for a CLI application), backed with the real LUCI implementation (e.g. when running under BuildBucket), and in tests (you can observe all outputs from this API without any live services or heavy mocks).

No-Op Mode

When no Build is in use in the context, the library behaves in 'no-op' mode. This should enable libraries to add `build` features which gracefully degrade into pure terminal output via logging.

  • Registered Properties functions will return when using GetInput, and will panic when using SetOutput/MutateOutput. However, you may explicitly use `build.Properties.Instantiate().SetInContext(ctx)` to initialize a property state in `ctx`. If desired, you can also provide an input value and/or notification callback to build.Properties.Instantiate. The notification callback could log manipulated output properties, or save them to a file, etc.
  • There will be no *State object, because there is no Start call.
  • StartStep/ScheduleStep will return a *Step which is detached. Step namespacing will still work in context (but name deduplication will not).
  • The result of State.Modify/Step.Modify (and Set) calls will be logged at DEBUG.
  • Step scheduled/started/ended messages will be logged at INFO. Ended log messages will include the final summary markdown as well.
  • Text logs will be logged line-by-line at INFO with fields set indicating which step and log they were emitted from. Debug text logs (those whose log names start with "$") will be logged at DEBUG level.
  • Non-text logs will be dropped with a WARNING indicating that they're being dropped.

Index

Constants

This section is empty.

Variables

View Source 
var Properties = &properties.Registry{}

Properties is the canonical registry for all `build` input and output properties.

Functions

func AttachStatus 

func AttachStatus(err error, status bbpb.Status, details *bbpb.StatusDetails) error

AttachStatus attaches a buildbucket status (and details) to a given error.

If such an error is handled by Step.End or State.End, that step/build will have its status updated to match.

AttachStatus allows overriding the attached status if the error already has one.

This panics if the status is a non-terminal status like SCHEDULED or STARTED.

This is a no-op if the error is nil.

func ExtractStatus 

func ExtractStatus(err error) (bbpb.Status, *bbpb.StatusDetails)

ExtractStatus retrieves the Buildbucket status (and details) from a given error.

This returns:

  • (SUCCESS, nil) on nil error
  • Any values attached with AttachStatus.
  • (CANCELED, nil) on context.Canceled
  • (INFRA_FAILURE, &bbpb.StatusDetails{Timeout: {}}) on context.DeadlineExceeded
  • (FAILURE, nil) otherwise

This function is used internally by Step.End and State.End, but is provided publically for completeness.

func Main 

func Main(cb func(context.Context, []string, *State) error)

Main implements all the 'command-line' behaviors of the luciexe 'exe' protocol, including:

  • parsing command line for "--output", "--help", etc.
  • parsing stdin (as appropriate) for the incoming Build message
  • creating and configuring a logdog client to send State evolutions.
  • Configuring a logdog client from the environment.
  • Writing Build state updates to the logdog "build.proto" stream.
  • Start'ing the build in this process.
  • End'ing the build with the returned error from your function

Input and Output properties can be manipulated by registering property schemas with the package-level Properties object.

CLI Arguments parsed:

  • -h / --help : Print help for this binary (including input/output property type info)
  • --output : luciexe "output" flag; See https://pkg.go.dev/go.chromium.org/luci/luciexe#hdr-Recursive_Invocation
  • --working-dir : The working directory to run from; Default is $PWD unless LUCIEXE_FAKEBUILD is set, in which case a temp dir is used and cleaned up after.
  • -- : Any extra arguments after a "--" token are passed to your callback as-is.

Example: 

// MyProps will serve as both the input and output properties schema. If you
// want to have differing schemas, you can use RegisterSplitProperty. If you
// want to have Input-only or Output-only properties, see the
// Register{Input,Output}Property variants.
//
// See docs on RegisterProperty for more info.
var topLevelProps = build.RegisterProperty[*MyProps]("")

func main() {
  build.Main(func(ctx context.Context, args []string, st *build.State) error {
    // actual build code here, build is already Start'differing
    parsedInput := topLevelProps.GetInput(ctx) // returns *MyProps
    topLevelProps.MutateOutput(ctx, func(val *MyProps) (mutated bool) {
      // modify `val` however you like - if you need to copy some value from
      // it, use e.g. proto.Clone to copy it.
      return true
    })
    return nil // will mark the Build as SUCCESS
  })
}

Main also supports running a luciexe build locally by specifying the LUCIEXE_FAKEBUILD environment variable. The value of the variable should be a path to a file containing a JSON-encoded Build proto.

TODO(iannucci): LUCIEXE_FAKEBUILD does not support nested invocations. It should set up bbagent and butler in order to aggregate logs.

func RegisterInputProperty 

func RegisterInputProperty[T any](namespace string, opts ...properties.RegisterOption) properties.RegisteredPropertyIn[T]

RegisterInputProperty registers a schema for this namespace, but ONLY for input.

func RegisterOutputProperty 

func RegisterOutputProperty[T any](namespace string, opts ...properties.RegisterOption) properties.RegisteredPropertyOut[T]

RegisterOutputProperty registers a schema for this namespace, but ONLY for output.

func RegisterProperty 

func RegisterProperty[T any](namespace string, opts ...properties.RegisterOption) properties.RegisteredProperty[T, T]

RegisterProperty registers a proto message, a *struct or a map as the input and output schema for `namespace` in the Properties registry (in this package). The returned RegisteredProperty will allow you to read the decoded input or manipulate the output properties any time during the execution of the Build.

See go.chromium.org/luci/luciexe/build/properties for more details, but as a quick reference:

  • The namespace "" indicates that this will be the schema for the 'top level' property message.
  • Multiple registrations for the same namespace will panic, even if the schema types align. If you want multiple Go packages to have access to the same region of the Build properties, consider registering the schema in a single shared package, or exporting functions from that package to interact with the properties at a higher level than just exposing raw property access (which can potentially act as a global variable in your program).

`T` may be any of the following:

  • A proto.Message type (e.g. *MyMessage), in which case it will be serialized as JSONPB. *structpb.Struct is special-cased to have pass-through encoding, in case your program needs to do something particularly special with the raw *Struct message.
  • A non proto.Message pointer-to-struct (e.g. *MyStruct), in which case it will be serialized with encoding/json.
  • A map[string]<something> (e.g. map[string]any), in which case it will be serialized with encoding/json.

Example: 

	package mypkg

	// message MyProtoMsg {
	//   string setting = 1;
	//   repeated string output = 2;
	// }
	var ioProp = build.RegisterProperty[*MyProtoMsg]("$mypkg")

	// ReadInitialSetting returns the value of Setting for mypkg that the build
	// started with (i.e. was an input property).
	func ReadInitialSetting(ctx context.Context) string {
	  // returns exactly `*MyProtoMsg`. If there is no Build in `ctx`, this
	  // returns nil.
	  return ioProp.GetInput(ctx).GetSetting()
	}

	// AppendToOutput adds a string to mypkg's output.
	func AppendToOutput(ctx context.Context, newVal string) {
    ioProp.MutateOutput(ctx, func(state *MyProtoMsg) (mutated bool) {
      state.Output = append(state.Output, newVal)
      return true  // will cause the build properties to be sent
    }
	}

This is just a shorthand (and summarized documentation) for: 

properties.MustRegister[T](build.Properties, namespace, opts...)

func RegisterSplitProperty 

func RegisterSplitProperty[InT, OutT any](namespace string, opts ...properties.RegisterOption) properties.RegisteredProperty[InT, OutT]

RegisterSplitProperty registers differing input and output schemas for this namespace.

Types

type Log 

type Log struct {
	io.Writer
	// contains filtered or unexported fields
}

Log represents a step or build log. It can be written to directly, and also provides additional information about the log itself.

The creator of the Log is responsible for cleaning up any resources associated with it (e.g. the Step or State this was created from). 

func (l *Log) UILink() string

UILink returns a URL to this log fit for surfacing in the LUCI UI.

This may return an empty string if there's no available LogDog infra being logged to, for instance in testing or during local execution where logdog streams are not sunk to the actual logdog service.

type Loggable 

type Loggable interface {
	// Log creates a new log stream (by default, line-oriented text) with the
	// given name.
	//
	// To uphold the requirements of the Build proto message, duplicate log names
	// will be deduplicated with the same algorithm used for deduplicating step
	// names.
	//
	// To create a binary stream, pass streamclient.Binary() as one of the
	// options.
	//
	// The stream will close when the associated object (step or build) is End'd.
	Log(name string, opts ...streamclient.Option) *Log

	// LogDatagram creates a new datagram-oriented log stream with the given name.
	//
	// To uphold the requirements of the Build proto message, duplicate log names
	// will be deduplicated with the same algorithm used for deduplicating step
	// names.
	//
	// The stream will close when the associated object (step or build) is End'd.
	LogDatagram(name string, opts ...streamclient.Option) streamclient.DatagramWriter
}

Loggable is the common interface for build entities which have log data associated with them.

Implemented by State and Step.

Logs all have a name which is an arbitrary bit of text to identify the log to human users (it will appear as the link on the build UI page). In particular it does NOT need to conform to the LogDog stream name alphabet.

The log name "log" is reserved, and will automatically capture all logging outputs generated with the "go.chromium.org/luci/common/logging" API.

type StartOption 

type StartOption func(*State)

StartOption is an object which can be passed to the Start function, and modifies the behavior of the luciexe/build library.

StartOptions are exclusively constructed from the Opt* functions in this package.

StartOptions are all unique per Start (i.e. you can only pass one of a kind per option to Start).

func OptLogsink 

func OptLogsink(c *streamclient.Client) StartOption

OptLogsink allows you to associate a streamclient with the started build.

See `streamclient.New` and `streamclient.NewFake` for how to create a client suitable to your needs (note that this includes a local filesystem option).

If a logsink is configured, it will be used as the output destination for the go.chromium.org/luci/common/logging library, and will recieve all data written via the Loggable interface.

If no logsink is configured, the go.chromium.org/luci/common/logging library will be unaffected, and data written to the Loggable interface will go to an ioutil.NopWriteCloser.

func OptSend 

func OptSend(lim rate.Limit, callback func(int64, *bbpb.Build)) StartOption

OptSend allows you to get a callback when the state of the underlying Build changes.

This callback will be called at most as frequently as `rate` allows, up to once per Build change, and is called with the version number and a copy of Build. Only one outstanding invocation of this callback can occur at once.

If new updates come in while this callback is blocking, they will apply silently in the background, and as soon as the callback returns (and rate allows), it will be invoked again with the current Build state.

Every modification of the Build state increments the version number by one, even if it doesn't result in an invocation of the callback. If your program modifies the build state from multiple threads, then the version assignment is arbitrary, but if you make 10 parallel changes, you'll see the version number jump by 10 (and you may, or may not, observe versions in between).

type State 

type State struct {
	// contains filtered or unexported fields
}

State is the state of the current Build.

This is properly initialized with the Start function, and as long as it isn't "End"ed, you can manipulate it with the State's various methods.

The State is preserved in the context.Context for use with the ScheduleStep and StartStep functions. These will add a new manipulatable step to the build State.

All manipulations to the build State will result in an invocation of the configured Send function (see OptSend).

func Start 

func Start(ctx context.Context, initial *bbpb.Build, opts ...StartOption) (*State, context.Context, error)

Start is the 'inner' entrypoint to this library.

If you're writing a standalone luciexe binary, see `Main` and `MainWithOutput`.

This function clones `initial` as the basis of all state updates (see OptSend) and RegisterProperty declarations. This also initializes the build State and properties.State in `ctx` and returns the manipulable State object.

You must End the returned State. To automatically map errors and panics to their correct visual representation, End the State like: 

var err error
state, ctx := build.Start(ctx, initialBuild, ...)
defer func() { state.End(err) }()

err = opThatErrsOrPanics(ctx)

NOTE: A panic will still crash the program as usual. This does NOT `recover()` the panic. Please use conventional Go error handling and control flow mechanisms.

func (*State) Build 

func (s *State) Build() *bbpb.Build

Build returns a copy of the initial Build state.

This is useful to access fields such as Infra, Tags, Ancestor ids etc.

Changes to this copy will not reflect anywhere in the live Build state and not affect other calls to Build().

NOTE: It is recommended to use the PropertyModifier/PropertyReader functionality of this package to interact with Build Input Properties; They are encoded as Struct proto messages, which are extremely cumbersome to work with directly.

func (*State) End 

func (s *State) End(err error)

End sets the build's final status, according to `err` (See ExtractStatus).

End will also be able to set INFRA_FAILURE status and log additional information if the program is panic'ing.

End must be invoked like: 

var err error
state, ctx := build.Start(ctx, initialBuild, ...)
defer func() { state.End(err) }()

err = opThatErrsOrPanics(ctx)

NOTE: A panic will still crash the program as usual. This does NOT `recover()` the panic. Please use conventional Go error handling and control flow mechanisms.

func (*State) Log 

func (s *State) Log(name string, opts ...streamclient.Option) *Log

Log creates a new step-level line-oriented text log stream with the given name. Returns a Log value which can be written to directly, but also provides additional information about the log itself.

The stream will close when the state is End'd.

func (*State) LogDatagram 

func (s *State) LogDatagram(name string, opts ...streamclient.Option) streamclient.DatagramWriter

LogDatagram creates a new build-level datagram log stream with the given name. Each call to WriteDatagram will produce a single datagram message in the stream.

You must close the stream when you're done with it.

func (*State) Modify 

func (s *State) Modify(cb func(*View))

Modify allows you to atomically manipulate the View on the build State.

Blocking in Modify will block other callers of Modify and Set*, as well as the ability for the build State to be sent (with the function set by OptSend).

The Set* methods should be preferred unless you need to read/modify/write View items.

func (*State) SetCritical 

func (s *State) SetCritical(critical bbpb.Trinary)

SetCritical atomically sets the build's Critical field.

func (*State) SetGitilesCommit 

func (s *State) SetGitilesCommit(gc *bbpb.GitilesCommit)

SetGitilesCommit atomically sets the GitilesCommit.

This will make a copy of the GitilesCommit object to store in the build State.

func (*State) SetSummaryMarkdown 

func (s *State) SetSummaryMarkdown(summaryMarkdown string)

SetSummaryMarkdown atomically sets the build's SummaryMarkdown field.

func (*State) SynthesizeIOProto 

func (s *State) SynthesizeIOProto(o io.Writer) error

SynthesizeIOProto synthesizes a `.proto` file from the input and ouptut property messages declared at Start() time.

type Step 

type Step struct {
	// contains filtered or unexported fields
}

Step represents the state of a single step.

This is properly initialized by the StartStep and ScheduleStep functions.

func ScheduleStep 

func ScheduleStep(ctx context.Context, name string) (*Step, context.Context)

ScheduleStep is like StartStep, except that it leaves the new step in the SCHEDULED status, and does not set a StartTime.

The step will move to STARTED when calling any other methods on the Step, when creating a sub-Step, or if you explicitly call Step.Start().

func StartStep 

func StartStep(ctx context.Context, name string) (*Step, context.Context)

StartStep adds a new step to the build.

The step will have a "STARTED" status with a StartTime.

If `name` contains `|` this function will panic, since this is a reserved character for delimiting hierarchy in steps.

Duplicate step names will be disambiguated by appending " (N)" for the 2nd, 3rd, etc. duplicate.

The returned context has the following changes:

  1. It contains the returned *Step as the current step, which means that calling the package-level StartStep/ScheduleStep on it will create sub-steps of this one.
  2. The returned context also has an updated `environ.FromCtx` containing a unique $LOGDOG_NAMESPACE value. If you launch a subprocess, you should use this environment to correctly namespace any logdog log streams your subprocess attempts to open. Using `go.chromium.org/luci/luciexe/build/exec` does this automatically.
  3. `go.chromium.org/luci/common/logging` is wired up to a new step log stream called "log".

You MUST call Step.End. To automatically map errors and panics to their correct visual representation, End the Step like: 

var err error
step, ctx := build.StartStep(ctx, "Step name")
defer func() { step.End(err) }()

err = opThatErrsOrPanics(ctx)

NOTE: A panic will still crash the program as usual. This does NOT `recover()` the panic. Please use conventional Go error handling and control flow mechanisms.

func (*Step) AddTagValue 

func (s *Step) AddTagValue(key, value string)

AddTagValue sets the step's tag field by appending the new tag to the existing list.

func (*Step) End 

func (s *Step) End(err error)

End sets the step's final status, according to `err` (See ExtractStatus).

End will also be able to set INFRA_FAILURE status and log additional information if the program is panic'ing.

End'ing a Step will Cancel the context associated with this step (returned from StartStep or ScheduleStep).

End must be invoked like: 

var err error
step, ctx := build.StartStep(ctx, ...)  // or build.ScheduleStep
defer func() { step.End(err) }()

err = opThatErrsOrPanics()

NOTE: A panic will still crash the program as usual. This does NOT `recover()` the panic. Please use conventional Go error handling and control flow mechanisms.

func (*Step) Log 

func (s *Step) Log(name string, opts ...streamclient.Option) *Log

Log creates a new step-level line-oriented text log stream with the given name. Returns a Log value which can be written to directly, but also provides additional information about the log itself.

The stream will close when the step is End'd.

func (*Step) LogDatagram 

func (s *Step) LogDatagram(name string, opts ...streamclient.Option) streamclient.DatagramWriter

LogDatagram creates a new step-level datagram log stream with the given name. Each call to WriteDatagram will produce a single datagram message in the stream.

The stream will close when the step is End'd.

func (*Step) Modify 

func (s *Step) Modify(cb func(*StepView))

Modify allows you to atomically manipulate the StepView for this Step.

Blocking in Modify will block other callers of Modify and Set*, as well as the ability for the build State to be sent (with the function set by OptSend).

The Set* methods should be preferred unless you need to read/modify/write View items.

This starts the step if it's still SCHEDULED.

func (*Step) ScheduleStep 

func (s *Step) ScheduleStep(ctx context.Context, name string) (*Step, context.Context)

ScheduleStep will create a child step of this one with `name` in the SCHEDULED status.

This behaves identically to the package level ScheduleStep, except that the 'current step' is `s` and is not pulled from `ctx. This includes all documented behaviors around changes to the returned context.

func (*Step) SetSummaryMarkdown 

func (s *Step) SetSummaryMarkdown(summaryMarkdown string)

SetSummaryMarkdown atomically sets the step's SummaryMarkdown field.

func (*Step) Start 

func (s *Step) Start()

Start will change the status of this Step from SCHEDULED to STARTED and initializes StartTime.

This must only be called for ScheduleStep invocations. If the step is already started (e.g. it was produced via StartStep() or Start() was already called), this does nothing.

func (*Step) StartStep 

func (s *Step) StartStep(ctx context.Context, name string) (*Step, context.Context)

StartStep will create a child step of this one with `name`.

This behaves identically to the package level StartStep, except that the 'current step' is `s` and is not pulled from `ctx. This includes all documented behaviors around changes to the returned context.

type StepView 

type StepView struct {
	SummaryMarkdown string
	Tags            map[string][]string
}

StepView is a window into the build State.

You can obtain/manipulate this with the Step.Modify method.

type View 

type View struct {
	SummaryMarkdown string
	Critical        bbpb.Trinary
	GitilesCommit   *bbpb.GitilesCommit
}

View is a window into the build State.

You can obtain/manipulate this with the State.Modify method.

Notes

Bugs

  • When OptLogsink is used and `logging` output is redirected to a Step log entitled "log", the current log format is reset to `gologger.StdFormat` instead of preserving the current log format from the context.

Source Files

View all Source files

Directories

Path Synopsis
Package cv exposes CV properties to luciexe binaries for builds.
 Package cv exposes CV properties to luciexe binaries for builds.
internal
testpb
Package properties encapsulates all logic and data structures for parsing the input properties and manipulating the output properties of a LUCI Build.
 Package properties encapsulates all logic and data structures for parsing the input properties and manipulating the output properties of a LUCI Build.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.