build

package
v0.0.0-...-3b47d42 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 26, 2025 License: BSD-3-Clause Imports: 65 Imported by: 0

Documentation

Overview

Package build is the core package of the build tool.

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrNoTarget is an error when target is not found in the Graph.
	ErrNoTarget = errors.New("target not found")

	// ErrTargetIsSource is an error when target is the source in the Graph.
	ErrTargetIsSource = errors.New("target is source")

	// ErrDuplicateStep is an error when step for the target is already created.
	ErrDuplicateStep = errors.New("duplicate step")

	// ErrMissingDeps is an error when step failed to get deps.
	ErrMissingDeps = errors.New("missing deps in depfile/depslog")
)
View Source 
var ErrManifestModified = errors.New("manifest modified")

ErrManifestModified is an error to indicate that manifest is modified.

Functions

func DetectExecRoot 

func DetectExecRoot(execRoot, crdir string) (string, error)

DetectExecRoot detects exec root from path given config repo dir crdir.

func SetDefaultForTest 

func SetDefaultForTest(limits Limits)

SetDefaultForTest updates default limits for test. Test should restore the original value after the test.

func SetExperimentForTest 

func SetExperimentForTest(v string)

SetExperimentForTest sets experiments for testing.

Types

type ActiveStepInfo 

type ActiveStepInfo struct {
	ID    string
	Desc  string
	Phase string

	// step duration since when it's ready to run.
	Dur string

	// step accumulated service duration (local exec, or remote call).
	// not including local execution queue, remote execution queue,
	// but including all of remote call (reapi or reproxy), uploading
	// inputs, downloading outputs etc.
	ServDur string
}

type Builder 

type Builder struct {
	// contains filtered or unexported fields
}

Builder is a builder.

func New 

func New(ctx context.Context, graph Graph, opts Options) (*Builder, error)

New creates new builder.

func (*Builder) ActiveSteps 

func (b *Builder) ActiveSteps() []ActiveStepInfo

func (*Builder) Build 

func (b *Builder) Build(ctx context.Context, name string, args ...string) (err error)

Build builds args with the name.

func (*Builder) Close 

func (b *Builder) Close() error

Close cleans up the builder.

func (*Builder) Stats 

func (b *Builder) Stats() Stats

Stats returns stats of the builder.

func (*Builder) TraceStats 

func (b *Builder) TraceStats() []*TraceStat

TraceStats returns trace stats of the builder.

type Cache 

type Cache struct {
	// contains filtered or unexported fields
}

Cache is a cache used in the builder.

func NewCache 

func NewCache(ctx context.Context, opts CacheOptions) (*Cache, error)

NewCache creates new cache.

func (*Cache) GetActionResult 

func (c *Cache) GetActionResult(ctx context.Context, cmd *execute.Cmd) error

GetActionResult gets action result for the cmd from cache.

type CacheOptions 

type CacheOptions struct {
	Store      cachestore.CacheStore
	EnableRead bool
}

CacheOptions is cache options.

type DependencyCycleError 

type DependencyCycleError struct {
	Targets []string
}

DependencyCycleError is error type for dependency cycle.

func (DependencyCycleError) Error 

func (d DependencyCycleError) Error() string

type Experiments 

type Experiments struct {
	// contains filtered or unexported fields
}

Experiments manages experimental features. Experiments are enabled by SISO_EXPERIMENTS environment variable. We don't guarantee experiment id in future versions. Unknown experiment id will be ignored.

func (*Experiments) Enabled 

func (e *Experiments) Enabled(k, format string, args ...any) bool

Enabled returns true if experimental feature k is enabled, and log error once with its hint if so.

func (*Experiments) Hint 

func (e *Experiments) Hint(k string) string

Hint shows hint message for experimental feature k.

func (*Experiments) ShowOnce 

func (e *Experiments) ShowOnce()

ShowOnce shows once about enabled experimental features.

func (*Experiments) String 

func (e *Experiments) String() string

func (*Experiments) Suggest 

func (e *Experiments) Suggest(k string) string

Suggest returns suggest message to enable experimental feature k.

type Graph 

type Graph interface {
	// NumTargets returns number of valid taraget id.
	NumTargets() int

	// Targets returns target paths for given args.
	Targets(context.Context, ...string) ([]Target, error)

	// SpellcheckTarget returns the most similar target from given target.
	SpellcheckTarget(string) (string, error)

	// Validations returns validation targets detected by past StepDef calls.
	Validations() []Target

	// TargetPath returns exec-root relative path of target.
	TargetPath(context.Context, Target) (string, error)

	// StepDef creates new StepDef for the target and its inputs/orderOnly/outputs targets.
	// if err is ErrTargetIsSource, target is source and no step to
	// generate the target.
	// if err is ErrDuplicateStep, a step that geneartes the target
	// is already processed.
	StepDef(context.Context, Target, StepDef) (StepDef, []Target, []Target, []Target, error)

	// InputDeps returns input dependencies.
	// input dependencies is a map from input path or label to
	// other files or labels needed for the key.
	// path is exec root relative and label contains ':'.
	// it's "input_deps" in Starlark config.
	InputDeps(context.Context) map[string][]string

	// StepLimits returns a map of maximum number of concurrent
	// steps by pool name.
	StepLimits(context.Context) map[string]int

	// Filenames returns filenames of build manifest (all files loaded by build.ninja).
	Filenames() []string
}

Graph provides a build graph, i.e. step definitions.

type IntervalMetric 

type IntervalMetric time.Duration

IntervalMetric is a time duration, but serialized as seconds in JSON.

func (IntervalMetric) MarshalJSON 

func (i IntervalMetric) MarshalJSON() ([]byte, error)

MarshalJSON marshals the IntervalMetric as float64 of seconds.

func (*IntervalMetric) UnmarshalJSON 

func (i *IntervalMetric) UnmarshalJSON(b []byte) error

UnmarshalJSON unmarshals float64 of seconds as an IntervalMetric.

type LayeredCache 

type LayeredCache struct {
	// contains filtered or unexported fields
}

LayeredCache is a multi-layer cache. It will attempt to read from caches in priority order, and when doing so, performs write-through caching to all faster caches. When writing, it only writes to the first layer to improve performance.

func NewLayeredCache 

func NewLayeredCache() *LayeredCache

func (*LayeredCache) AddLayer 

func (lc *LayeredCache) AddLayer(cache cachestore.CacheStore)

AddLayer adds a layer to the cache. layered cache with an extra layer attached.

func (*LayeredCache) GetActionResult 

func (lc *LayeredCache) GetActionResult(ctx context.Context, d digest.Digest) (ar *rpb.ActionResult, err error)

GetActionResult gets the action result of the action identified by the digest.

func (*LayeredCache) GetContent 

func (lc *LayeredCache) GetContent(ctx context.Context, d digest.Digest, f string) (content []byte, err error)

GetContent gets the content of the file identified by the digest.

func (*LayeredCache) HasContent 

func (lc *LayeredCache) HasContent(ctx context.Context, d digest.Digest) bool

HasContent checks whether content of the digest exists in the cache.

func (*LayeredCache) SetActionResult 

func (lc *LayeredCache) SetActionResult(ctx context.Context, d digest.Digest, ar *rpb.ActionResult) error

SetActionResult sets the action result of the action identified by the digest. If a failing action is provided, caching will be skipped.

func (*LayeredCache) SetContent 

func (lc *LayeredCache) SetContent(ctx context.Context, d digest.Digest, f string, content []byte) error

SetContent sets the content of the file identified by the digest.

func (*LayeredCache) Source 

func (lc *LayeredCache) Source(ctx context.Context, d digest.Digest, f string) digest.Source

Source returns digest source for the name identified by the digest.

type Limits 

type Limits struct {
	Step      int
	Preproc   int
	ScanDeps  int
	Local     int
	FastLocal int
	Remote    int
	REWrap    int
	Cache     int
	Thread    int
}

Limits specifies the resource limits used in siso build process. zero limit means default.

func DefaultLimits 

func DefaultLimits(ctx context.Context) Limits

DefaultLimits returns default semaphore limits. It checks SISO_LIMITS environment variable to override limits. SISO_LIMITS is comma-separated <key>=<value> pair. e.g. 

SISO_LIMITS=step=1024,local=8,remote=80

func UnitTestLimits 

func UnitTestLimits(ctx context.Context) Limits

UnitTestLimits returns limits used in unit tests. It sets 2 for all limits. Otherwise, builder will start many steps, so hard to test !hasReady before b.failuresAllowed in Build func in builder.go

type LocalCache 

type LocalCache struct {
	// contains filtered or unexported fields
}

LocalCache implements CacheStore interface with local files.

func NewLocalCache 

func NewLocalCache(dir string) (*LocalCache, error)

NewLocalCache returns new local cache.

func (*LocalCache) GarbageCollectIfRequired 

func (c *LocalCache) GarbageCollectIfRequired(ctx context.Context)

GarbageCollectIfRequired performs garbage collection if it has not been performed within localCacheTTL.

func (*LocalCache) GetActionResult 

func (c *LocalCache) GetActionResult(ctx context.Context, d digest.Digest) (*rpb.ActionResult, error)

GetActionResult gets the action result of the action identified by the digest.

func (*LocalCache) GetContent 

func (c *LocalCache) GetContent(ctx context.Context, d digest.Digest, _ string) ([]byte, error)

GetContent returns content of the fname identified by the digest.

func (*LocalCache) HasContent 

func (c *LocalCache) HasContent(ctx context.Context, d digest.Digest) bool

HasContent checks whether content of the digest exists in the local cache.

func (*LocalCache) IOMetrics 

func (c *LocalCache) IOMetrics() *iometrics.IOMetrics

IOMetrics returns io metrics of the local cache.

func (*LocalCache) SetActionResult 

func (c *LocalCache) SetActionResult(ctx context.Context, d digest.Digest, ar *rpb.ActionResult) error

SetActionResult sets the action result of the action identified by the digest. If a failing action is provided, caching will be skipped.

func (*LocalCache) SetContent 

func (c *LocalCache) SetContent(ctx context.Context, d digest.Digest, fname string, buf []byte) error

SetContent sets content of fname identified by the digest.

func (*LocalCache) Source 

func (c *LocalCache) Source(_ context.Context, d digest.Digest, fname string) digest.Source

Source returns digest source for fname identified by the digest.

type MissingSourceError 

type MissingSourceError struct {
	Target   string
	NeededBy string
}

MissingSourceError is an error of missing source needed for build.

func (MissingSourceError) Error 

func (e MissingSourceError) Error() string

type Options 

type Options struct {
	JobID     string
	ID        string
	StartTime time.Time

	Metadata          metadata.Metadata
	ProjectID         string
	Path              *Path
	HashFS            *hashfs.HashFS
	REAPIClient       *reapi.Client
	RECacheEnableRead bool
	// TODO(b/266518906): enable RECacheEnableWrite option for read-only client.
	// RECacheEnableWrite bool
	ReproxyAddr string
	ActionSalt  []byte

	OutputLocal          OutputLocalFunc
	Cache                *Cache
	FailureSummaryWriter io.Writer
	FailedCommandsWriter io.Writer
	OutputLogWriter      io.Writer
	ExplainWriter        io.Writer
	LocalexecLogWriter   io.Writer
	MetricsJSONWriter    io.Writer
	NinjaLogWriter       io.Writer
	TraceJSON            string
	Pprof                string
	TraceExporter        *trace.Exporter
	PprofUploader        *sisopprof.Uploader
	ResultstoreUploader  *resultstore.Uploader

	// Clobber forces to rebuild ignoring existing generated files.
	Clobber bool

	// Build inputs of targets, but not build targets itself.
	Prepare bool

	// Verbose shows all command lines while building rather than step description.
	Verbose bool

	// VerboseFailures shows failed command lines.
	VerboseFailures bool

	// DryRun just prints the command to build, but does nothing.
	DryRun bool

	// allow failures at most FailuresAllowed.
	FailuresAllowed int

	// don't delete @response files on success
	KeepRSP bool

	// RebuildManifest is a build manifest filename (i.e. build.ninja)
	// when rebuilding manifest.
	// empty for normal build.
	RebuildManifest string

	// Limits specifies resource limits.
	Limits Limits
}

Options is builder options.

type OutputLocalFunc 

type OutputLocalFunc func(context.Context, string) bool

OutputLocalFunc is a function to determine the file should be downloaded or not.

type Path 

type Path struct {
	ExecRoot string
	Dir      string // relative to ExecRoot, use slashes
	// contains filtered or unexported fields
}

Path manages paths used by the build.

func NewPath 

func NewPath(execRoot, dir string) *Path

NewPath returns new path for the build.

func (*Path) AbsFromWD 

func (p *Path) AbsFromWD(path string) string

AbsFromWD converts cwd relative to absolute path.

func (*Path) Check 

func (p *Path) Check() error

Check checks the path is valid.

func (*Path) FromWD 

func (p *Path) FromWD(path string) (string, error)

FromWD converts cwd relative to exec root relative, slash-separated. It keeps absolute path if it is out of exec root.

func (*Path) Intern 

func (p *Path) Intern(path string) string

Intern interns the path.

func (*Path) MaybeFromWD 

func (p *Path) MaybeFromWD(ctx context.Context, path string) string

MaybeFromWD attempts to convert cwd relative to exec root relative. It logs an error and returns the path as-is if this fails.

func (*Path) MaybeToWD 

func (p *Path) MaybeToWD(ctx context.Context, path string) string

MaybeToWD converts exec root relative to cwd relative, slash-separated. It keeps absolute path as is. It logs an error and returns the path as-is if this fails.

type Stats 

type Stats struct {
	Done            int // completed actions, including skipped, failed
	Fail            int // failed actions
	Pure            int // pure actions
	Skipped         int // skipped actions, because they were still up-to-date
	NoExec          int // actions that was completed by handler without execute cmds e.g. stamp, copy
	FastDepsSuccess int // actions that ran successfully when we used deps from the deps cache
	FastDepsFailed  int // actions that failed when we used deps from the deps cache
	ScanDepsFailed  int // actions that scandeps failed
	CacheHit        int // actions for which we got a cache hit
	Local           int // locally executed actions
	Remote          int // remote executed actions
	LocalFallback   int // actions for which remote execution failed, and we did a local fallback
	RemoteRetry     int // accumulated remote retry counts
	Total           int // total actions that ran during this build
}

Stats keeps statistics about the build, such as the number of total, skipped or remote actions.

type Step 

type Step struct {
	// contains filtered or unexported fields
}

Step is a build step.

func (*Step) Done 

func (s *Step) Done() bool

Done checks the step is done.

func (*Step) NumWaits 

func (s *Step) NumWaits() int

NumWaits returns number of waits for the step.

func (*Step) ReadyToRun 

func (s *Step) ReadyToRun(prev string, out Target) bool

ReadyToRun checks whether the step is ready to run when prev step's out becomes ready.

func (*Step) String 

func (s *Step) String() string

String returns id of the step.

type StepDef 

type StepDef interface {
	// String returns id of the step.
	String() string

	// Next returns next step's def.
	Next() StepDef

	// Ensure siso rule is applied
	EnsureRule(context.Context)

	// RuleName returns rule name of the step. empty for no rule.
	RuleName() string

	// ActionName returns action name of the step.
	ActionName() string

	// Args returns command line arguments of the step.
	Args(context.Context) []string

	// IsPhony returns true if the step is phony.
	IsPhony() bool

	// Binding returns binding value.
	Binding(string) string

	// Depfile returns exec-root relative depfile path, or empty if not set.
	Depfile(context.Context) string

	// Rspfile returns exec-root relative rspfile path, or empty if not set.
	Rspfile(context.Context) string

	// Inputs returns inputs of the step.
	Inputs(context.Context) []string

	// TriggerInputs returns inputs of the step that would trigger
	// the step's action.  no order-only.
	// For  deps in deps log, use DepInputs.
	TriggerInputs(context.Context) []string

	// DepInputs returns iterator for inputs via depfile of the step.
	// if depfile is not set, returns emptyIter, nil
	// if depfile or deplog is not found, returns wrapped ErrMissingDeps.
	// TODO: use iter.Seq[string] in go 1.23
	DepInputs(context.Context) (func(yield func(string) bool), error)

	// ToolInputs returns tool inputs of the step.
	// ToolInputs is added to deps inputs.
	ToolInputs(context.Context) []string

	// ExpandedCaseSensitives returns expanded filenames if platform is case-sensitive.
	ExpandedCaseSensitives(context.Context, []string) []string

	// ExpandedInputs returns expanded inputs of the step.
	ExpandedInputs(context.Context) []string

	// RemoteInputs maps file used in remote to file exists on local.
	// path in remote action -> local path
	RemoteInputs() map[string]string

	// REProxyConfig returns configuration options for using reproxy.
	REProxyConfig() *execute.REProxyConfig

	// CheckInputDeps checks dep can be found in its direct/indirect inputs.
	// Returns true if it is unknown bad deps, false otherwise.
	CheckInputDeps(context.Context, []string) (bool, error)

	// Handle runs a handler for the cmd.
	Handle(context.Context, *execute.Cmd) error

	// Outputs returns outputs of the step.
	Outputs(context.Context) []string

	// LocalOutputs returns outputs of the step that should be written to the local disk.
	LocalOutputs(context.Context) []string

	// Pure indicates the step is pure or not.
	Pure() bool

	// Platform returns platform properties for remote execution.
	Platform() map[string]string

	// RecordDeps records deps.
	RecordDeps(context.Context, string, time.Time, []string) (bool, error)

	// RuleFix returns required fix for the rule of the step.
	RuleFix(ctx context.Context, inadds, outadds []string) []byte
}

StepDef is a build step definition. unless specified, path is execroot relative.

type StepError 

type StepError struct {
	Target string
	Cause  error
}

StepError is step execution error.

func (StepError) Error 

func (e StepError) Error() string

func (StepError) Unwrap 

func (e StepError) Unwrap() error

type StepMetric 

type StepMetric struct {
	BuildID string `json:"build_id,omitempty"` // the unique id of the current build (trace)
	StepID  string `json:"step_id,omitempty"`  // the unique id of this step (top span)

	Rule     string `json:"rule,omitempty"`      // the rule name of the step
	Action   string `json:"action,omitempty"`    // the action name of the step
	Output   string `json:"output,omitempty"`    // the name of the first output file of the step
	GNTarget string `json:"gn_target,omitempty"` // inferred gn target

	// The ID and name of the first output of the previous step.
	// The "previous" step is defined as the last step that updated
	// the output that is used as part of this step's inputs.
	PrevStepID  string `json:"prev,omitempty"`
	PrevStepOut string `json:"prev_out,omitempty"`

	// Ready is the time it took since build start until the action became
	// ready for execution (= all inputs are available).
	Ready IntervalMetric `json:"ready,omitempty"`
	// Start is the time it took until Siso's scheduler was ready to work
	// on the step (concurrency limited by stepSema) and pass it to an
	// execution strategy.
	Start IntervalMetric `json:"start,omitempty"`
	// Duration is the time it took for the action to do its job, measured
	// from start of work until it is completed.
	// It includes siso-overhead (preproc etc) and command executon
	// (RunTime).
	// for full build metric, it's duration to process all scheduled steps.
	Duration IntervalMetric `json:"duration,omitempty"`

	// WeightedDuration is an estimate of the "true duration" of the action
	// that tries to accommodate for the impact of other actions running in
	// parallel. It is calculated by summing up small slices of time (~100ms)
	// while the action is running, where each slice's duration is divided by
	// the number of concurrently running actions at that point in time.
	WeightedDuration IntervalMetric `json:"weighted_duration,omitempty"`

	// The hash of the command-line of the build step.
	CmdHash string `json:"cmdhash,omitempty"`
	// The hash of the action proto of this build step.
	Digest string `json:"digest,omitempty"`

	DepsLog     bool `json:"deps_log,omitempty"`     // whether the action used the deps log.
	DepsLogErr  bool `json:"deps_log_err,omitempty"` // whether the action failed with deps log.
	ScandepsErr bool `json:"scandeps_err,omitempty"` // whether the action failed in scandeps.

	NoExec      bool `json:"no_exec,omitempty"`      // whether the action didn't run any command (i.e. just use handler).
	IsRemote    bool `json:"is_remote,omitempty"`    // whether the action uses remote result.
	IsLocal     bool `json:"is_local,omitempty"`     // whether the action uses local result.
	FastLocal   bool `json:"fast_local,omitempty"`   // whether the action chooses local for fast build.
	Cached      bool `json:"cached,omitempty"`       // whether the action was a cache hit.
	Fallback    bool `json:"fallback,omitempty"`     // whether the action failed remotely and was retried locally.
	Err         bool `json:"err,omitempty"`          // whether the action failed.
	RemoteRetry int  `json:"remote_retry,omitempty"` // count of remote retry

	// DepsScanTime is the time it took in calculating deps for cmd inputs.
	// TODO: set in reproxy mode too
	DepsScanTime IntervalMetric `json:"depsscan,omitempty"`

	// ActionStartTime is the time it took since build start until
	// the action starts. After ActionStartTime, scandeps/retry/fallback
	// may happen and there might be internal waiting time. e.g. remote
	// exec semaphore.
	ActionStartTime IntervalMetric `json:"action_start,omitempty"`
	// RunTime is the total duration of the action execution, including
	// overhead such as uploading / downloading files.
	RunTime IntervalMetric `json:"run,omitempty"`
	// QueueTime is the time it took until the worker could begin executing
	// the action.
	// TODO: set in reproxy mode too
	QueueTime IntervalMetric `json:"queue,omitempty"`
	// ExecStartTime is set if the action was not cached, containing the time
	// measured when the execution strategy started the process.
	ExecStartTime IntervalMetric `json:"exec_start,omitempty"`
	// ExecTime is the time measured from the execution strategy starting
	// the process until the process exited.
	ExecTime IntervalMetric `json:"exec,omitempty"`
	// ActionEndTime is the time it took since build start until
	// the action completes.
	ActionEndTime IntervalMetric `json:"action_end,omitempty"`

	Inputs  int `json:"inputs,omitempty"`  // how many input files.
	Outputs int `json:"outputs,omitempty"` // how many output files.

	// resource used by local process.
	MaxRSS  int64          `json:"max_rss,omitempty"` // max rss in local cmd.
	Majflt  int64          `json:"majflt,omitempty"`  // major page faults
	Inblock int64          `json:"inblock,omitempty"` // block input operations.
	Oublock int64          `json:"oublock,omitempty"` // block output operations.
	Utime   IntervalMetric `json:"utime,omitempty"`   // user CPU time used for local cmd.
	Stime   IntervalMetric `json:"stime,omitempty"`   // system CPU time used for local cmd.
	// contains filtered or unexported fields
}

StepMetric contains metrics about a build Step.

type Target 

type Target int

Target is a build target used in Graph. 0 means no target. valid range is [0, NumTargets).

type TargetError 

type TargetError struct {
	Suggests []string
	// contains filtered or unexported fields
}

TargetError is an error of unknown target for build.

func (TargetError) Error 

func (e TargetError) Error() string

func (TargetError) Unwrap 

func (e TargetError) Unwrap() error

type TraceStat 

type TraceStat struct {
	// Name is trace name.
	Name string

	// N is count of the trace.
	N int

	// NErr is error count of the trace.
	NErr int

	// Total is total duration of the trace.
	Total time.Duration

	// Max is max duration of the trace.
	Max time.Duration

	// Buckets are buckets of trace durations.
	//
	//  0: [0,10ms)
	//  1: [10ms, 100ms)
	//  2: [100ms, 1s)
	//  3: [1s, 10s)
	//  4: [10s, 1m)
	//  5: [1m, 10m)
	//  6: >=10m
	Buckets [7]int
}

TraceStat is trace statistics.

func (*TraceStat) Avg 

func (t *TraceStat) Avg() time.Duration

Avg returns average duration of the trace.

Source Files

View all Source files

Directories

Path Synopsis
Package buildconfig provides build config for `siso ninja`.
 Package buildconfig provides build config for `siso ninja`.
Package cachestore provides the interface CacheStore.
 Package cachestore provides the interface CacheStore.
Package metadata provides a data structure to hold build metadata.
 Package metadata provides a data structure to hold build metadata.
Package ninjabuild provides build steps by ninja.
 Package ninjabuild provides build steps by ninja.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.