lucicfg

package
v0.0.0-...-35d8de9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 1, 2019 License: Apache-2.0 Imports: 57 Imported by: 0

Documentation

Overview

Package lucicfg contains LUCI config generator.

All Starlark code is executed sequentially in a single goroutine from inside Generate function, thus this package doesn't used any mutexes or other synchronization primitives. It is safe to call Generate concurrently though, since there's no global shared state, each Generate call operates on its own state.

Index

Constants

View Source 
const (
	TOK_LB    = iota // non-escaped '{'
	TOK_RB           // non-escaped '}'
	TOK_NUM          // a sequence of digits
	TOK_RUNES        // an arbitrary sequence of non-special runes
	TOK_COMMA        // ','
	TOK_DOTS         // '..'
)
View Source 
const (
	// Version is the version of lucicfg tool.
	//
	// It ends up in CLI output and in User-Agent headers.
	Version = "1.9.0"

	// UserAgent is used for User-Agent header in HTTP requests from lucicfg.
	UserAgent = "lucicfg v" + Version
)

Variables

This section is empty.

Functions

func FindTrackedFiles 

func FindTrackedFiles(dir string, patterns []string) ([]string, error)

FindTrackedFiles recursively discovers all regular files in the given directory whose names match given patterns.

See TrackedSet for the format of `patterns`. If the directory doesn't exist, returns empty slice.

Returned file names are sorted, slash-separated and relative to `dir`.

func TrackedSet 

func TrackedSet(patterns []string) func(string) (bool, error)

TrackedSet returns a predicate that classifies whether a slash-separated path belongs to a tracked set or not.

Each entry in `patterns` is either `<glob pattern>` (a "positive" glob) or `!<glob pattern>` (a "negative" glob). A path is considered tracked if its base name matches any of the positive globs and none of the negative globs. If `patterns` is empty, no paths are considered tracked. If all patterns are negative, single `**/*` positive pattern is implied as well.

The predicate returns an error if some pattern is malformed.

Types

type BacktracableError 

type BacktracableError interface {
	error

	// Backtrace returns a user-friendly error message describing the stack
	// of calls that led to this error, along with the error message itself.
	Backtrace() string
}

BacktracableError is an error that has a starlark backtrace attached to it.

Implemented by Error here, by starlark.EvalError and graph errors.

type BlobDatum 

type BlobDatum []byte

BlobDatum is a Datum which is just a raw byte blob.

func (BlobDatum) Bytes 

func (b BlobDatum) Bytes() ([]byte, error)

Bytes is a raw file body to put on disk.

func (BlobDatum) SemanticallySame 

func (b BlobDatum) SemanticallySame(other []byte) bool

SemanticallySame is true if 'other == b'.

type ConfigSet 

type ConfigSet struct {
	// Name is a name of this config set, e.g. "projects/something".
	//
	// It is used by LUCI Config to figure out how to validate files in the set.
	Name string

	// Data is files belonging to the config set.
	//
	//  Keys are slash-separated filenames, values are corresponding file bodies.
	Data map[string][]byte
}

ConfigSet is an in-memory representation of a single config set.

func ReadConfigSet 

func ReadConfigSet(dir, name string) (ConfigSet, error)

ReadConfigSet reads all regular files in the given directory (recursively) and returns them as a ConfigSet with given name.

func (ConfigSet) AsOutput 

func (cs ConfigSet) AsOutput(root string) Output

AsOutput converts this config set into Output that have it at the given root path (usually ".").

func (ConfigSet) Files 

func (cs ConfigSet) Files() []string

Files returns a sorted list of file names in the config set.

func (ConfigSet) Validate 

func (cs ConfigSet) Validate(ctx context.Context, val ConfigSetValidator) *ValidationResult

Validate sends the config set for validation to LUCI Config service.

Returns ValidationResult with a list of validation message (errors, warnings, etc). The list of messages may be empty if the config set is 100% valid.

If the RPC call itself failed, ValidationResult is still returned, but it has only ConfigSet and RPCError fields populated.

type ConfigSetValidator 

type ConfigSetValidator interface {
	// Validate sends the validation request to the service.
	//
	// Returns errors only on RPC errors. Actual validation errors are
	// communicated through []*ValidationMessage.
	Validate(ctx context.Context, req *ValidationRequest) ([]*ValidationMessage, error)
}

ConfigSetValidator is primarily implemented through config.Service, but can also be mocked in tests.

func RemoteValidator 

func RemoteValidator(svc *config.Service) ConfigSetValidator

RemoteValidator returns ConfigSetValidator that makes RPCs to LUCI Config.

type Datum 

type Datum interface {
	// Bytes is a raw file body to put on disk.
	Bytes() ([]byte, error)
	// SemanticallySame is true if 'other' represents the same datum.
	SemanticallySame(other []byte) bool
}

Datum represents one generated output file.

type Error 

type Error struct {
	Msg   string
	Stack *builtins.CapturedStacktrace
}

Error is a single error message emitted by the config generator.

It holds a stack trace responsible for the error.

func (*Error) Backtrace 

func (e *Error) Backtrace() string

Backtrace is part of BacktracableError interface.

func (*Error) Error 

func (e *Error) Error() string

Error is part of 'error' interface.

type Inputs 

type Inputs struct {
	Code  interpreter.Loader // a package with the user supplied code
	Entry string             // a name of the entry point script in this package
	// contains filtered or unexported fields
}

Inputs define all inputs for the config generator.

type MessageDatum 

type MessageDatum struct {
	Header  string
	Message *starlarkproto.Message
	// contains filtered or unexported fields
}

MessageDatum is a Datum constructed from a proto message.

func (*MessageDatum) Bytes 

func (m *MessageDatum) Bytes() ([]byte, error)

Bytes is a raw file body to put on disk.

func (*MessageDatum) SemanticallySame 

func (m *MessageDatum) SemanticallySame(other []byte) bool

SemanticallySame is true if 'other' deserializes and equals b.Message.

On deserialization or comparison errors returns false.

type Meta 

type Meta struct {
	ConfigServiceHost string   `json:"config_service_host"` // LUCI config host name
	ConfigDir         string   `json:"config_dir"`          // output directory to place generated files or '-' for stdout
	TrackedFiles      []string `json:"tracked_files"`       // e.g. ["*.cfg", "!*-dev.cfg"]
	FailOnWarnings    bool     `json:"fail_on_warnings"`    // true to treat validation warnings as errors
	// contains filtered or unexported fields
}

Meta contains configuration for the configuration generator itself.

It influences how generator produces output configs. It is settable through lucicfg.config(...) statements on the Starlark side or through command line flags. Command line flags override what was set via lucicfg.config(...).

See @stdlib//internal/lucicfg.star for full meaning of fields.

func (*Meta) AddFlags 

func (m *Meta) AddFlags(fs *flag.FlagSet)

AddFlags registers command line flags that correspond to Meta fields.

func (*Meta) Log 

func (m *Meta) Log(ctx context.Context)

Log logs the values of the meta parameters to Debug logger.

func (*Meta) PopulateFromTouchedIn 

func (m *Meta) PopulateFromTouchedIn(t *Meta)

PopulateFromTouchedIn takes all touched values in `t` and copies them to `m`, overriding what's in `m`.

func (*Meta) RebaseConfigDir 

func (m *Meta) RebaseConfigDir(root string)

RebaseConfigDir changes ConfigDir, if it is set, to be absolute by appending it to the given root.

Doesn't touch "-", which indicates "stdout".

func (*Meta) WasTouched 

func (m *Meta) WasTouched(field string) bool

WasTouched returns true if the field (given by its Starlark snake_case name) was explicitly set via CLI flags or via lucicfg.config(...) in Starlark.

Panics if the field is unrecognized.

type Output 

type Output struct {
	// Data is all output files.
	//
	// Keys are slash-separated filenames, values are corresponding file bodies.
	Data map[string]Datum

	// Roots is mapping "config set name => its root".
	//
	// Roots are given as slash-separated paths relative to the output root, e.g.
	// '.' matches ALL output files.
	Roots map[string]string
}

Output is an in-memory representation of all generated output files.

Output may span zero or more config sets, each defined by its root directory. Config sets may intersect (though this is rare).

func (Output) Compare 

func (o Output) Compare(dir string, semantic bool) (changed, unchanged []string, err error)

Compare compares files on disk to what's in the output.

Returns names of files that are different ('changed') and same ('unchanged').

If 'semantic' is true, for output files based on proto messages uses semantic comparison, i.e. loads the file on disk as a proto message and compares it to the output message.

If 'semantic' is false, compares files as byte blobs.

Files on disk that are not in the output set are totally ignored. Missing files that are listed in the output set are considered changed.

Returns an error if some file on disk can't be read or some output file can't be serialized.

func (Output) ConfigSets 

func (o Output) ConfigSets() ([]ConfigSet, error)

ConfigSets partitions this output into 0 or more config sets based on Roots.

Returns an error if some output Datum can't be serialized.

func (Output) DebugDump 

func (o Output) DebugDump()

DebugDump writes the output to stdout in a format useful for debugging.

func (Output) DiscardChangesToUntracked 

func (o Output) DiscardChangesToUntracked(ctx context.Context, tracked []string, dir string) error

DiscardChangesToUntracked replaces bodies of the files that are in the output set, but not in the `tracked` set (per TrackedSet semantics) with what's on disk in the given `dir`.

This allows to construct partially generated output: some configs (the ones in the tracked set) are generated, others are loaded from disk.

If `dir` is "-" (which indicates that the output is going to be dumped to stdout rather then to disk), just removes untracked files from the output.

func (Output) Files 

func (o Output) Files() []string

Files returns a sorted list of file names in the output.

func (Output) Read 

func (o Output) Read(dir string) error

Read replaces values in o.Data by reading them from disk as blobs.

Returns an error if some file can't be read.

func (Output) Write 

func (o Output) Write(dir string) (changed, unchanged []string, err error)

Write updates files on disk to match the output.

Returns a list of updated files and a list of files that are already up-to-date.

When comparing files does byte-to-byte comparison, not a semantic one. This ensures all written files are formatted in a consistent style. Use Compare to explicitly check the semantic difference.

Creates missing directories. Not atomic. All files have mode 0666.

type State 

type State struct {
	Inputs Inputs // all inputs, exactly as passed to Generate.
	Output Output // all generated config files, populated at the end
	Meta   Meta   // lucicfg parameters, settable through Starlark
	// contains filtered or unexported fields
}

State is mutated throughout execution of the script and at the end contains the final execution result.

It is available in the implementation of native functions exposed to the Starlark side. Starlark code operates with the state exclusively through these functions.

All Starlark code is executed sequentially in a single goroutine, thus the state is not protected by any mutexes.

func Generate 

func Generate(ctx context.Context, in Inputs) (*State, error)

Generate interprets the high-level config.

Returns a multi-error with all captured errors. Some of them may implement BacktracableError interface.

type ValidationMessage 

type ValidationMessage = config.ComponentsConfigEndpointValidationMessage

Alias some ridiculously long type names that we round-trip in the public API.

type ValidationRequest 

type ValidationRequest = config.LuciConfigValidateConfigRequestMessage

Alias some ridiculously long type names that we round-trip in the public API.

type ValidationResult 

type ValidationResult struct {
	ConfigSet string               `json:"config_set"`          // a config set being validated
	Failed    bool                 `json:"failed"`              // true if the config is bad
	Messages  []*ValidationMessage `json:"messages"`            // errors, warnings, infos, etc.
	RPCError  string               `json:"rpc_error,omitempty"` // set if the RPC itself failed
}

ValidationResult is what we get after validating a config set.

func (*ValidationResult) Log 

func (vr *ValidationResult) Log(ctx context.Context)

Log all messages in the result to the logger at an appropriate logging level.

func (*ValidationResult) OverallError 

func (vr *ValidationResult) OverallError(failOnWarnings bool) error

OverallError is nil if the validation succeeded or non-nil if failed.

Beware: mutates Failed field accordingly.

Source Files

View all Source files

Directories

Path Synopsis
cli
Package cli contains command line interface for lucicfg tool.
 Package cli contains command line interface for lucicfg tool.
base
Package base contains code shared by other CLI subpackages.
 Package base contains code shared by other CLI subpackages.
cmds/diff
Package diff implements 'semantic-diff' subcommand.
 Package diff implements 'semantic-diff' subcommand.
cmds/generate
Package generate implements 'generate' subcommand.
 Package generate implements 'generate' subcommand.
cmds/validate
Package validate implements 'validate' subcommand.
 Package validate implements 'validate' subcommand.
cmd
docgen
Command docgen is the documentation generator.
 Command docgen is the documentation generator.
lucicfg
Command lucicfg is CLI for LUCI config generator.
 Command lucicfg is CLI for LUCI config generator.
Package doc generates starlark documentation.
 Package doc generates starlark documentation.
Package docgen generates documentation from Starlark code.
 Package docgen generates documentation from Starlark code.
ast
Package ast defines AST relevant for the documentation generation.
 Package ast defines AST relevant for the documentation generation.
docstring
Package docstring parses docstrings into more structured representation.
 Package docstring parses docstrings into more structured representation.
symbols
Package symbols defines a data model representing Starlark symbols.
 Package symbols defines a data model representing Starlark symbols.
Package graph implements a DAG used internally to represent config objects.
 Package graph implements a DAG used internally to represent config objects.
Package normalize contains functions to "flatten" and normalize LUCI configs.
 Package normalize contains functions to "flatten" and normalize LUCI configs.
Package starlark is generated by go.chromium.org/luci/tools/cmd/assets.
 Package starlark is generated by go.chromium.org/luci/tools/cmd/assets.
Package vars implement lucicfg.var() support.
 Package vars implement lucicfg.var() support.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.