migrator

Documentation

Overview

Package migrator provides interfaces and tooling for migrating LUCI configuration files across all known LUCI projects.

The tool revolves around the user providing a plugin to perform bulk analysis and optional fixes across all LUCI projects' configuration folders, from the point of view of the 'luci-config' service.

The plugin currently has two main points of extension:

• FindProblems - Analyze the LUCI project and config file contents to see if the project has migrated. This can inspect the project name, as well as the presence and contents of any config files.
• ApplyFix - If FindProblems revealed issues, ApplyFix will be run in the context of a local sparse+shallow checkout containing the configuration files. This has the ability to run programs in the checkout, as well as stat/read/modify files.

This package contains the interface definitions for the migrator plugin.

Index

• Constants
• func NonActionable(r *Report)
• type API
• type ConfigFile
• type InstantiateAPI
• type LocalProject
• type Project
• type Repo
• type Report
  • func NewReportFromCSVRow(row []string) (ret *Report, err error)
  • func (r *Report) Clone() *Report
  • func (r *Report) ToCSVRow() []string
• type ReportDump
  • func NewReportDumpFromCSV(raw io.Reader) (*ReportDump, error)
  • func (r *ReportDump) Add(reports ...*Report)
  • func (r *ReportDump) Clone() *ReportDump
  • func (r *ReportDump) Empty() bool
  • func (r *ReportDump) Iterate(cb func(ReportID, []*Report) bool)
  • func (r *ReportDump) PrettyPrint(out io.Writer, header []string, row func(r *Report) []string)
  • func (r *ReportDump) UpdateFrom(other *ReportDump) int
  • func (r *ReportDump) WriteToCSV(out io.Writer) error
• type ReportID
  • func (r ReportID) ConfigSet() config.Set
  • func (r ReportID) String() string
• type ReportOption
  • func MetadataOption(key string, values ...string) ReportOption
• type Reportable
• type Shell

Constants

const TieStderr = "2>&1"

TieStderr is a special token for Shell.{Run,Retval,Stdout} to indicate that Stderr content should be redirected to Stdout.

Functions

func NonActionable

func NonActionable(r *Report)

NonActionable is-a ReportOption which indicates that this Report cannot be fixed by ApplyFix. If there are no Actionable Reports for a given project in FindProblems, the checkout and ApplyFix phase will be skipped.

Types

type API

type API interface {
	// FindProblems allows you to report problems about a Project, or about
	// certain configuration files within the project.
	//
	// If the method finds issues which warrant followup, it should use
	// proj.Report and/or proj.ConfigFiles()["filename"].Report. Reporting one or
	// more problems will cause the migrator tool to set up a checkout for this
	// project.
	//
	// Logging is set up for this context, and will be diverted to a per-project
	// logfile.
	//
	// `proj` is implemented with LUCI Config API calls; it reflects the state of
	// the project as currently known by the luci-config service.
	//
	// This function should panic on error.
	FindProblems(ctx context.Context, proj Project)

	// ApplyFix allows you to attempt to automatically fix problems within a repo.
	//
	// Note that for real implementations you may want to keep details on the
	// `impl` struct; this will let you carry over information from
	// FindProblems.
	//
	// Logging is set up for this context, and will be diverted to a per-project
	// logfile.
	//
	// `proj` is implemented with local filesystem interactions on the checked-out
	// repo. This may differ from the current state of the luci-config service.
	//
	// This function should panic on error.
	ApplyFix(ctx context.Context, proj LocalProject)
}

API is the implementation of a plugin, as returned by the plugin's InstantiateAPI function.

One API instance will be created per LUCI project during 'scan', and `FindProblems` will be invoked before `ApplyFix`.

type ConfigFile

type ConfigFile interface {
	Reportable

	// Path returns the path relative to the repo's generated configuration
	// directory.
	Path() string

	// RawData returns the data of this configuration file as a string.
	//
	// May do a (cached) network operation to retrieve the raw data.
	RawData() string

	// TextPb parses the raw data as a text proto with "heredoc" processing into
	// `out`.
	//
	// May do a (cached) network operation to retrieve the raw data.
	//
	// If there are errors during parsing, this panics.
	TextPb(out proto.Message)
}

ConfigFile encapsulates as single configuration file from a Project.

type InstantiateAPI

type InstantiateAPI func() API

InstantiateAPI is the factory for API instances (one per LUCI project).

It should return a new instance of API.

type LocalProject

type LocalProject interface {
	Project

	// ConfigRoot returns the path to the config file 'root' within the repo.
	//
	// This would be the directory with the root of the lucicfg Starlark code
	// tree, if the repo has one, otherwise it is the root of the repository.
	//
	// This an 'absolute-style' path (see Shell).
	ConfigRoot() string

	// GeneratedConfigRoot returns the path to the generated config files (i.e.
	// the ones seen by the luci-config service).
	//
	// This an 'absolute-style' path (see Shell).
	GeneratedConfigRoot() string

	// Repo is the repository that contains this project.
	Repo() Repo

	// Shell returns a new shell object that can be used to modify configs.
	//
	// Its cwd is set to the ConfigRoot.
	Shell() Shell

	// RegenerateConfigs runs lucicfg to regenerate project configs.
	RegenerateConfigs()
}

LocalProject is a checked out project that can be modified.

type Project

type Project interface {
	Reportable

	// ID is the project identifier, as it appears in projects.cfg.
	ID() string

	// ConfigFiles returns a mapping of path to config file.
	//
	// May do a (cached) network operation to retrieve the data.
	ConfigFiles() map[string]ConfigFile
}

Project encapsulates the pertinent details of a single LUCI Project's configuration files.

type Repo

type Repo interface {
}

Repo represents a checked-out git repo on disk.

It contains configs of one or more LUCI projects.

type Report

type Report struct {
	ReportID

	Tag     string
	Problem string

	// If true, indicates that this report can be fixed by ApplyFix.
	Actionable bool

	Metadata map[string]stringset.Set
}

Report stores a single tagged problem (and metadata).

func NewReportFromCSVRow

func NewReportFromCSVRow(row []string) (ret *Report, err error)

NewReportFromCSVRow creates a new Report from a CSVRow written with ToCSVRow.

func (*Report) Clone

func (r *Report) Clone() *Report

Clone returns a deep copy of this Report.

func (*Report) ToCSVRow

func (r *Report) ToCSVRow() []string

ToCSVRow returns a CSV row:

Checkout, Project, ConfigFile, Tag, Problem, Actionable, Metadata*

Where Metadata* is one key:value entry per value in Metadata.

type ReportDump

type ReportDump struct {
	// contains filtered or unexported fields
}

ReportDump is a mapping of all reports, generated via DumpReports(ctx).

It maps the ReportID to a list of all Reports found for that ReportID.

func NewReportDumpFromCSV

func NewReportDumpFromCSV(raw io.Reader) (*ReportDump, error)

NewReportDumpFromCSV reads a raw CSV and returns the *ReportDump.

CSV must be a compatible schema version (i.e. produced with a similar version of ReportDump.WriteToCSV.

func (*ReportDump) Add

func (r *ReportDump) Add(reports ...*Report)

Add appends one or more reports to this ReportDump.

func (*ReportDump) Clone

func (r *ReportDump) Clone() *ReportDump

Clone makes a deep copy of this ReportDump.

func (*ReportDump) Empty

func (r *ReportDump) Empty() bool

Empty returns true iff this ReportDump has no entries.

func (*ReportDump) Iterate

func (r *ReportDump) Iterate(cb func(ReportID, []*Report) bool)

Iterate invokes `cb` for each ReportID with all Reports from that ReportID.

`cb` will be called in sorted order on ReportID. If it returns `false`, iteration will stop.

func (*ReportDump) PrettyPrint

func (r *ReportDump) PrettyPrint(out io.Writer, header []string, row func(r *Report) []string)

PrettyPrint formats the report for terminal output.

For each report calls `row` callback to get the list of columns for it. If the callback returns an empty list, this report is skipped.

func (*ReportDump) UpdateFrom

func (r *ReportDump) UpdateFrom(other *ReportDump) int

UpdateFrom appends `other` to this ReportDump.

Returns the number of Report records in `other`.

func (*ReportDump) WriteToCSV

func (r *ReportDump) WriteToCSV(out io.Writer) error

WriteToCSV writes this ReportDump out as CSV.

type ReportID

type ReportID struct {
	Checkout   string
	Project    string
	ConfigFile string
}

ReportID is a (Checkout, Project, ConfigFile) tuple and identifies the object which generated the report.

func (ReportID) ConfigSet

func (r ReportID) ConfigSet() config.Set

ConfigSet returns the luci-config "config.Set" for this report.

e.g. "projects/${Project}"

func (ReportID) String

func (r ReportID) String() string

type ReportOption

type ReportOption func(*Report)

ReportOption allows attaching additional optional data to reports.

func MetadataOption

func MetadataOption(key string, values ...string) ReportOption

MetadataOption returns a ReportOption which allows attaching a string-string multimap of metadatadata to a Report.

type Reportable

type Reportable interface {
	// Report logs a problem about this object.
	//
	// `tag` should be a CAPS_STRING which makes sense to your particular
	// application.
	//
	// `description` should be a human readable explanation of the problem.
	//
	// Example:
	//    tag: MISSING_BLARF
	//    description: "blarf.cfg file is missing"
	Report(tag, description string, opts ...ReportOption)
}

Reportable is an interface for reporting problems about an object.

Implemented by Project and ConfigFile.

type Shell

type Shell interface {
	// Cd changes the current directory.
	//
	// Absolute paths (i.e. beginning with '/') are interpreted as relative to the
	// repo root.
	//
	// Attempting to Cd out of the top of the repo is an error.
	Cd(path string)

	// ModifyFile allows trivial modification of a file.
	//
	// This will call `modify` with the contents of the old file ("" if the file
	// didn't exist), and will write the returned string back to `path`, if the
	// returned string is different. This will create missing intermediate
	// directories.
	//
	// Allows supplying FileMode. If omitted, defaults to 0666 (i.e. a+rw) for new
	// files, and otherwise will keep the mode of the existing file.
	ModifyFile(path string, modify func(oldContents string) string, mode ...os.FileMode)

	// Stat returns the FileInfo for the entity at `path`, or nil if no such
	// entity exists.
	Stat(path string) os.FileInfo

	// Run executes a command `name` with arguments `args`.
	//
	// `name` may be relative to the Shell's cwd (e.g. `./main.star`), or to $PATH
	// (e.g. `git`).
	//
	// The command is expected to return exitcode 0.
	//
	// The command runs with the cwd of the Shell.
	//
	// Stdout+Stderr are redirected to logging (with Stdout logged at 'Info' level
	// and Stderr logged at 'Error' level).
	//
	// If the very last item in `args` is the TieStderr constant, both will be
	// tied together and all output will be logged at 'Info' level.
	Run(name string, args ...string)

	// Same as `Run`, but returns the exitcode of the process (instead of
	// asserting an exit code of 0).
	Retval(name string, args ...string) int

	// Same as `Run`, but returns the stdout of the process. If TieStderr is the
	// last argument, then this captures Stderr, too.
	Stdout(name string, args ...string) string
}

Shell is a basic interface for interacting with the Repo.

Paths

All `path` arguments to the Shell are either:

• start with '/' and are relative to the corresponding Repo's root.
• OR; are relative to the Shell's current working directory.

It is not permitted to access a path outside the Repo's root.

Errors

All functions of Shell will panic under error conditions. This is consistent with the API of the ApplyFix plugin method.