analyzer

Documentation

Index

• Variables
• type Analyzer
  • func New(pinpointJobID string, opts ...Options) *Analyzer
  • func (a *Analyzer) AnalysisResults() []*cpb.AnalysisResult
  • func (a *Analyzer) Diagnostics() *Diagnostics
  • func (a *Analyzer) ExperimentSpec() *cpb.ExperimentSpec
  • func (a *Analyzer) Run(ctx context.Context) ([]Results, error)
  • func (a *Analyzer) RunChecker(ctx context.Context, c Checker) error
• type Checker
  • func NewChecker(opts ...CheckerOptions) Checker
• type CheckerOptions
  • func ExpectRunRequestDimKeys(v ...string) CheckerOptions
  • func ExpectRunRequestTagKeys(v ...string) CheckerOptions
  • func ExpectRunResultBotDimKeys(v ...string) CheckerOptions
  • func ExpectRunResultTagKeys(v ...string) CheckerOptions
  • func IgnoreDimKeys(v ...string) CheckerOptions
  • func IgnoreTagKeys(v ...string) CheckerOptions
• type Diagnostics
  • func (d *Diagnostics) AnalysisDiagnostics() *cpb.AnalysisDiagnostics
• type Options
  • func WithCASResultReader(r backends.CASResultReader) Options
  • func WithExperimentSpec(s *cpb.ExperimentSpec) Options
  • func WithSwarmingTaskReader(r backends.SwarmingTaskReader) Options
• type ReplicaDiagnostics
• type Results
• type SwarmingTaskDiagnostics

Variables

var (
	// DefaultCheckerOpts is a set of basic default checks to run on experiment data.
	DefaultCheckerOpts = []CheckerOptions{
		IgnoreDimKeys("id"),
		IgnoreTagKeys("id"),
		ExpectRunRequestDimKeys("os"),
		ExpectRunResultBotDimKeys("cpu", "synthetic_product_name"),
		ExpectRunRequestTagKeys("benchmark", "storyfilter"),
		ExpectRunResultTagKeys("os", "benchmark", "storyfilter"),
	}
)

Types

type Analyzer

type Analyzer struct {
	// contains filtered or unexported fields
}

Analyzer encapsulates the state of an Analyzer process execution. Its lifecycle follows a request to process all of the output of an A/B benchmark experiment run. Users of Analyzer must instantiate and attach the necessary service dependencies.

func New

func New(pinpointJobID string, opts ...Options) *Analyzer

New returns a new instance of Analyzer. Set either pinpointJobID, or controlDigests and treatmentDigests.

func (*Analyzer) AnalysisResults

func (a *Analyzer) AnalysisResults() []*cpb.AnalysisResult

AnalysisResults returns a slice of AnalysisResult protos populated with data from the experiment.

func (*Analyzer) Diagnostics

func (a *Analyzer) Diagnostics() *Diagnostics

func (*Analyzer) ExperimentSpec

func (a *Analyzer) ExperimentSpec() *cpb.ExperimentSpec

func (*Analyzer) Run

func (a *Analyzer) Run(ctx context.Context) ([]Results, error)

Run executes the whole Analyzer process for a single, complete experiment. TODO(seanmccullough): break this up into distinct, testable stages with one function per stage.

func (*Analyzer) RunChecker

func (a *Analyzer) RunChecker(ctx context.Context, c Checker) error

RunChecker verifies some assumptions we need to make about the experiment data input for our analyses.

type Checker

type Checker interface {
	// Findings returns a list of strings describing potential issues that the checker identified.
	Findings() []string
	// CheckSwarmingTask validates a single swarming task in isolation.
	CheckSwarmingTask(taskInfo *apipb.TaskRequestMetadataResponse)
	// CheckRunTask validates a single swarming run task request/result pair in isolation.
	CheckRunTask(taskInfo *apipb.TaskRequestMetadataResponse)
	// CheckArmComparability validates assumptions about how treatment and control arm tasks may
	// differ from each other, and how tasks within an arm may differ from each other.
	CheckArmComparability(controls, treatments *processedArmTasks)
	// CheckControlTreatmentSpecMatch validates assumptions that the run spec of treatment and control
	// should match.
	CheckControlTreatmentSpecMatch(controlSpec, treatmentSpec *specpb.ExperimentSpec) error
}

Checker performs diagnostic checks on experiment artifacts prior to analysis. Its main use case is akin to a compiler's static type checker - it identifies conditions that violate assumptions about the input data before proceeding with the rest of the input processing steps.

func NewChecker

func NewChecker(opts ...CheckerOptions) Checker

NewChecker returns an instance of Checker with the specified configuration options. If you are unsure which CheckerOptions make sense, try starting with DefaultCheckerOpts.

type CheckerOptions

type CheckerOptions func(*checker)

CheckerOptions configure the behavior of Checker.

func ExpectRunRequestDimKeys

func ExpectRunRequestDimKeys(v ...string) CheckerOptions

ExpectRunRequestDimKeys tells Checker to verify the existence of the specified keys.

func ExpectRunRequestTagKeys

func ExpectRunRequestTagKeys(v ...string) CheckerOptions

ExpectRunRequestTagKeys tells Checker to verify the existence of the specified keys.

func ExpectRunResultBotDimKeys

func ExpectRunResultBotDimKeys(v ...string) CheckerOptions

ExpectRunResultBotDimKeys tells Checker to verify the existence of the specified keys.

func ExpectRunResultTagKeys

func ExpectRunResultTagKeys(v ...string) CheckerOptions

ExpectRunResultTagKeys tells Checker to verify the existence of the specified keys.

func IgnoreDimKeys

func IgnoreDimKeys(v ...string) CheckerOptions

IgnoreDimKeys tells Checker to ignore the specified keys when running checks.

func IgnoreTagKeys

func IgnoreTagKeys(v ...string) CheckerOptions

IgnoreTagKeys tells Checker to ignore the specified keys when running checks.

type Diagnostics

type Diagnostics struct {

	// Bad news: things that had to be excluded from the analysis, and why.
	ExcludedSwarmingTasks map[string]*SwarmingTaskDiagnostics `json:",omitempty"`
	ExcludedReplicas      map[int]*ReplicaDiagnostics         `json:",omitempty"`

	// Good news: things that were included in the analysis.
	IncludedSwarmingTasks map[string]*SwarmingTaskDiagnostics `json:",omitempty"`
	IncludedReplicas      map[int]*ReplicaDiagnostics         `json:",omitempty"`
	// contains filtered or unexported fields
}

Diagnostics contains diagnostic messages about the replica task pairs and individual tasks generated by the Analyzer.

func (*Diagnostics) AnalysisDiagnostics

func (d *Diagnostics) AnalysisDiagnostics() *cpb.AnalysisDiagnostics

type Options

type Options func(*Analyzer)

Options configure one or more fields of an Analyzer instance.

func WithCASResultReader

func WithCASResultReader(r backends.CASResultReader) Options

WithCASResultReader configures an Analyzer instance to use the given CASResultReader.

func WithExperimentSpec

func WithExperimentSpec(s *cpb.ExperimentSpec) Options

WithExperimentSpec configures an Analyzer instance to use the given ExperimentSpec.

func WithSwarmingTaskReader

func WithSwarmingTaskReader(r backends.SwarmingTaskReader) Options

WithTaskResultsReader configures an Analyzer instance to use the given TaskResultsReader.

type ReplicaDiagnostics

type ReplicaDiagnostics struct {
	Number          int
	ControlTaskID   string
	TreatmentTaskID string
	Message         []string `json:",omitempty"`
}

ReplicaDiagnostics contains replica, or task pair-specific diagnostic messages generated by the Analyzer.

type Results

type Results struct {
	// Benchmark is the name of a perf benchmark suite, such as Speedometer2 or JetStream
	Benchmark string
	// Workload is the name of a benchmark-specific workload, such as TodoMVC-ReactJS
	WorkLoad string
	// BuildConfig is the name of a build configuration, e.g. "Mac arm Builder Perf PGO"
	BuildConfig string
	// RunConfig is the name of a run configuration, e.g. "Macmini9,1_arm64-64-Apple_M1_16384_1_4744421.0"
	RunConfig string
	// Statistics summarizes the difference between the treatment and control arms for the given
	// Benchmark and Workload on the hardware described by RunConfig, using the binary built using
	// the given BuildConfig.
	Statistics *cabe_stats.BerfWilcoxonSignedRankedTestResult
}

Results encapsulates a response from the Go statistical package after it has processed swarming task data and verified the experimental setup is valid for analysis.

type SwarmingTaskDiagnostics

type SwarmingTaskDiagnostics struct {
	TaskID  string
	Message []string `json:",omitempty"`
}

SwarmingTaskDiagnostics contains task-specific diagnostic messages generated by the Analyzer.