Documentation ¶
Index ¶
- Variables
- type Analyzer
- type Checker
- type CheckerOptions
- func ExpectRunRequestDimKeys(v ...string) CheckerOptions
- func ExpectRunRequestTagKeys(v ...string) CheckerOptions
- func ExpectRunResultBotDimKeys(v ...string) CheckerOptions
- func ExpectRunResultTagKeys(v ...string) CheckerOptions
- func IgnoreDimKeys(v ...string) CheckerOptions
- func IgnoreTagKeys(v ...string) CheckerOptions
- type Diagnostics
- type Options
- type ReplicaDiagnostics
- type Results
- type SwarmingTaskDiagnostics
Constants ¶
This section is empty.
Variables ¶
var ( // DefaultCheckerOpts is a set of basic default checks to run on experiment data. DefaultCheckerOpts = []CheckerOptions{ IgnoreDimKeys("id"), IgnoreTagKeys("id"), ExpectRunRequestDimKeys("os"), ExpectRunResultBotDimKeys("cpu", "synthetic_product_name"), ExpectRunRequestTagKeys("benchmark", "storyfilter"), ExpectRunResultTagKeys("os", "benchmark", "storyfilter"), } )
Functions ¶
This section is empty.
Types ¶
type Analyzer ¶
type Analyzer struct {
// contains filtered or unexported fields
}
Analyzer encapsulates the state of an Analyzer process execution. Its lifecycle follows a request to process all of the output of an A/B benchmark experiment run. Users of Analyzer must instantiate and attach the necessary service dependencies.
func New ¶
New returns a new instance of Analyzer. Set either pinpointJobID, or controlDigests and treatmentDigests.
func (*Analyzer) AnalysisResults ¶
func (a *Analyzer) AnalysisResults() []*cpb.AnalysisResult
AnalysisResults returns a slice of AnalysisResult protos populated with data from the experiment.
func (*Analyzer) Diagnostics ¶
func (a *Analyzer) Diagnostics() *Diagnostics
func (*Analyzer) ExperimentSpec ¶
func (a *Analyzer) ExperimentSpec() *cpb.ExperimentSpec
type Checker ¶
type Checker interface { // Findings returns a list of strings describing potential issues that the checker identified. Findings() []string // CheckSwarmingTask validates a single swarming task in isolation. CheckSwarmingTask(taskInfo *apipb.TaskRequestMetadataResponse) // CheckRunTask validates a single swarming run task request/result pair in isolation. CheckRunTask(taskInfo *apipb.TaskRequestMetadataResponse) // CheckArmComparability validates assumptions about how treatment and control arm tasks may // differ from each other, and how tasks within an arm may differ from each other. CheckArmComparability(controls, treatments *processedArmTasks) // CheckControlTreatmentSpecMatch validates assumptions that the run spec of treatment and control // should match. CheckControlTreatmentSpecMatch(controlSpec, treatmentSpec *specpb.ExperimentSpec) error }
Checker performs diagnostic checks on experiment artifacts prior to analysis. Its main use case is akin to a compiler's static type checker - it identifies conditions that violate assumptions about the input data before proceeding with the rest of the input processing steps.
func NewChecker ¶
func NewChecker(opts ...CheckerOptions) Checker
NewChecker returns an instance of Checker with the specified configuration options. If you are unsure which CheckerOptions make sense, try starting with DefaultCheckerOpts.
type CheckerOptions ¶
type CheckerOptions func(*checker)
CheckerOptions configure the behavior of Checker.
func ExpectRunRequestDimKeys ¶
func ExpectRunRequestDimKeys(v ...string) CheckerOptions
ExpectRunRequestDimKeys tells Checker to verify the existence of the specified keys.
func ExpectRunRequestTagKeys ¶
func ExpectRunRequestTagKeys(v ...string) CheckerOptions
ExpectRunRequestTagKeys tells Checker to verify the existence of the specified keys.
func ExpectRunResultBotDimKeys ¶
func ExpectRunResultBotDimKeys(v ...string) CheckerOptions
ExpectRunResultBotDimKeys tells Checker to verify the existence of the specified keys.
func ExpectRunResultTagKeys ¶
func ExpectRunResultTagKeys(v ...string) CheckerOptions
ExpectRunResultTagKeys tells Checker to verify the existence of the specified keys.
func IgnoreDimKeys ¶
func IgnoreDimKeys(v ...string) CheckerOptions
IgnoreDimKeys tells Checker to ignore the specified keys when running checks.
func IgnoreTagKeys ¶
func IgnoreTagKeys(v ...string) CheckerOptions
IgnoreTagKeys tells Checker to ignore the specified keys when running checks.
type Diagnostics ¶
type Diagnostics struct { // Bad news: things that had to be excluded from the analysis, and why. ExcludedSwarmingTasks map[string]*SwarmingTaskDiagnostics `json:",omitempty"` ExcludedReplicas map[int]*ReplicaDiagnostics `json:",omitempty"` // Good news: things that were included in the analysis. IncludedSwarmingTasks map[string]*SwarmingTaskDiagnostics `json:",omitempty"` IncludedReplicas map[int]*ReplicaDiagnostics `json:",omitempty"` // contains filtered or unexported fields }
Diagnostics contains diagnostic messages about the replica task pairs and individual tasks generated by the Analyzer.
func (*Diagnostics) AnalysisDiagnostics ¶
func (d *Diagnostics) AnalysisDiagnostics() *cpb.AnalysisDiagnostics
type Options ¶
type Options func(*Analyzer)
Options configure one or more fields of an Analyzer instance.
func WithCASResultReader ¶
func WithCASResultReader(r backends.CASResultReader) Options
WithCASResultReader configures an Analyzer instance to use the given CASResultReader.
func WithExperimentSpec ¶
func WithExperimentSpec(s *cpb.ExperimentSpec) Options
WithExperimentSpec configures an Analyzer instance to use the given ExperimentSpec.
func WithSwarmingTaskReader ¶
func WithSwarmingTaskReader(r backends.SwarmingTaskReader) Options
WithTaskResultsReader configures an Analyzer instance to use the given TaskResultsReader.
type ReplicaDiagnostics ¶
type ReplicaDiagnostics struct { Number int ControlTaskID string TreatmentTaskID string Message []string `json:",omitempty"` }
ReplicaDiagnostics contains replica, or task pair-specific diagnostic messages generated by the Analyzer.
type Results ¶
type Results struct { // Benchmark is the name of a perf benchmark suite, such as Speedometer2 or JetStream Benchmark string // Workload is the name of a benchmark-specific workload, such as TodoMVC-ReactJS WorkLoad string // BuildConfig is the name of a build configuration, e.g. "Mac arm Builder Perf PGO" BuildConfig string // RunConfig is the name of a run configuration, e.g. "Macmini9,1_arm64-64-Apple_M1_16384_1_4744421.0" RunConfig string // Statistics summarizes the difference between the treatment and control arms for the given // Benchmark and Workload on the hardware described by RunConfig, using the binary built using // the given BuildConfig. Statistics *cabe_stats.BerfWilcoxonSignedRankedTestResult }
Results encapsulates a response from the Go statistical package after it has processed swarming task data and verified the experimental setup is valid for analysis.
type SwarmingTaskDiagnostics ¶
SwarmingTaskDiagnostics contains task-specific diagnostic messages generated by the Analyzer.