search

package
v0.0.0-...-03d6fc4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 23, 2019 License: BSD-3-Clause Imports: 26 Imported by: 0

Documentation

Overview

search contains the core functionality for searching for digests across a tile.

search contains the core functionality for searching for digests across a tile.

Index

Constants

View Source 
const (
	// SORT_FIELD_COUNT indicates that the image counts should be used for sorting.
	SORT_FIELD_COUNT = "count"

	// SORT_FIELD_DIFF indicates that the diff field should be used for sorting.
	SORT_FIELD_DIFF = "diff"
)
View Source 
const (
	// REF_CLOSEST_POSTIVE identifies the diff to the closest positive digest.
	REF_CLOSEST_POSTIVE = "pos"

	// REF_CLOSEST_NEGATIVE identifies the diff to the closest negative digest.
	REF_CLOSEST_NEGATIVE = "neg"
)
View Source 
const (
	// SORT_ASC indicates that we want to sort in ascending order.
	SORT_ASC = "asc"

	// SORT_DESC indicates that we want to sort in descending order.
	SORT_DESC = "desc"

	// MAX_ROW_DIGESTS is the maximum number of digests we'll compare against
	// before limiting the result to avoid overload.
	MAX_ROW_DIGESTS = 200

	// MAX_LIMIT is the maximum limit we will return.
	MAX_LIMIT = 200
)
View Source 
const (
	GROUP_TEST_MAX_COUNT = "count"
)

TODO: filter within tests.

View Source 
const (
	// MAX_REF_DIGESTS is the maximum number of digests we want to show
	// in a dotted line of traces. We assume that showing more digests yields
	// no additional information, because the trace is likely to be flaky.
	MAX_REF_DIGESTS = 9
)

Variables

This section is empty.

Functions

func DigestUrl 

func DigestUrl(baseURL, digest string) string

GetURL returns the URL given a base URL and the digest.

func ParseCTQuery 

func ParseCTQuery(r io.ReadCloser, limitDefault int32, ctQuery *CTQuery) error

ParseCTQuery parses JSON from the given ReadCloser into the given pointer to an instance of CTQuery. It will fill in values and validate key fields of the query. It will return an error if parsing failed for some reason and always close the ReadCloser. testName is the name of the test that should be compared and limitDefault is the default limit for the row and column queries.

func ParseQuery 

func ParseQuery(r *http.Request, query *Query) error

ParseQuery parses the request parameters from the URL query string or from the form parameters and stores the parsed and validated values in query.

func WriteExportTestRecords 

func WriteExportTestRecords(testRecs []*ExportTestRecord, writer io.Writer) error

WriteTestRecords writes the retrieved information about tests to the given writer JSON.

func WriteExportTestRecordsFile 

func WriteExportTestRecordsFile(testRecs []*ExportTestRecord, outputPath string) error

WriteExportTestRecordsFile writes the retrieved information about tests to a file as JSON.

Types

type AcceptFn 

type AcceptFn func(params paramtools.Params, digests []string) (bool, interface{})

AcceptFn is the callback function used by iterTile to determine whether to include a trace into the result or not. The second return value is an intermediate result that will be passed to addFn to avoid redundant computation. The second return value is application dependent since it will be passed into the call to the corresponding AddFn. Determining whether to accept a result might require an expensive computation and we want to avoid repeating that computation in the 'add' step. So we can return it here and it will be passed into the instance of AddFn.

type AddFn 

type AddFn func(test, digest, traceID string, trace *types.GoldenTrace, acceptResult interface{})

AddFn is the callback function used by iterTile to add a digest and it's trace to the result. acceptResult is the same value returned by the AcceptFn.

type CTDiffMetrics 

type CTDiffMetrics struct {
	*diff.DiffMetrics
	CTDigestCount
}

CTDiffMetrics contains diff metric between the contain digest and the corresponding row digest.

type CTDigestCount 

type CTDigestCount struct {
	Digest string `json:"digest"`
	N      int    `json:"n"`
}

CTDigestCount captures the digest and how often it appears in the tile.

type CTGrid 

type CTGrid struct {
	// Rows contains the row digest and the number of times it occurs.
	Rows []*CTRow `json:"rows"`

	// RowsTotal contains the total number of rows for the given query.
	RowsTotal int `json:"rowTotal"`

	// Columns contains the reference points calculated for each row digests.
	Columns []string `json:"columns"` // Contains the column types.

	// ColumnsTotal contains the total number of column digests.
	ColumnsTotal int `json:"columnsTotal"`
}

CTGrid contains the grid of diff values returned by CompareTest.

type CTQuery 

type CTQuery struct {
	// RowQuery is the query to select the row digests.
	RowQuery *Query `json:"rowQuery"`

	// ColumnQuery is the query to select the column digests.
	ColumnQuery *Query `json:"columnQuery"`

	// Match is the list of parameter fields where the column digests have to match
	// the value of the row digests. That means column digests will only be included
	// if the corresponding parameter values match the corresponding row digest.
	Match []string `json:"match"`

	// SortRows defines by what to sort the rows.
	SortRows string `json:"sortRows"`

	// SortColumns defines by what to sort the digest.
	SortColumns string `json:"sortColumns"`

	// RowsDir defines the sort direction for rows.
	RowsDir string `json:"rowsDir"`

	// ColumnsDir defines the sort direction for columns.
	ColumnsDir string `json:"columnsDir"`

	// Metric is the diff metric to use for sorting.
	Metric string `json:"metric"`
}

CTQuery is the input structure to the CompareTest function.

type CTResponse 

type CTResponse struct {
	Grid      *CTGrid               `json:"grid"`
	Name      string                `json:"name"`
	Corpus    string                `json:"source_type"`
	Summaries map[string]*CTSummary `json:"summaries"`
	Positive  int                   `json:"pos"`
	Negative  int                   `json:"neg"`
	Untriaged int                   `json:"unt"`
}

CTResponse is the structure returned by the CompareTest.

func CompareTest 

func CompareTest(ctq *CTQuery, storages *storage.Storage, idx *indexer.SearchIndex) (*CTResponse, error)

CompareTest allows to compare the digests within one test. It assumes that the provided instance of CTQuery is consistent in that the row query and column query contain the same test names and the same corpus field.

type CTRow 

type CTRow struct {
	CTDigestCount
	TestName string           `json:"test"`
	Values   []*CTDiffMetrics `json:"values"`
}

CTRow is used by CTGrid to encode row digest information.

type CTSummary 

type CTSummary struct {
	Pos       int `json:"pos"`
	Neg       int `json:"neg"`
	Untriaged int `json:"untriaged"`
}

type CommitRange 

type CommitRange struct {
}

CommitRange is a range of commits, starting at the git hash Begin and ending at End, inclusive.

Currently unimplemented in search.

type Diff 

type Diff struct {
	Diff float32 `json:"diff"` // The smaller of the Pos and Neg diff.

	// Either may be nil if there's no positive or negative to compare against.
	Pos *DiffDigest `json:"pos"`
	Neg *DiffDigest `json:"neg"`

	Blame *blame.BlameDistribution `json:"blame"`
}

Diff is only populated for digests that are untriaged? Might still be useful to find diffs to closest pos for a neg, and vice-versa. Will also be useful if we ever get a canonical trace or centroid.

type DiffDigest 

type DiffDigest struct {
	Closest  *digesttools.Closest `json:"closest"`
	ParamSet map[string][]string  `json:"paramset"`
}

DiffDigest is information about a digest different from the one in Digest.

type Digest 

type Digest struct {
	Test     string              `json:"test"`
	Digest   string              `json:"digest"`
	Status   string              `json:"status"`
	ParamSet map[string][]string `json:"paramset"`
	Traces   *Traces             `json:"traces"`
	Diff     *Diff               `json:"diff"`
}

Digest's are returned from Search, one for each match to Query.

type DigestSlice 

type DigestSlice []*Digest

DigestSlice is a utility type for sorting slices of Digest by their max diff.

func (DigestSlice) Len 

func (p DigestSlice) Len() int

func (DigestSlice) Less 

func (p DigestSlice) Less(i, j int) bool

func (DigestSlice) Swap 

func (p DigestSlice) Swap(i, j int)

type DigestStatus 

type DigestStatus struct {
	Digest string `json:"digest"`
	Status string `json:"status"`
}

DigestStatus is a digest and its status, used in Traces.

type ExpSlice 

type ExpSlice []types.Expectations

func (ExpSlice) Classification 

func (e ExpSlice) Classification(test, digest string) types.Label

type ExportDigestInfo 

type ExportDigestInfo struct {
	*SRDigest        // Same digest information as returned by search results.
	URL       string // URL from which to retrieve the image.
}

ExportDigestInfo contains information about one test result. This include the parameter sets.

type ExportTestRecord 

type ExportTestRecord struct {
	TestName string              `json:"testName"`
	Digests  []*ExportDigestInfo `json:"digests"`
}

ExportTestRecord accumulates the images/digests generated for one test. This is the format of the meta.json file.

func GetExportRecords 

func GetExportRecords(searchResp *NewSearchResponse, imgBaseURL string) []*ExportTestRecord

GetExportRecords converts a given search response into a slice of ExportTestRecords.

func ReadExportTestRecords 

func ReadExportTestRecords(reader io.Reader) ([]*ExportTestRecord, error)

ReadExportTestRecords loads a file with test records.

type IssueResponse 

type IssueResponse struct {
	QueryPatchsets []int64
}

IssueResponse contains specific query responses when we search for a trybot issue. Currently it extends trybot.IssueDetails.

type IssueSummary 

type IssueSummary struct {
	ID          int64              `json:"id"`          // issue ID
	TimeStampMs int64              `json:"timeStampMs"` // timestamp of the last update
	PatchSets   []*PatchsetSummary `json:"patchsets"`   // patchset information
}

IssueSummary contains a highl level summary of a Gerrit issue.

type NewSearchResponse 

type NewSearchResponse struct {
	Digests []*SRDigest        `json:"digests"`
	Offset  int                `json:"offset"`
	Size    int                `json:"size"`
	Commits []*tiling.Commit   `json:"commits"`
	Issue   *tryjobstore.Issue `json:"issue"`
}

NewSearchResponse is the structure returned by the Search(...) function of SearchAPI and intended to be returned as JSON in an HTTP response.

type PatchsetSummary 

type PatchsetSummary struct {
	PatchsetID   int64 `json:"patchsetID"`   // id of the patchset withing the issue
	TotalJobs    int   `json:"totalJobs"`    // total number of tryjobs known to gold
	FinishedJobs int   `json:"finishedJobs"` // number of tryjobs ingested by gold
	TotalImg     int   `json:"totalImg"`     // total images created by all tryjobs
	NewImg       int   `json:"newImg"`       // number of new images (not in master)
	UntriagedImg int   `json:"untriagedImg"` // number of untriaged images (<= newImg)
}

PatchsetSummary contains a high level summary of a Gerrit patchset.

type Point 

type Point struct {
	X int `json:"x"` // The commit index [0-49].
	Y int `json:"y"`
	S int `json:"s"` // Status of the digest: 0 if the digest matches our search, 1-8 otherwise.
}

Point is a single point. Used in Trace.

type Query 

type Query struct {
	// Diff metric to use.
	Metric string   `json:"metric"`
	Sort   string   `json:"sort"`
	Match  []string `json:"match"`

	// Blaming
	BlameGroupID string `json:"blame"`

	// Image classification
	Pos            bool `json:"pos"`
	Neg            bool `json:"neg"`
	Head           bool `json:"head"`
	Unt            bool `json:"unt"`
	IncludeIgnores bool `json:"include"`

	// URL encoded query string
	QueryStr string     `json:"query"`
	Query    url.Values `json:"-"`

	// URL encoded query string to select the right hand side of comparisons.
	RQueryStr string              `json:"rquery"`
	RQuery    paramtools.ParamSet `json:"-"`

	// Trybot support.
	IssueStr      string  `json:"issue"`
	Issue         int64   `json:"-"`
	PatchsetsStr  string  `json:"patchsets"` // Comma-separated list of patchsets.
	Patchsets     []int64 `json:"-"`
	IncludeMaster bool    `json:"master"` // Include digests also contained in master when searching code review issues.

	// Filtering.
	FCommitBegin string  `json:"fbegin"`     // Start commit
	FCommitEnd   string  `json:"fend"`       // End commit
	FRGBAMin     int32   `json:"frgbamin"`   // Min RGBA delta
	FRGBAMax     int32   `json:"frgbamax"`   // Max RGBA delta
	FDiffMax     float32 `json:"fdiffmax"`   // Max diff according to metric
	FGroupTest   string  `json:"fgrouptest"` // Op within grouped by test.
	FRef         bool    `json:"fref"`       // Only digests with reference.

	// Pagination.
	Offset int32 `json:"offset"`
	Limit  int32 `json:"limit"`

	// Do not include diffs in search.
	NoDiff bool `json:"nodiff"`
}

Query is the query that Search understands.

type RefDiffer 

type RefDiffer struct {
	// contains filtered or unexported fields
}

RefDiffer aggregates the helper objects needed to calculate reference diffs.

func NewRefDiffer 

func NewRefDiffer(exp ExpSlice, diffStore diff.DiffStore, idx *indexer.SearchIndex) *RefDiffer

func (*RefDiffer) GetRefDiffs 

func (r *RefDiffer) GetRefDiffs(metric string, match []string, test, digest string, params paramtools.ParamSet, rhsQuery paramtools.ParamSet, includeIgnores bool) (string, map[string]*SRDiffDigest)

GetRefDiffs calculates the reference diffs between the given digest and the other digests in the same test based on the given metric. 'match' is the list of parameters that need to match between the digests that are compared, i.e. this allows to restrict comparison of gamma correct images to other digests that are also gamma correct.

type SRDiffDigest 

type SRDiffDigest struct {
	*diff.DiffMetrics
	Test     string              `json:"test"`
	Digest   string              `json:"digest"`
	Status   string              `json:"status"`
	ParamSet map[string][]string `json:"paramset"`
	N        int                 `json:"n"`
}

SRDiffDigest captures the diff information between a primary digest and the digest given here. The primary digest is given by the context where this is used.

type SRDigest 

type SRDigest struct {
	Test       string                   `json:"test"`
	Digest     string                   `json:"digest"`
	Status     string                   `json:"status"`
	ParamSet   map[string][]string      `json:"paramset"`
	Traces     *Traces                  `json:"traces"`
	ClosestRef string                   `json:"closestRef"`
	RefDiffs   map[string]*SRDiffDigest `json:"refDiffs"`
	Blame      *blame.BlameDistribution `json:"blame"`
}

SRDigest is a single search result digest returned by the Search function below.

type SRDigestDetails 

type SRDigestDetails struct {
	Digest  *SRDigest        `json:"digest"`
	Commits []*tiling.Commit `json:"commits"`
}

DigestDetails contains details about a digest.

type SRDigestDiff 

type SRDigestDiff struct {
	Left  *SRDigest     `json:"left"`  // The left hand digest and its params.
	Right *SRDiffDigest `json:"right"` // The right hand digest, its params and the diff result.
}

SRDigestDiff contains the result of comparing two digests.

func CompareDigests 

func CompareDigests(test, left, right string, storages *storage.Storage, idx *indexer.SearchIndex) (*SRDigestDiff, error)

CompareDigests compares two digests that were generated by the given test. It returns an instance of DigestDiff.

type SearchAPI 

type SearchAPI struct {
	// contains filtered or unexported fields
}

SearchAPI is type that exposes a search API to query the current tile (all images for the most recent commits).

func NewSearchAPI 

func NewSearchAPI(storages *storage.Storage, ixr *indexer.Indexer) (*SearchAPI, error)

Create a new instance of SearchAPI.

func (*SearchAPI) GetDigestDetails 

func (s *SearchAPI) GetDigestDetails(test, digest string) (*SRDigestDetails, error)

GetDigestDetails returns details about a digest as an instance of SRDigestDetails.

func (*SearchAPI) Search 

func (s *SearchAPI) Search(ctx context.Context, q *Query) (*NewSearchResponse, error)

Search queries the current tile based on the parameters specified in the instance of Query.

func (*SearchAPI) Summary 

func (s *SearchAPI) Summary(issueID int64) (*IssueSummary, error)

Summary returns a high level summary of a Gerrit issue and the tryjobs that have been run for it.

type SearchResponse 

type SearchResponse struct {
	Digests       []*Digest
	Total         int32
	Commits       []*tiling.Commit
	IssueResponse *IssueResponse
}

SearchResponse is the standard search response. Depending on the query some fields might be empty, i.e. IssueDetails only makes sense if a trybot isssue was given in the query.

type Trace 

type Trace struct {
	Data   []Point           `json:"data"`  // One Point for each test result.
	ID     string            `json:"label"` // The id of the trace. Keep the json as label to be compatible with dots-sk.
	Params map[string]string `json:"params"`
}

Trace describes a single trace, used in Traces.

type Traces 

type Traces struct {
	TileSize int            `json:"tileSize"`
	Traces   []Trace        `json:"traces"`  // The traces where this digest appears.
	Digests  []DigestStatus `json:"digests"` // The other digests that appear in Traces.
}

Traces is info about a group of traces. Used in Digest.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.