artifacts

package
v0.0.0-...-e7509a3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 22, 2024 License: Apache-2.0 Imports: 30 Imported by: 0

Documentation

Overview

Package artifacts contains tools for reading and processing artifacts.

Index

Constants

View Source 
const (
	RFC3339FullDate = "2006-01-02"
	PstTimeZone     = "America/Los_Angeles"
	UtcTimeZone     = "UTC"
)

Variables

View Source 
var BQQueryTimeOutErr = errors.New("query can't finish within the deadline: please try different filters or tighter time range")
View Source 
var ContentHashRe = regexp.MustCompile(fmt.Sprintf(`^%s:%s$`, hashFunc, sha256Pattern))
View Source 
var SupportedContentTypes = map[string]bool{
	"application/octet-stream": true,
	"application/x-gzip":       true,
}

Functions

func AddHashPrefix 

func AddHashPrefix(hash string) string

AddHashPrefix adds HashFunc to a given hash string.

func FetchPassingHashes 

func FetchPassingHashes(ctx context.Context, client *bigquery.Client, realm string, testID string, artifactID string, numPassesToCompare int) (map[int64]struct{}, error)

FetchPassingHashes fetches the content of numPassesToCompare examples of the given log file from passing test results and normalizes and then hashes each line in the same fashion as HashLine.

This is currently implemented using BigQuery, and the hashing and normalization is done inside the query for parralellization and to minimize data transfer.

func HashLine 

func HashLine(line string) int64

HashLine calculates a hash for a line of a log file. It first normalizes the line to increase the probablility that it will match in two different test runs (e.g. removing timestamps). It uses the farm.Fingerprint64 hash as it needs to match the hash done in BigQuery above.

func MustParseName 

func MustParseName(name string) (invID invocations.ID, testID, resultID, artifactID string)

MustParseName extracts invocation, test, result and artifactIDs. Test and result IDs are "" if this is a invocation-level artifact. Panics on failure.

func ParentID 

func ParentID(testID, resultID string) string

ParentID returns a value for Artifacts.ParentId Spanner column.

func ParseParentID 

func ParseParentID(parentID string) (testID, resultID string, err error)

ParseParentID parses parentID into testID and resultID. If the artifact's parent is invocation, then testID and resultID are "".

func RegexPattern 

func RegexPattern(matcher *pb.ArtifactContentMatcher) string

func ToFailureOnlyLineRanges 

func ToFailureOnlyLineRanges(artifactID string, contentType string, content []byte, passingHashes map[int64]struct{}, includeContent bool) ([]*pb.QueryArtifactFailureOnlyLinesResponse_LineRange, error)

ToFailureOnlyLineRanges fetches the content of an artifact and returns only the line ranges that do not hash to one fo the values in passingHashes (using HashLine).

This function will optionally include the content of the failure only lines in the ranges.

func ToLogLines 

func ToLogLines(artifactID string, contentType string, content []byte, year, maxLines, maxBytes int) ([]*pb.ArtifactLine, error)

ToLogLines retrieves and processes an artifact and returns its content as a set of log lines. It executes best effort extraction of the timestamp and the severity of each log line. The `year` is used as a fallback if the year was absent from the log line's timestamp. The max specified the maximum number of results to return, if the max <= 0 it will return all lines.

func TrimHashPrefix 

func TrimHashPrefix(hash string) string

TrimHashPrefix removes HashFunc from a given hash string.

func VerifyReadArtifactPermission 

func VerifyReadArtifactPermission(ctx context.Context, name string) error

VerifyReadArtifactPermission verifies if the caller has enough permissions to read the artifact.

Types

type Artifact 

type Artifact struct {
	*pb.Artifact
	RBECASHash string
}

Artifact contains pb.Artifact and its RBECAS hash.

func Read 

func Read(ctx context.Context, name string) (*Artifact, error)

Read reads an artifact and returns an Artifact row from the database. It contains the following fields: * ContentType * Size * GcsURI * RBECASHash

type ArtifactGroup 

type ArtifactGroup struct {
	// For invocation level artifact, this field will be empty.
	TestID string
	// For invocation level artifact, this is union of all variants of test results directly included by the invocation.
	// For result level artifact, this is the test variant.
	Variant bigquery.NullJSON
	// A hash of the variant above.
	VariantHash string
	ArtifactID  string
	// At most 3 matching artifacts are included here.
	// MatchingArtifacts are sorted in partition_time DESC, invocation_id, result_id order.
	Artifacts []*MatchingArtifact
	// Total number of matching artifacts.
	MatchingCount int32
	// Maximum partition_time of artifacts in this group.
	MaxPartitionTime time.Time
}

ArtifactGroup represents matching artifacts in each (test id, variant hash, artifact id) group. This can be either invocation or test result level artifacts.

type BQClient 

type BQClient interface {
	ReadArtifactGroups(ctx context.Context, opts ReadArtifactGroupsOpts) (groups []*ArtifactGroup, nextPageToken string, err error)
	ReadArtifacts(ctx context.Context, opts ReadArtifactsOpts) (rows []*MatchingArtifact, nextPageToken string, err error)
}

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client to read ResultDB text_artifacts.

func NewClient 

func NewClient(ctx context.Context, gcpProject string) (*Client, error)

NewClient creates a new client for reading text_artifacts table.

func (*Client) Close 

func (c *Client) Close() error

Close releases any resources held by the client.

func (*Client) ReadArtifactGroups 

func (c *Client) ReadArtifactGroups(ctx context.Context, opts ReadArtifactGroupsOpts) (groups []*ArtifactGroup, nextPageToken string, err error)

ReadArtifactGroups reads either test result level artifacts or invocation level artifacts depending on opts.IsInvocationLevel.

func (*Client) ReadArtifacts 

func (c *Client) ReadArtifacts(ctx context.Context, opts ReadArtifactsOpts) (rows []*MatchingArtifact, nextPageToken string, err error)

ReadArtifacts reads either test result artifacts or invocation level artifacts, depending on opts.IsInvocationLevel.

type MatchingArtifact 

type MatchingArtifact struct {
	InvocationID string
	// For invocation level artifact, this field will be empty.
	ResultID string
	// The creation time of the parent invocation.
	PartitionTime time.Time
	// Test status associated with this artifact.
	// For invocation level artifact, this field will be empty.
	TestStatus bigquery.NullString
	// The first occurrence of a match in artifact content.
	Match string
	// Artifact content that is immediately before the match. At most one line above.
	MatchWithContextBefore string
	// Artifact content that is immediately after the match. At most one line below.
	MatchWithContextAfter string
}

MatchingArtifact is a single matching artifact.

type Query 

type Query struct {
	InvocationIDs       invocations.IDSet
	ParentIDRegexp      string
	FollowEdges         *pb.ArtifactPredicate_EdgeTypeSet
	TestResultPredicate *pb.TestResultPredicate
	ContentTypeRegexp   string
	ArtifactIDRegexp    string
	PageSize            int // must be positive
	PageToken           string
	WithRBECASHash      bool
	WithGcsURI          bool
}

Query specifies artifacts to fetch.

func (*Query) FetchProtos 

func (q *Query) FetchProtos(ctx context.Context) (arts []*pb.Artifact, nextPageToken string, err error)

FetchProtos returns a page of artifact protos matching q.

Returned artifacts are ordered by level (invocation or test result). Test result artifacts are sorted by parent invocation ID, test ID and artifact ID.

func (*Query) Run 

func (q *Query) Run(ctx context.Context, f func(*Artifact) error) error

Run calls f for artifacts matching the query.

Refer to Fetch() for the ordering of returned artifacts.

type ReadArtifactGroupsOpts 

type ReadArtifactGroupsOpts struct {
	Project      string
	SearchString *pb.ArtifactContentMatcher
	// This field is ignore, if IsInvocationLevel is True.
	TestIDMatcher     *pb.IDMatcher
	ArtifactIDMatcher *pb.IDMatcher
	// If true, query invocation level artifacts. Otherwise, query test result level artifacts.
	IsInvocationLevel bool
	StartTime         time.Time
	EndTime           time.Time
	SubRealms         []string
	Limit             int
	PageToken         string
}

type ReadArtifactsOpts 

type ReadArtifactsOpts struct {
	Project      string
	SearchString *pb.ArtifactContentMatcher
	// If true, query invocation level artifact. Otherwise, query test result level artifacts.
	IsInvocationLevel bool
	// If IsInvocationLevel is true, this field is ignored.
	TestID string
	// If IsInvocationLevel is true, this field is the hash of invocaiton variant union (union of all variants of test results directly included by the invocation).
	// Otherwise, this is the test variant hash.
	VariantHash string
	ArtifactID  string
	StartTime   time.Time
	EndTime     time.Time
	SubRealms   []string
	Limit       int
	PageToken   string
}

type Severity 

type Severity struct {
	Name    string
	Regexes []*regexp.Regexp
}

Source Files

View all Source files

Directories

Path Synopsis
Package testutil provides utility functions for testing with artifacts package.
 Package testutil provides utility functions for testing with artifacts package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.