testvariantbranch

package
v0.0.0-...-a70aae3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 14, 2024 License: Apache-2.0 Imports: 18 Imported by: 0

Documentation

Overview

Package testvariantbranch handles test variant branch of change point analysis.

Index

Constants

View Source 
const (

	// StatisticsRetentionDays is the number of days to keep statistics about
	// evicted source verdicts. See Statistics proto for more.
	//
	// This is a minimum period driven by 7 days of lookback required for
	// functional reasons, plus four days to allow for late data ingestion.
	// Our deletion logic will tend to keep retain data for longer (but this is
	// OK as it is not user data).
	StatisticsRetentionDays = 11
)

Variables

This section is empty.

Functions

func AddCounts 

func AddCounts(counts *cpb.Counts, runs []*inputbuffer.Run) *cpb.Counts

AddCounts updates counts with the given new runs.

func DecodeProtoMessage 

func DecodeProtoMessage(bytes []byte, m proto.Message) error

DecodeProtoMessage decodes a byte slice into a proto message. It is the inverse of EncodeProtoMessage.

func DecodeSegment 

func DecodeSegment(bytes []byte) (*cpb.Segment, error)

DecodeSegment decodes []byte in to Segment.

func DecodeSegments 

func DecodeSegments(bytes []byte) (*cpb.Segments, error)

DecodeSegments decodes []byte in to Segments.

func DecodeSourceRef 

func DecodeSourceRef(bytes []byte) (*pb.SourceRef, error)

DecodeSourceRef decodes []byte in to SourceRef.

func DecodeStatistics 

func DecodeStatistics(bytes []byte) (*cpb.Statistics, error)

DecodeStatistics decodes []byte into a Statistics proto.

func EncodeProtoMessage 

func EncodeProtoMessage(m proto.Message) ([]byte, error)

EncodeProtoMessage uses zstd to encode a proto message into []byte.

func EncodeSegment 

func EncodeSegment(seg *cpb.Segment) ([]byte, error)

func EncodeSegments 

func EncodeSegments(segs *cpb.Segments) ([]byte, error)

func EncodeSourceRef 

func EncodeSourceRef(sourceRef *pb.SourceRef) ([]byte, error)

func EncodeStatistics 

func EncodeStatistics(stats *cpb.Statistics) ([]byte, error)

func NewRunStreamAggregator 

func NewRunStreamAggregator(partialVerdict *cpb.PartialSourceVerdict) *runStreamAggregator

NewRunStreamAggregator intialises a new streaming aggregator for converting test runs (in ascending commit position order) to source verdicts, optionally recovering the aggregation state from a partial source verdict.

func ReadF 

func ReadF(ctx context.Context, ks []Key, f func(i int, e *Entry) error) error

ReadF fetches rows from TestVariantBranch spanner table and calls the specified callback function for each row.

Important: The caller must not retain a reference to the provided *Entry after each call, as the object may be re-used for in the next call. If an entry must be retained, it should be copied.

The callback function will be passed the read row and the corresponding index in the key list that was read. If an item does not exist, the provided callback function will be called with the value 'nil'. Absent any errors, the callback function will be called exactly once for each key. The callback function may not be called in order of keys, as Spanner does not return items in order.

This function assumes that it is running inside a transaction.

func ToRun 

func ToRun(v *rdbpb.RunTestVerdict, partitionTime time.Time, src *pb.Sources) inputbuffer.Run

func ToRuns 

func ToRuns(tv *rdbpb.TestVariant, partitionTime time.Time, claimedInvs map[string]bool, src *pb.Sources) ([]inputbuffer.Run, error)

ToRuns converts a test verdict to a set of runs for the input buffer. The runs in verdict details are ordered by: - UnexpectedCount, descendingly, then - ExpectedCount, descendingly.

Types

type Entry 

type Entry struct {
	// IsNew is a boolean to denote if the TestVariantBranch is new or already
	// existed in Spanner.
	// It is used for reducing the number of mutations. For example, the Variant
	// field is only inserted once.
	IsNew       bool
	Project     string
	TestID      string
	VariantHash string
	Variant     *pb.Variant
	RefHash     []byte
	SourceRef   *pb.SourceRef
	InputBuffer *inputbuffer.Buffer
	// If this is true, it means we should trigger a write of FinalizingSegment
	// to Spanner.
	IsFinalizingSegmentDirty bool
	// The finalizing segment, if any.
	// The count for the finalizing segment should only include the verdicts
	// that are not in the input buffer anymore.
	FinalizingSegment *cpb.Segment
	// If this is true, it means we should trigger a write of FinalizedSegments
	// to Spanner.
	IsFinalizedSegmentsDirty bool
	// The finalized segments for the test variant branch.
	FinalizedSegments *cpb.Segments
	// If true, it means we should trigger a write of Statistics to Spanner.
	IsStatisticsDirty bool
	// Statistics about verdicts which have been evicted from the input buffer.
	Statistics *cpb.Statistics
}

Entry represents one row in the TestVariantBranch spanner table. See go/luci-test-variant-analysis-design for details.

func New 

func New() *Entry

New creates a new empty test variant branch entry, with a preallocated input buffer.

func QueryVariantBranches 

func QueryVariantBranches(ctx context.Context, project, testID string, refHash []byte, opts QueryVariantBranchesOptions) (entries []*Entry, nextPageToken string, err error)

func Read 

func Read(ctx context.Context, ks []Key) (rs []*Entry, retErr error)

Read fetches rows from TestVariantBranch spanner table and returns the objects fetched. The returned slice will have the same length and order as the TestVariantBranchKey slices. If a record is not found, the corresponding element will be set to nil. This function assumes that it is running inside a transaction.

func (*Entry) ApplyRetentionPolicyForFinalizedSegments 

func (tvb *Entry) ApplyRetentionPolicyForFinalizedSegments(fromTime time.Time)

ApplyRetentionPolicyForFinalizedSegments applies retention policy to finalized segments. The following retention policy applies to finalized segments:

  • At most 100 finalized segments can be stored.
  • Finalized segments are retained for 5 years from when they closed.

fromTime is the time when the 5 year period is calculated from.

The retention policy to delete test variant branches without test results in 90 days will be enforced separately with a cron job.

func (*Entry) Clear 

func (tvb *Entry) Clear()

Clear resets a test variant branch entry to an empty state, similar to after a call to New().

func (*Entry) Copy 

func (tvb *Entry) Copy() *Entry

Copy makes a deep copy of a test variant branch entry.

func (*Entry) HourlyStatistics 

func (tvb *Entry) HourlyStatistics() map[int64]HourlyStats

HourlyStatistics returns statistics about the verdicts ingested for given test variant branch. Statistics comprise data from both the input buffer and the output buffer.

The results are in a map keyed by hour (unix seconds / 3600).

func (*Entry) InsertFinalizedSegment 

func (tvb *Entry) InsertFinalizedSegment(segment *cpb.Segment)

InsertFinalizedSegment inserts a segment to the end of finalized segments.

func (*Entry) InsertToInputBuffer 

func (tvb *Entry) InsertToInputBuffer(r inputbuffer.Run) bool

InsertToInputBuffer attempts to insert data of a new test run into the input buffer. This method returns true if the test run could be inserted, false if it is too far out of order.

func (*Entry) PopulateFromSpannerRow 

func (tvb *Entry) PopulateFromSpannerRow(row *spanner.Row, hs *inputbuffer.HistorySerializer) error

func (*Entry) ToMutation 

func (tvb *Entry) ToMutation(hs *inputbuffer.HistorySerializer) (*spanner.Mutation, error)

ToMutation returns a spanner Mutation to insert a TestVariantBranch to Spanner table.

func (*Entry) UpdateOutputBuffer 

func (tvb *Entry) UpdateOutputBuffer(evictedSegments []inputbuffer.EvictedSegment)

UpdateOutputBuffer updates the output buffer with the evicted segments from the input buffer. evictedSegments should contain only finalized segments, except for the last segment (if any), which must be a finalizing segment. evictedSegments is sorted in ascending order of commit position (oldest segment first).

type HourlyStats 

type HourlyStats struct {
	TotalSourceVerdicts      int64
	FlakySourceVerdicts      int64
	UnexpectedSourceVerdicts int64
}

type Key 

type Key struct {
	Project     string
	TestID      string
	VariantHash string
	// Make this as a string here so it can be used as key in map.
	// Note that it is a sequence of bytes, not a sequence of characters.
	RefHash RefHash
}

Key denotes the primary key for the TestVariantBranch table.

type QueryVariantBranchesOptions 

type QueryVariantBranchesOptions struct {
	PageSize  int
	PageToken string
}

QueryVariantBranchesOptions specifies options for QueryVariantBranches().

type RefHash 

type RefHash string

RefHash is used for RefHash field in TestVariantBranchKey.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.