dataframe

Documentation

Overview

Package dataframe provides DataFrame which is a TraceSet with a calculated ParamSet and associated commit info.

Index

• Constants
• func DownSample(sample []*vcsinfo.IndexCommit, n int) ([]*vcsinfo.IndexCommit, int)
• type ColumnHeader
• type DataFrame
  • func Join(a, b *DataFrame) *DataFrame
  • func NewEmpty() *DataFrame
  • func NewHeaderOnly(vcs vcsinfo.VCS, begin, end time.Time, downsample bool) *DataFrame
  • func (d *DataFrame) BuildParamSet()
  • func (d *DataFrame) FilterOut(f TraceFilter)
  • func (d *DataFrame) Slice(offset, size int) (*DataFrame, error)
• type DataFrameBuilder
• type FrameRequest
  • func (f *FrameRequest) Id() string
• type FrameRequestProcess
  • func (p *FrameRequestProcess) Response() *FrameResponse
  • func (p *FrameRequestProcess) Run(ctx context.Context)
  • func (p *FrameRequestProcess) Status() (ProcessState, string, float32, error)
• type FrameResponse
  • func ResponseFromDataFrame(ctx context.Context, df *DataFrame, git *gitinfo.GitInfo, truncate bool, ...) (*FrameResponse, error)
• type ProcessState
• type Refresher
  • func NewRefresher(ctx context.Context, vcs vcsinfo.VCS, dfBuilder DataFrameBuilder, ...) (*Refresher, error)
  • func (f *Refresher) Get() *DataFrame
• type RequestType
• type RunningFrameRequests
  • func NewRunningFrameRequests(git *gitinfo.GitInfo, dfBuilder DataFrameBuilder) *RunningFrameRequests
  • func (fr *RunningFrameRequests) Add(ctx context.Context, req *FrameRequest) string
  • func (fr *RunningFrameRequests) Response(id string) (*FrameResponse, error)
  • func (fr *RunningFrameRequests) Status(id string) (ProcessState, string, float32, error)
• type TraceFilter

Constants

const (
	PROCESS_RUNNING ProcessState = "Running"
	PROCESS_SUCCESS ProcessState = "Success"
	PROCESS_ERROR   ProcessState = "Error"

	// Values for FrameRequest.RequestType.
	REQUEST_TIME_RANGE RequestType = 0
	REQUEST_COMPACT    RequestType = 1

	DEFAULT_COMPACT_NUM_COMMITS = 200
)

const (
	MAX_TRACES_IN_RESPONSE = 350

	// MAX_FINISHED_PROCESS_AGE is the amount of time to keep a finished
	// FrameRequestProcess around before deleting it.
	MAX_FINISHED_PROCESS_AGE = 10 * time.Minute
)

const (
	// DEFAULT_NUM_COMMITS is the number of commits in the DataFrame returned
	// from New().
	DEFAULT_NUM_COMMITS = 50

	MAX_SAMPLE_SIZE = 256
)

Functions

func DownSample

func DownSample(sample []*vcsinfo.IndexCommit, n int) ([]*vcsinfo.IndexCommit, int)

DownSample the given slice of IndexCommits so that there's no more than 'n' IndexCommits returned.

Returns the downsamples IndexCommits and the number of commits that were skipped. I.e. if N is returned then the returned slice will contain every (N+1)th commit.

Types

type ColumnHeader

type ColumnHeader struct {
	Source    string `json:"source"`
	Offset    int64  `json:"offset"`
	Timestamp int64  `json:"timestamp"` // In seconds from the Unix epoch.
}

ColumnHeader describes each column in a DataFrame.

type DataFrame

type DataFrame struct {
	TraceSet types.TraceSet      `json:"traceset"`
	Header   []*ColumnHeader     `json:"header"`
	ParamSet paramtools.ParamSet `json:"paramset"`
	Skip     int                 `json:"skip"`
}

DataFrame stores Perf measurements in a table where each row is a Trace indexed by a structured key (see go/query), and each column is described by a ColumnHeader, which could be a commit or a trybot patch level.

Skip is the number of commits skipped to bring the DataFrame down to less than MAX_SAMPLE_SIZE commits. If Skip is zero then no commits were skipped.

The name DataFrame was gratuitously borrowed from R.

func Join

func Join(a, b *DataFrame) *DataFrame

Join create a new DataFrame that is the union of 'a' and 'b'.

Will handle the case of a and b having data for different sets of commits, i.e. a.Header doesn't have to equal b.Header.

func NewEmpty

func NewEmpty() *DataFrame

NewEmpty returns a new empty DataFrame.

func NewHeaderOnly

func NewHeaderOnly(vcs vcsinfo.VCS, begin, end time.Time, downsample bool) *DataFrame

NewHeaderOnly returns a DataFrame with a populated Header, with no traces. The 'progress' callback is called periodically as the query is processed.

If 'downsample' is true then the number of commits returned is limited to MAX_SAMPLE_SIZE.

func (*DataFrame) BuildParamSet

func (d *DataFrame) BuildParamSet()

BuildParamSet rebuilds d.ParamSet from the keys of d.TraceSet.

func (*DataFrame) FilterOut

func (d *DataFrame) FilterOut(f TraceFilter)

FilterOut removes traces from d.TraceSet if the filter function 'f' returns true for a trace.

FilterOut rebuilds the ParamSet to match the new set of traces once filtering is complete.

func (*DataFrame) Slice

func (d *DataFrame) Slice(offset, size int) (*DataFrame, error)

Slice returns a dataframe that contains a subset of the current dataframe, starting from 'offset', the next 'size' num points will be returned as a new dataframe. Note that the data is composed of slices of the original data, not copies, so the returned dataframe must not be altered.

type DataFrameBuilder

type DataFrameBuilder interface {
	// New returns a populated DataFrame of the last 50 commits or a non-nil
	// error if there was a failure retrieving the traces.
	New(progress types.Progress) (*DataFrame, error)

	// NewN returns a populated DataFrame of the last N commits or a non-nil
	// error if there was a failure retrieving the traces.
	NewN(progress types.Progress, n int) (*DataFrame, error)

	// NewFromQueryAndRange returns a populated DataFrame of the traces that match
	// the given time range [begin, end) and the passed in query, or a non-nil
	// error if the traces can't be retrieved. The 'progress' callback is called
	// periodically as the query is processed.
	NewFromQueryAndRange(begin, end time.Time, q *query.Query, downsample bool, progress types.Progress) (*DataFrame, error)

	// NewFromKeysAndRange returns a populated DataFrame of the traces that match
	// the given set of 'keys' over the range of [begin, end). The 'progress'
	// callback is called periodically as the query is processed.
	NewFromKeysAndRange(keys []string, begin, end time.Time, downsample bool, progress types.Progress) (*DataFrame, error)

	// NewFromCommitIDsAndQuery returns a populated DataFrame of the traces that
	// match the given time set of commits 'cids' and the query 'q'. The 'progress'
	// callback is called periodically as the query is processed.
	NewFromCommitIDsAndQuery(ctx context.Context, cids []*cid.CommitID, cidl *cid.CommitIDLookup, q *query.Query, progress types.Progress) (*DataFrame, error)

	// NewNFromQuery returns a populated DataFrame of condensed traces of N data
	// points ending at the given 'end' time that match the given query.
	NewNFromQuery(ctx context.Context, end time.Time, q *query.Query, n int32, progress types.Progress) (*DataFrame, error)

	// NewNFromQuery returns a populated DataFrame of condensed traces of N data
	// points ending at the given 'end' time for the given keys.
	NewNFromKeys(ctx context.Context, end time.Time, keys []string, n int32, progress types.Progress) (*DataFrame, error)
}

DataFrameBuilder is an interface for things that construct DataFrames.

type FrameRequest

type FrameRequest struct {
	Begin       int         `json:"begin"`       // Beginning of time range in Unix timestamp seconds.
	End         int         `json:"end"`         // End of time range in Unix timestamp seconds.
	Formulas    []string    `json:"formulas"`    // The Formulae to evaluate.
	Queries     []string    `json:"queries"`     // The queries to perform encoded as a URL query.
	Hidden      []string    `json:"hidden"`      // The ids of traces to remove from the response.
	Keys        string      `json:"keys"`        // The id of a list of keys stored via shortcut2.
	TZ          string      `json:"tz"`          // The timezone the request is from. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/DateTimeFormat/resolvedOptions
	NumCommits  int32       `json:"num_commits"` // If RequestType is REQUEST_COMPACT, then the number of commits to show before End, and Begin is ignored.
	RequestType RequestType `json:"request_type"`
}

FrameRequest is used to deserialize JSON frame requests.

func (*FrameRequest) Id

func (f *FrameRequest) Id() string

type FrameRequestProcess

type FrameRequestProcess struct {
	// contains filtered or unexported fields
}

FrameRequestProcess keeps track of a running Go routine that's processing a FrameRequest to build a FrameResponse.

func (*FrameRequestProcess) Response

func (p *FrameRequestProcess) Response() *FrameResponse

Response returns the FrameResponse of the completed FrameRequestProcess.

func (*FrameRequestProcess) Run

func (p *FrameRequestProcess) Run(ctx context.Context)

Run does the work in a FrameRequestProcess. It does not return until all the work is done or the request failed. Should be run as a Go routine.

func (*FrameRequestProcess) Status

func (p *FrameRequestProcess) Status() (ProcessState, string, float32, error)

Status returns the ProcessingState, the message, and the percent complete of a FrameRequestProcess of the given 'id'.

type FrameResponse

type FrameResponse struct {
	DataFrame *DataFrame    `json:"dataframe"`
	Ticks     []interface{} `json:"ticks"`
	Skps      []int         `json:"skps"`
	Msg       string        `json:"msg"`
}

FrameResponse is serialized to JSON as the response to frame requests.

func ResponseFromDataFrame

func ResponseFromDataFrame(ctx context.Context, df *DataFrame, git *gitinfo.GitInfo, truncate bool, tz string) (*FrameResponse, error)

ResponseFromDataFrame fills out the rest of a FrameResponse for the given DataFrame.

If truncate is true then the number of traces returned is limited.

tz is the timezone, and can be the empty string if the default (Eastern) timezone is acceptable.

type ProcessState

type ProcessState string

type Refresher

type Refresher struct {
	// contains filtered or unexported fields
}

Refresher keeps a fresh DataFrame of the last DEFAULT_NUM_COMMITS commits.

func NewRefresher

func NewRefresher(ctx context.Context, vcs vcsinfo.VCS, dfBuilder DataFrameBuilder, period time.Duration, n int) (*Refresher, error)

NewRefresher creates a new Refresher that updates the dataframe every 'period'.

A non-nil error will be returned if the initial DataFrame cannot be populated. I.e. if NewRefresher returns w/o error than the caller can be assured that Get() will return a non-nil DataFrame.

func (*Refresher) Get

func (f *Refresher) Get() *DataFrame

Get returns a DataFrame. It is not safe for modification, only for reading.

type RequestType

type RequestType int

type RunningFrameRequests

type RunningFrameRequests struct {
	// contains filtered or unexported fields
}

RunningFrameRequests keeps track of all the FrameRequestProcess's.

Once a FrameRequestProcess is complete the results will be kept in memory for MAX_FINISHED_PROCESS_AGE before being deleted.

func NewRunningFrameRequests

func NewRunningFrameRequests(git *gitinfo.GitInfo, dfBuilder DataFrameBuilder) *RunningFrameRequests

func (*RunningFrameRequests) Add

func (fr *RunningFrameRequests) Add(ctx context.Context, req *FrameRequest) string

Add starts a new running FrameRequestProcess and returns the ID of the process to be used in calls to Status() and Response().

func (*RunningFrameRequests) Response

func (fr *RunningFrameRequests) Response(id string) (*FrameResponse, error)

Response returns the FrameResponse of the completed FrameRequestProcess.

func (*RunningFrameRequests) Status

func (fr *RunningFrameRequests) Status(id string) (ProcessState, string, float32, error)

Status returns the ProcessingState, the message, and the percent complete of a FrameRequestProcess of the given 'id'.

type TraceFilter

type TraceFilter func(tr types.Trace) bool

TraceFilter is a function type that should return true if trace 'tr' should be removed from a DataFrame. It is used in FilterOut.