performance

README

Performance tests

Knative performance tests are tests geared towards producing useful performance metrics of the knative system. As such they can choose to take a blackbox point-of-view of the system and use it just like an end-user might see it. They can also go more whiteboxy to narrow down the components under test.

Load Generator

Knative has a load generator library that can be used to generate load. The load generator uses Fortio for generating load based on the generator options set in the test. The load generator provides the generator results which includes all the data points for the load generation request and any calculated latencies.

For eg.

opts := loadgenerator.GeneratorOptions{Duration: 1*time.Minute, NumThreads: 1}
resp, err := opts.RunLoadTest(false /* resolvableDomain */)
if err != nil {
  t.Fatalf("Generating traffic via fortio failed: %v", err)
}

Prometheus metrics

Knative provides a prometheus wrapper that provides methods to wait for prometheus to scrap for the metrics once the test is finished. It also provides a way to query the prometheus server for any server-side metrics and then display those in testgrid

For eg.

promAPI, err := prometheus.PromAPI()
if err != nil {
  logger.Error("Cannot setup prometheus API")
}
query := fmt.Sprintf("%s{namespace_name=%q, configuration_name=%q, revision_name=%q}", metric, test.ServingNamespace, names.Config, names.Revision)
val, err := prometheus.RunQuery(context.Background(), logger, promAPI, query)
if err != nil {
  logger.Infof("Error querying metric %s: %v", metric, err)
}

Displaying metrics

Once the test is done, each test can define which metrics they want to be captured and shown on testgrid. For each metric, create a testgrid testcase by using the CreatePerfTestCase() method.

For eg.

import perf "knative.dev/test-infra/shared/performance"
testName := "TestPerformanceLatency"
var tc []testgrid.TestCase
for name, val := range metrics {
  tc = append(tc, perf.CreatePerfTestCase(val, name, testName))
}

Once we create all the test cases for the metrics, we can use the CreateTestgridXML() method to create the output xml that will be used as input to testgrid. If the test is run locally, it will create a file called junit_<testName>.xml and save it locally under knative/serving/test/performance/artifacts/(Note that this will create a directory called artifacts, if not present. Will use the existing one, if present). When the test is run with Prow, this file will be stored with the other artifacts generated by Prow like the build log.

For eg.

if err = testgrid.CreateTestgridXML(tc, testName); err != nil {
  t.Fatalf("Cannot create output xml: %v", err)
}

All the metrics are appended in the junit_knative.xml file and can be seen on testgrid

Adding performance tests

Performance tests are simple go tests, that use Fortio for generating load and Tesgrid to see the metrics on a continuous basis. For whitebox tests, the performance tests also bring up the prometheus service along with knative-serving. So, you can query prometheus to get server side metrics with the load generator metrics.

Testgrid

Knative uses testgrid to show all its metrics. Performance metrics are shown on a separate tab for each repo like serving. It will pick up the junit_knative.xml file generated by the tests and display the metric and the value in the grid in the performance tab automatically.

Documentation

Index

• Constants
• func TearDown(client *Client, names test.ResourceNames, logf logging.FormatLogger)
• type Client
  • func Setup(t *testing.T, monitoring ...int) (*Client, error)

Constants

const (
	EnablePrometheus = iota
)

Enable monitoring components

Functions

func TearDown

func TearDown(client *Client, names test.ResourceNames, logf logging.FormatLogger)

TearDown cleans up resources used

Types

type Client

type Client struct {
	E2EClients *test.Clients
	PromClient *prometheus.PromProxy
}

Client is the client used in the performance tests.

func Setup

func Setup(t *testing.T, monitoring ...int) (*Client, error)

Setup creates all the clients that we need to interact with in our tests