perf

README

Performance Testing Framework

The perf sub-module provides a framework for writing and running performance tests.

Default Command Options

Flag	Short Flag	Default Value	Description
--duration	-d	10 seconds	How long to run an individual performance test
--test-proxies	-x	N/A	A semicolon separated list of proxy urls.
--warmup	-w	3 seconds	How long to allow the connection to warm up.

Adding Performance Tests to an SDK

1. Create a performance test directory at testdata/perf within your module. For example, the storage performance tests live in sdk/storage/azblob/testdata/perf.

2. Run go mod init to create a new module. Add the sdk/internal module and the module you are testing for to your go.mod file and run go mod tidy

3. Create a struct that maintains all global values for the performance test (ie. account name, blob name, etc.).

4. Implement GlobalPerfTest interface by adding the NewPerfTest(context.Context, *PerfTestOptions) (PerfTest, error) and the GlobalCleanup(context.Context) functions on the global struct. The NewPerfTest method creates a new struct that is responsible for running a performance test in its own goroutine. GlobalCleanup takes care of any session level cleanup.

5. Implement the PerfTest interface on the struct returned from the NewPerfTest method. Add a Run(context.Context) and a Cleanup(context.Context) function. The Run method is the method you want to measure the performance of. The Cleanup method is for cleaning up within a single goroutine, most of the time this will be empty.

6. Create a main package that uses the perf.Run function to provide all the global constructor methods. perf.Run takes a map[string]PerfMethods

7. (Optional): Create a method that registers local flags.

package main

import (
    "github.com/Azure/azure-sdk-for-go/sdk/internal/perf"
)

// uploadTestRegister registers flags for the "UploadBlobTest"
// This is optional and does not to be included for every test.
func uploadTestRegister() {
	flag.IntVar(&uploadTestOpts.size, "size", 10240, "Size in bytes of data to be transferred in upload or download tests. Default is 10240.")
}

func main() {
	perf.Run(map[string]perf.PerfMethods{
		"UploadBlobTest":   {Register: uploadTestRegister, New: NewUploadTest},
		"ListBlobTest":     {Register: listTestRegister, New: NewListTest},
		"DownloadBlobTest": {Register: downloadTestRegister, New: NewDownloadTest},
	})
}

Writing a test

The following walks through the azblob Blob Upload performance test:

GlobalPerfTest Interface

This struct handles global set up for your account and spawning structs for each goroutine. Make sure to embed the perf.PerfTestOptions struct in your global struct. The NewUploadTest will be run once per process and is used to spawn each parallel test.

type uploadTestGlobal struct {
	perf.PerfTestOptions
	containerName         string
	blobName              string
	globalContainerClient azblob.ContainerClient
}

// NewUploadTest is called once per process
func NewUploadTest(ctx context.Context, options perf.PerfTestOptions) (perf.GlobalPerfTest, error) {
	u := &uploadTestGlobal{
		PerfTestOptions: options,
		containerName:   "uploadcontainer",
		blobName:        "uploadblob",
	}

	connStr, ok := os.LookupEnv("AZURE_STORAGE_CONNECTION_STRING")
	if !ok {
		return nil, fmt.Errorf("the environment variable 'AZURE_STORAGE_CONNECTION_STRING' could not be found")
	}

	containerClient, err := azblob.NewContainerClientFromConnectionString(connStr, u.containerName, nil)
	if err != nil {
		return nil, err
	}
	u.globalContainerClient = containerClient
	_, err = u.globalContainerClient.Create(context.Background(), nil)
	if err != nil {
		return nil, err
	}

	return u, nil
}

The GlobalCleanup method cleans up the account, resetting it into the default.

func (m *uploadTestGlobal) GlobalCleanup(ctx context.Context) error {
	connStr, ok := os.LookupEnv("AZURE_STORAGE_CONNECTION_STRING")
	if !ok {
		return fmt.Errorf("the environment variable 'AZURE_STORAGE_CONNECTION_STRING' could not be found")
	}

	containerClient, err := azblob.NewContainerClientFromConnectionString(connStr, m.containerName, nil)
	if err != nil {
		return err
	}

	_, err = containerClient.Delete(context.Background(), nil)
	return err
}

NewPerfTest is called once per goroutine, and creates a new PerfTest interface which will be used by each goroutine. This method should also include the setup for the eventual returned struct.

// uploadPerfTest implements perf.PerfTest and is used to test performance of your event
type uploadPerfTest struct {
	*uploadTestGlobal
	perf.PerfTestOptions
	data       io.ReadSeekCloser
	blobClient azblob.BlockBlobClient
}

// NewPerfTest is called once per goroutine
func (g *uploadTestGlobal) NewPerfTest(ctx context.Context, options *perf.PerfTestOptions) (perf.PerfTest, error) {
	u := &uploadPerfTest{
		uploadTestGlobal: g,
		PerfTestOptions:  *options,
	}

	connStr, ok := os.LookupEnv("AZURE_STORAGE_CONNECTION_STRING")
	if !ok {
		return nil, fmt.Errorf("the environment variable 'AZURE_STORAGE_CONNECTION_STRING' could not be found")
	}

	containerClient, err := azblob.NewContainerClientFromConnectionString(
		connStr,
		u.uploadTestGlobal.containerName,
		&azblob.ClientOptions{
			Transporter: u.PerfTestOptions.Transporter,
		},
	)
	if err != nil {
		return nil, err
	}
	bc := containerClient.NewBlockBlobClient(u.blobName)
	u.blobClient = bc

	data, err := perf.NewRandomStream(uploadTestOpts.size)
	if err != nil {
		return nil, err
	}
	u.data = data

	return u, nil
}

PerfTest Interface

The PerfTest interface is an interface that is responsible for running a single performance test. Each performance test in run within a single goroutine, these goroutines are created by the perf framework. The Run method is the method that is being measured and the Cleanup method is responsible for cleanup work after each goroutine completes or errors out.

func (m *uploadPerfTest) Run(ctx context.Context) error {
	_, err := m.data.Seek(0, io.SeekStart) // rewind to the beginning
	if err != nil {
		return err
	}
	_, err = m.blobClient.Upload(ctx, m.data, nil)
	return err
}

func (m *uploadPerfTest) Cleanup(ctx context.Context) error {
	return nil
}

Constructor

Each method needs to have a constructor that returns an instantialized perf.GlobalPerfTest.

// NewUploadTest is called once per process
func NewUploadTest(ctx context.Context, options perf.PerfTestOptions) (perf.GlobalPerfTest, error) {
	u := &uploadTestGlobal{
		PerfTestOptions: options,
		containerName:   "uploadcontainer",
		blobName:        "uploadblob",
	}

	connStr, ok := os.LookupEnv("AZURE_STORAGE_CONNECTION_STRING")
	if !ok {
		return nil, fmt.Errorf("the environment variable 'AZURE_STORAGE_CONNECTION_STRING' could not be found")
	}

	containerClient, err := azblob.NewContainerClientFromConnectionString(connStr, u.containerName, nil)
	if err != nil {
		return nil, err
	}
	_, err = containerClient.Create(context.Background(), nil)
	if err != nil {
		return nil, err
	}

	return u, nil
}

main.go file

The main function must have the perf.Run method with a map of each performance test constructor. Flag parsing, test selection, reporting, proxy interfaces, and all other portions are taken care of for the user in the perf.Run method.

package main

import (
	"github.com/Azure/azure-sdk-for-go/sdk/internal/perf"
)

func main() {
	perf.Run(map[string]perf.PerfMethods{
		"UploadBlobTest":   {Register: uploadTestRegister, New: NewUploadTest},
		"ListBlobTest":     {Register: listTestRegister, New: NewListTest},
		"DownloadBlobTest": {Register: downloadTestRegister, New: NewDownloadTest},
	})
}

Running with the test proxy

To run with the test proxy, configure your client to route requests to the test proxy using the Transporter parameters for the Transport in your client options. If the --test-proxy flag is not specified, Transporter is the same default HTTP client from azcore.

func (k *keysPerfTest) GlobalSetup() error {
    ...
    options = &azkeys.ClientOptions{
        ClientOptions: azcore.ClientOptions{
            Transport: k.Transporter,
        },
    }

    client, err := azkeys.NewClient("<my-vault-url>", cred, options)
    if err != nil {return err}
    ...
}

Registering Local Arguments

We use the flag library for argument parsing. Each perf test suite can have optional arguments and they are registered using the perf.RegisterArguments method, which takes a simple function that registers arguments using flag.Int, flag.String, etc. If you have no local arguments, you can skip this step.

type uploadTestOptions struct {
	size int
}

var uploadTestOpts uploadTestOptions = uploadTestOptions{size: 10240}

// uploadTestRegister is called once per process
func uploadTestRegister() {
	flag.IntVar(&uploadTestOpts.size, "size", 10240, "Size in bytes of data to be transferred in upload or download tests.")
}

The main.go file changes to:

package main

import "github.com/Azure/azure-sdk-for-go/sdk/internal/perf"

func main() {
	perf.Run(map[string]perf.PerfMethods{
		"UploadBlobTest":   {Register: uploadTestRegister, New: NewUploadTest},
		"ListBlobTest":     {Register: listTestRegister, New: NewListTest},
		"DownloadBlobTest": {Register: downloadTestRegister, New: NewDownloadTest},
	})
}

Running Performance Tests

First, make sure you have go version 1.17 or later installed. You can install go from the go.dev site.

Navigate to your SDK performance test, (ie. sdk/security/keyvault/azkeys/testdata/perf folder and run go mod download to download all the requirements for the performance framework.

To run a single performance test specify the test as the first argument:

go run . CreateEntityTest

To specify flags for a performance test, add them after the first argument:

go run . CreateEntityTest --duration 7 --test-proxies https://localhost:5001 --num-blobs 100

To run against multiple proxies and with multiple parallel tests running at the same time use the -p/--parallel flag

go run . CreateEntityTest --duration 7 --parallel 4 --test-proxies "https://localhost:5001;http://localhost:5000" --num-blobs 100

This runs four goroutines of the same test and splits the traffic to two different proxy addresses. The first and third routine will target https://localhost:5001, the second and fourth will use http://localhost:5000.

For help run go run . --help. A list of registered performance tests, global command flags, and local command flags will print out.

Documentation

Index

• func NewRandomStream(length int) (io.ReadSeekCloser, error)
• func Run(tests map[string]PerfMethods)
• type GlobalPerfTest
• type HTTPClient
• type NewPerfTest
• type PerfMethods
• type PerfTest
• type PerfTestOptions
• type RecordingHTTPClient
  • func NewProxyTransport(options *TransportOptions) *RecordingHTTPClient
  • func (c RecordingHTTPClient) Do(req *http.Request) (*http.Response, error)
  • func (c *RecordingHTTPClient) SetMode(mode string)
• type TransportOptions

Functions

func NewRandomStream

func NewRandomStream(length int) (io.ReadSeekCloser, error)

func Run

func Run(tests map[string]PerfMethods)

Run runs an individual test, registers, and parses command line flags

Types

type GlobalPerfTest

type GlobalPerfTest interface {
	// NewPerfTest creates an instance of a PerfTest for each goroutine.
	NewPerfTest(context.Context, *PerfTestOptions) (PerfTest, error)

	// GlobalCleanup is run one time per performance test, as the final method.
	GlobalCleanup(context.Context) error
}

GlobalPerfTest methods execute once per process

type HTTPClient

type HTTPClient interface {
	Do(*http.Request) (*http.Response, error)
}

HTTPClient is the same interface as azcore.Transporter

type NewPerfTest

type NewPerfTest func(context.Context, PerfTestOptions) (GlobalPerfTest, error)

NewPerfTest returns an instance of PerfTest and embeds the given `options` in the struct

type PerfMethods

type PerfMethods struct {
	Register func()
	New      func(context.Context, PerfTestOptions) (GlobalPerfTest, error)
}

PerfMethods is the struct given to the Run method.

type PerfTest

type PerfTest interface {
	// Run is the function that is being measured.
	Run(context.Context) error

	// Cleanup is run once for each parallel instance.
	Cleanup(context.Context) error
}

PerfTest methods once per goroutine

type PerfTestOptions

type PerfTestOptions struct {
	// Transporter is the azcore.Transporter instance used for sending requests.
	Transporter HTTPClient

	// Name is the name of an individual test
	Name string
	// contains filtered or unexported fields
}

PerfTestOptions are the options for a performance test. Name and Transporter can be used by an individual performance test.

type RecordingHTTPClient

type RecordingHTTPClient struct {
	// contains filtered or unexported fields
}

func NewProxyTransport

func NewProxyTransport(options *TransportOptions) *RecordingHTTPClient

NewRecordingHTTPClient returns a type that implements `azcore.Transporter`. This will automatically route tests on the `Do` call.

func (RecordingHTTPClient) Do

func (c RecordingHTTPClient) Do(req *http.Request) (*http.Response, error)

func (*RecordingHTTPClient) SetMode

func (c *RecordingHTTPClient) SetMode(mode string)

Change the recording mode

type TransportOptions

type TransportOptions struct {
	TestName string
	// contains filtered or unexported fields
}