perf

Documentation

Overview

Package perf is a helper library for writing Mixer perf tests. It is designed to have a serializable, declarative configuration, that can be run in various execution modes transparently. The package is designed to work seamlessly with the Go perf testing infrastructure.

The entry point to the tests is the "Run*" methods. The benchmarks are expected to call one of the methods, passing in testing.B, the test declaration (i.e. Setup struct), an environment variable that contains the ambient template, adapter info (i.e. Env struct), along with any other Run-method specific inputs.

The top-level struct for test declaration is the Setup struct. This contains two major fields, namely Config and Load. The Config field contains the full configuration needed for a Mixer Server, including global and service config Yaml files, as well as the adapters and templates that should be incorporated from the ambient context. The Load section is a customizable declaration of the type of the load that should be applied to Mixer during test. There is a standard set of configs/setups available in this package as well, to simplify test authoring.

Currently, the tests can be run in two modes: inprocess, or coprocess. Inprocess creates a client within the same process as the test, whereas coprocess creates the client in a separate, external process. Creating the client in an external process isolates the server code as the only executing code within the benchmark, thus enables more accurate measurement of various perf outputs (i.e. memory usage, cpu usage etc.)

Execution-wise, the test is orchestrated by the Controller struct, which internally uses testing.B. When the Run* entry method is called, it first creates a Controller, which establishes an Rpc server for registration of agents (i.e. clients). Then, the run method would either launch the external exe hosting a ClientServer, or it will simply start one locally. Once the ClientServer starts, it registers itself with the Controller. Once the registration is done, the controller initializes the client(s) by uploading Setup for the test and giving the address of the Mixer server. Then, controller commands the client(s) to execute the load, multiplied by an iteration factor which is obtained from testing.B. This can happen multiple times. Finally, the controller signals the clients to gracefully close and ends the benchmark.

Index

• Variables
• func Run(b *testing.B, setup *Setup, settings Settings)
• type BasicCheck
  • func (c BasicCheck) MarshalJSON() ([]byte, error)
• type BasicReport
  • func (r BasicReport) MarshalJSON() ([]byte, error)
• type ClientServer
  • func NewClientServer(controllerLoc ServiceLocation) (*ClientServer, error)
  • func (s *ClientServer) InitializeClient(params ClientServerInitParams, _ *struct{}) error
  • func (s *ClientServer) Run(iterations int, _ *struct{}) error
  • func (s *ClientServer) Shutdown(struct{}, *struct{}) error
  • func (s *ClientServer) Wait()
• type ClientServerInitParams
• type Config
• type Controller
  • func (c *Controller) RegisterClient(loc ServiceLocation, _ *struct{}) error
• type Load
  • func (l *Load) MarshalJSON() ([]byte, error)
  • func (l *Load) UnmarshalJSON(bytes []byte) error
• type Request
• type RunMode
• type ServiceLocation
  • func (s ServiceLocation) String() string
• type Settings
• type Setup

Variables

var MinimalConfig = Config{
	Service: minimalSvcCfg,
	Global:  minimalGlobalCfg,
}

MinimalConfig is a very basic configuration, mainly useful for testing the perf infrastructure itself,

var MinimalSetup = Setup{
	Config: MinimalConfig,
	Loads: []Load{{
		Multiplier:  100,
		StableOrder: false,
		Requests: []Request{

			BasicReport{
				Attributes: map[string]interface{}{
					"foo": "bar",
					"baz": 42,
				},
			},

			BasicCheck{
				Attributes: map[string]interface{}{
					"bar": "baz",
					"foo": 23,
				},

				Quotas: map[string]istio_mixer_v1.CheckRequest_QuotaParams{
					"q1": {
						Amount:     23,
						BestEffort: true,
					},
					"q2": {
						Amount:     54,
						BestEffort: false,
					},
				},
			},
		},
	}},
}

MinimalSetup is a very basic setup, mainly useful for testing the perf infrastructure itself.

Functions

func Run

func Run(b *testing.B, setup *Setup, settings Settings)

Run executes the benchmark using specified setup and settings.

Types

type BasicCheck

type BasicCheck struct {
	Attributes map[string]interface{}                             `json:"attributes,omitempty"`
	Quotas     map[string]istio_mixer_v1.CheckRequest_QuotaParams `json:"quotas,omitempty"`
}

BasicCheck is an implementation of Request that is specified declaratively by the author.

func (BasicCheck) MarshalJSON

func (c BasicCheck) MarshalJSON() ([]byte, error)

MarshalJSON marshals the report as JSON.

type BasicReport

type BasicReport struct {
	Attributes map[string]interface{} `json:"attributes,omitempty"`
}

BasicReport is an implementation of Request that is used to explicitly specify a Report request.

func (BasicReport) MarshalJSON

func (r BasicReport) MarshalJSON() ([]byte, error)

MarshalJSON marshals the report as JSON.

type ClientServer

type ClientServer struct {
	// contains filtered or unexported fields
}

ClientServer is an RPC server that the Controller connects to remotely control a Mixer perf test client.

func NewClientServer

func NewClientServer(controllerLoc ServiceLocation) (*ClientServer, error)

NewClientServer creates a new ClientServer and returns it. Before doing so, it connects to the controller and registers itself with it.

func (*ClientServer) InitializeClient

func (s *ClientServer) InitializeClient(params ClientServerInitParams, _ *struct{}) error

InitializeClient is a remote RPC call that is invoked by the controller to initiate setup of the client environment. The Mixer client connects to the server at the given address, and keeps the setup metadata to generate load during upcoming run requests.

func (*ClientServer) Run

func (s *ClientServer) Run(iterations int, _ *struct{}) error

Run is a remote RPC call that is invoked by the controller to request the mixer to run the load for the specified number of iterations.

func (*ClientServer) Shutdown

func (s *ClientServer) Shutdown(struct{}, *struct{}) error

Shutdown is a remote RPC call that is invoked by the controller after the benchmark execution has completed.

func (*ClientServer) Wait

func (s *ClientServer) Wait()

Wait blocks until the server is instructed to exit. This should be called only once.

type ClientServerInitParams

type ClientServerInitParams struct {

	// Setup is the YAML-serialized load object.
	Load []byte

	// Address of the Mixer Server.
	Address string
}

ClientServerInitParams is a collection of parameters that are passed as part of the InitializeClient call.

type Config

type Config struct {
	Global         string `json:"global"`
	Service        string `json:"rpcServer"`
	EnableLog      bool   `json:"enableLog,omitempty"`
	EnableDebugLog bool   `json:"enableDebugLog,omitempty"`
	SingleThreaded bool   `json:"singleThreaded,omitempty"`

	// Templates is the name of the templates to use in this test. If left empty, a standard set of templates
	// will be used.
	Templates []string `json:"templates,omitempty"`

	// Adapters is the name of the adapters to use for this test. If left empty, a standard set of adapters
	// will be used.
	Adapters []string `json:"adapters,omitempty"`
}

Config is the Mixer server configuration to use during perf tests. TODO: We should ideally combine this file with pkg/server/Args. Unfortunately, pkg/serverArgs is not serializable.

type Controller

type Controller struct {
	// contains filtered or unexported fields
}

Controller is the top-level perf benchmark controller. It drives the test by managing the client(s) that generate load against a Mixer instance.

func (*Controller) RegisterClient

func (c *Controller) RegisterClient(loc ServiceLocation, _ *struct{}) error

RegisterClient is an RPC method called by the clients to registers with this controller.

type Load

type Load struct {

	// Requests is the set of requests to use.
	// TODO: These should all be collapsed to "Requests",
	Requests []Request `json:"requests,omitempty"`

	// Multiplier is the number of times to apply the specified load on target. If not specified, it will default to 1.
	Multiplier int `json:"multiplier,omitempty"`

	// StableOrder indicates that the requests will be executed in a stable order. If not set to true, then the
	// requests will be randomized before being sent over the wire.
	// This is here mostly for debugging.
	StableOrder bool `json:"stableOrder,omitempty"`

	// RandomSeed is the random seed to use when randomizing load. If omitted (i.e. when set to 0), a time-based seed
	// will be used.
	// This is here mostly for debugging.
	RandomSeed int64 `json:"randomSeed,omitempty"`
}

Load is the load to apply on the Mixer during the perf test. TODO: This should become an interface, and we should be able to specify different load generation strategies.

func (*Load) MarshalJSON

func (l *Load) MarshalJSON() ([]byte, error)

MarshalJSON marshals the load as JSON.

func (*Load) UnmarshalJSON

func (l *Load) UnmarshalJSON(bytes []byte) error

UnmarshalJSON unmarshals the load from JSON.

type Request

type Request interface {
	// contains filtered or unexported methods
}

Request interface is the common interface for all different types of requests.

type RunMode

type RunMode int

RunMode configures the run mode for the perf.

const (
	// InProcess run mode indicates that an in-process client should be used to drive the test.
	// This potentially affects collected profile data, as it will include client's execution as well.
	InProcess RunMode = iota

	// CoProcess run mode indicates that the client should be run in a separate process.
	// This avoids client's execution polluting the profile data, but may cause variations in timings
	// as the communication overhead between the client and server is counted in timings.
	CoProcess

	// InProcessBypassGrpc run mode indicates that the test should be run against the runtime.Dispatcher interface
	// directly. This is useful to reduce the scope that needs to be profiled, and allows discounting gRpc
	// and attribute preprocessing related overhead.
	InProcessBypassGrpc
)

type ServiceLocation

type ServiceLocation struct {
	// Address is the network address of the service.
	Address string

	// Path is the HTTP path of the RPC service.
	Path string
}

ServiceLocation is a struct that combines the address and the path of an rpc server.

func (ServiceLocation) String

func (s ServiceLocation) String() string

type Settings

type Settings struct {
	RunMode   RunMode
	Templates map[string]template.Info
	Adapters  []adapter.InfoFn

	// ExecutableSearchSuffix indicates the search suffix to use when trying to locate the co-process
	// executable for perf testing. The process is located through an iterative search, starting with the
	// current working directory, and using the executablePathSuffix as the
	// search suffix for executable (e.g. "bazel-bin/mixer/test/perf/perfclient/perfclient").
	ExecutablePathSuffix string
}

Settings is the perf test settings.

type Setup

type Setup struct {
	// Config is the Mixer config to use for this test.
	Config Config `json:"config"`

	// Loads is an array of Load passed from different clients in parallel.
	Loads []Load `json:"loads"`
}

Setup is the self-contained, top-level definition of a perf test. This structure is meant to be fully-serializable to support intra-process communication. As such, it should only have serializable members.