sandboxes

README

Sandboxes for remote Salvo executions

This directory contains definition of Salvo sandboxes. A sandbox is the collection of resources needed for a single Salvo execution. These resources are deployed in AWS. At the minimum a sandbox will contain:

• a VM running Nighthawk (the load generator).
• a VM running Envoy (the system under test or SUT).
• a VM running the Nighthawk test server (the test backend).

Note that the subnets used between the components are part of the common infrastructure and are already deployed as part of the VPC configuration done in the ci-infra repository.

Various sandbox types differ in the amount of instances deployed, amount of support systems deployed and the topology.

Available Sandbox types

The default sandbox

The default sandbox contains the bare minimum of components needed to execute Salvo tests. The diagram below outlines the topology and content of the default sandbox. The default sandbox is currently available for the x64 architecture only.

Sandbox creation

When Salvo executes, an execution of the AZP CI pipeline is started. Each CI pipeline execution is uniquely identified by a build ID. This execution will produce binaries and VM disk images for all the sandbox components.

Once the binaries and VM disk images are built, the AZP CI pipeline starts a job in the salvo-control agent pool on AZP. This job is then picked up by a Salvo control VM running a Salvo controller that instruments the sandbox creation.

The Salvo controller uses the Terraform templates found in this directory, to deploy the sandbox components in AWS.

Sandbox instance life-cycle

The Terraform configuration for sandboxes uses shared state bucket deployed in S3 on AWS. The sandbox instances that will be deployed or destroyed are determined based on the variables passed to Terraform.

When a sandbox is starting up, it needs to locate the binaries and VM disk images that were produced by the AZP CI pipeline. This is achieved by providing the sandbox with the build ID that uniquely identifies the AZP CI pipeline execution that produced them.

Each of the following variables is a list of these build IDs. Each list represents one sandbox type. Each listed build ID represents an instance of the sandbox type that should be deployed.

Supported sandbox types:

• default_sandbox_x64_build_ids.

For example the Terraform command to deploy a single instance of the default sandbox with build ID 136112 is:

terraform apply --var="default_sandbox_build_ids=[\"136112\"]"

Any instances that were deployed, but are not named in the variables passed to Terraform will be destroyed when Terraform is executed. To remove all deployed sandboxes, simply execute terraform apply in this directory with no arguments.

terraform apply

Documentation

Overview

instances.go defined data types that described deployed sandbox instances.

json.go contains functions that unmarshal JSON messages received from Terraform.

Package sandboxes manages the life cycle of a Salvo remote sandbox.

Index

• type Instance
  • func (i *Instance) String() string
• type Instances
• type Manager
  • func NewManager(opts ...Option) *Manager
  • func (m *Manager) Start(ctx context.Context, sbxs map[Type]Instances) (map[int64]*Instance, error)
  • func (m *Manager) StopAll(ctx context.Context) error
• type Option
• type Type
  • func (t Type) String() string
• type VM

Types

type Instance

type Instance struct {
	// Type identifies the type of the sandbox instance.
	Type Type

	// LoadGenerators are the load generator VMs deployed in the sandbox.
	LoadGenerators []*VM

	// SUTs are the system under test VMs deployed in the sandbox.
	SUTs []*VM

	// Backends are the backend VMs deployed in the sandbox.
	Backends []*VM
}

Instance describes a single sandbox instance.

func (*Instance) String

func (i *Instance) String() string

String implements fmt.Stringer.

type Instances

type Instances []int64

Instances represent sandbox instances, each integer is an AZP Build ID corresponding to AZP pipeline execution that built the components for the sandbox.

type Manager

type Manager struct {
	// contains filtered or unexported fields
}

Manager manages sandbox instances.

func NewManager

func NewManager(opts ...Option) *Manager

NewManager creates a new Sandbox manager.

func (*Manager) Start

func (m *Manager) Start(ctx context.Context, sbxs map[Type]Instances) (map[int64]*Instance, error)

Start starts the specified sandbox instances. Returns a map of sandbox build IDs to information about the sandbox instance.

func (*Manager) StopAll

func (m *Manager) StopAll(ctx context.Context) error

StopAll stops all sandbox instances.

type Option

type Option interface {
	// contains filtered or unexported methods
}

Option is used to provide options to NewManager.

type Type

type Type int

Type enumerates the types of sandboxes supported.

const (

	// TypeDefaultSandboxX64 is the default sandbox, see https://github.com/envoyproxy/envoy-perf/tree/main/salvo-remote/sandboxes/terraform/default_sandbox_x64.
	TypeDefaultSandboxX64 Type
)

func (Type) String

func (t Type) String() string

String implements fmt.Stringer()

type VM

type VM struct {
	// IP is the primary private IP address of the VM.
	// This is the IP address on the interface that either originates or receives load.
	IP net.IP

	// ControlIP is a private IP address that can be used by the control VM when accessing the VM.
	// Note, the control VM is the VM that runs salvo-remote (this code).
	ControlIP net.IP
}

VM describes a VM deployed in a sandbox.