vttest

package
v0.15.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 25, 2022 License: Apache-2.0 Imports: 36 Imported by: 1

Documentation

Overview

Package vttest contains helpers to set up Vitess for testing.

Index

Constants

View Source 
const (
	// DefaultCharset is the default charset used by MySQL instances
	DefaultCharset = "utf8mb4"
)
View Source 
const DefaultMySQLFlavor = "MySQL56"

DefaultMySQLFlavor is the MySQL flavor used by vttest when MYSQL_FLAVOR is not set in the environment

Variables

View Source 
var NewDefaultEnv = defaultEnvFactory

NewDefaultEnv is an user-configurable callback that returns a new Environment instance with the default settings. This callback is only used in cases where the user hasn't explicitly set the Env variable when initializing a LocalCluster

View Source 
var QueryServerArgs = []string{
	"--queryserver-config-pool-size", "4",
	"--queryserver-config-query-timeout", "300",
	"--queryserver-config-schema-reload-time", "60",
	"--queryserver-config-stream-pool-size", "4",
	"--queryserver-config-transaction-cap", "4",
	"--queryserver-config-transaction-timeout", "300",
	"--queryserver-config-txpool-timeout", "300",
}

QueryServerArgs are the default arguments passed to all Vitess query servers

Functions

func GetMySQLOptions 

func GetMySQLOptions(flavor string) (string, []string, error)

GetMySQLOptions returns the default option set for the given MySQL flavor. If flavor is not set, the value from the `MYSQL_FLAVOR` env variable is used, and if this is not set, DefaultMySQLFlavor will be used. Returns the name of the MySQL flavor being used, the set of MySQL CNF files specific to this flavor, and any errors.

func GetShardName 

func GetShardName(shard, total int) string

GetShardName returns an appropriate shard name, as a string. A single shard name is simply 0; otherwise it will attempt to split up 0x100 into multiple shards. For example, in a two sharded keyspace, shard 0 is -80, shard 1 is 80-. This function currently only applies to sharding setups where the shard count is 256 or less, and all shards are equal width.

func GetShardNames 

func GetShardNames(total int) (names []string)

GetShardNames creates a slice of shard names for N shards

func LoadSQLFile 

func LoadSQLFile(filename, sourceroot string) ([]string, error)

LoadSQLFile loads a parses a .sql file from disk, removing all the different comments that mysql/mysqldump inserts in these, and returning each individual SQL statement as its own string. If sourceroot is set, that directory will be used when resolving `source ` statements in the SQL file.

Types

type Config 

type Config struct {
	// Topology defines the fake cluster's topology. This field is mandatory.
	// See: vt/proto/vttest.VTTestTopology
	Topology *vttestpb.VTTestTopology

	// Seed can be set with a SeedConfig struct to enable
	// auto-initialization of the database cluster with random data.
	// If nil, no random initialization will be performed.
	// See: SeedConfig
	Seed *SeedConfig

	// SchemaDir is the directory for schema files. Within this dir,
	// there should be a subdir for each keyspace. Within each keyspace
	// dir, each file is executed as SQL after the database is created on
	// each shard.
	// If the directory contains a `vschema.json“ file, it will be used
	// as the VSchema for the V3 API
	SchemaDir string

	// DefaultSchemaDir is the default directory for initial schema files.
	// If no schema is found in SchemaDir, default to this location.
	DefaultSchemaDir string

	// DataDir is the directory where the data files will be placed.
	// If no directory is specified a random directory will be used
	// under VTDATAROOT.
	DataDir string

	// Charset is the default charset used by MySQL
	Charset string

	// PlannerVersion is the planner version to use for the vtgate.
	// Choose between V3, Gen4, Gen4Greedy and Gen4Fallback
	PlannerVersion string

	// ExtraMyCnf are the extra .CNF files to be added to the MySQL config
	ExtraMyCnf []string

	// OnlyMySQL can be set so only MySQL is initialized as part of the
	// local cluster configuration. The rest of the Vitess components will
	// not be started.
	OnlyMySQL bool

	// PersistentMode can be set so that MySQL data directory is not cleaned up
	// when LocalCluster.TearDown() is called. This is useful for running
	// vttestserver as a database container in local developer environments. Note
	// that db and vschema migration files (-schema_dir option) and seeding of
	// random data (-initialize_with_random_data option) will only run during
	// cluster startup if the data directory does not already exist.
	PersistentMode bool

	// MySQL protocol bind address.
	// vtcombo will bind to this address when exposing the mysql protocol socket
	MySQLBindHost string
	// SnapshotFile is the path to the MySQL Snapshot that will be used to
	// initialize the mysqld instance in the cluster. Note that some environments
	// do not suppport initialization through snapshot files.
	SnapshotFile string

	// Enable system settings to be changed per session at the database connection level
	EnableSystemSettings bool

	// TransactionMode is SINGLE, MULTI or TWOPC
	TransactionMode string

	TransactionTimeout float64

	// The host name to use for the table otherwise it will be resolved from the local hostname
	TabletHostName string

	// Whether to enable/disable workflow manager
	InitWorkflowManager bool

	// Authorize vschema ddl operations to a list of users
	VSchemaDDLAuthorizedUsers string

	// How to handle foreign key constraint in CREATE/ALTER TABLE.  Valid values are "allow", "disallow"
	ForeignKeyMode string

	// Allow users to submit, view, and control Online DDL
	EnableOnlineDDL bool

	// Allow users to submit direct DDL statements
	EnableDirectDDL bool

	// Allow users to start a local cluster using a remote topo server
	ExternalTopoImplementation string

	ExternalTopoGlobalServerAddress string

	ExternalTopoGlobalRoot string
}

Config are the settings used to configure the self-contained Vitess cluster. The LocalCluster struct embeds Config so it's possible to either initialize a LocalCluster with the given settings, or set the settings directly after initialization. All settings must be set before LocalCluster.Setup() is called.

func (*Config) DbName 

func (cfg *Config) DbName() string

DbName returns the default name for a database in this cluster. If OnlyMySQL is set, this will be the name of the single database created in MySQL. Otherwise, this will be blank.

func (*Config) InitSchemas 

func (cfg *Config) InitSchemas(keyspace, schema string, vschema *vschemapb.Keyspace) error

InitSchemas is a shortcut for tests that just want to setup a single keyspace with a single SQL file, and/or a vschema. It creates a temporary directory, and puts the schema/vschema in there. It then sets the right value for cfg.SchemaDir. At the end of the test, the caller should os.RemoveAll(cfg.SchemaDir).

type Environment 

type Environment interface {
	// BinaryPath returns the full path to the given executable
	BinaryPath(bin string) string

	// MySQLManager is the constructor for the MySQL manager that will
	// be used by the cluster. The manager must take care of initializing
	// and destructing the MySQL instance(s) that will be used by the cluster.
	// See: vttest.MySQLManager for the interface the manager must implement
	MySQLManager(mycnf []string, snapshot string) (MySQLManager, error)

	// TopoManager is the constructor for the Topology manager that will
	// be used by the cluster. It's only used when we run the local cluster with
	// a remote topo server instead of in-memory topo server within vtcombo process
	// See: vttest.TopoManager for the interface of topo manager
	TopoManager(topoImplementation, topoServerAddress, topoRoot string, topology *vttest.VTTestTopology) TopoManager

	// Directory is the path where the local cluster will store all its
	// data and metadata. For local testing, this should probably be an
	// unique temporary directory.
	Directory() string

	// LogDirectory is the directory where logs for all services in the
	// cluster will be stored.
	LogDirectory() string

	// VtcomoboArguments are the extra commandline arguments that will be
	// passed to `vtcombo`
	VtcomboArguments() []string

	// ProcessHealthCheck returns a HealthChecker for the given service.
	// The HealthChecker takes an address and attempts to check whether
	// the service is up and healthy.
	// If a given service does not require any custom health checks,
	// nil can be returned.
	ProcessHealthCheck(name string) HealthChecker

	// DefaultProtocol is the protocol used to communicate with the
	// Vitess cluster. This is usually "grpc".
	DefaultProtocol() string

	// PortForProtocol returns the listening port for a given service
	// on the given protocol. If protocol is empty, the default protocol
	// for each service is assumed.
	PortForProtocol(name, protocol string) int

	// EnvVars returns the environment variables that will be passed
	// to all Vitess processes spawned by the local cluster. These variables
	// always take precedence over the variables inherited from the current
	// process.
	EnvVars() []string

	// TearDown is called during LocalCluster.TearDown() to cleanup
	// any temporary data in the environment. Environments that can
	// last through several test runs do not need to implement it.
	TearDown() error
}

Environment is the interface that customizes the global settings for the test cluster. Usually the same environment settings are shared by all the LocalCluster instances in a given test suite, with each instance receiving a different Config for specific tests. For Environments that create temporary data on-disk and clean it up on termination, a brand new instance of Environment should be passed to each LocalCluster.

type FieldGenerator 

type FieldGenerator func(name, t string, rng *rand.Rand) (string, error)

FieldGenerator is a callback that generates the value of a random field in when seeding the database with random data. `name` is the name of the column where the field belongs, `t` is its SQL tyoe, and `rng` is the RNG currently being used, as to ensure reproducible generation between runs. A FieldGenerator must return the raw SQL data for the field, ready to be placed into a SQL statement. The returned value will _NOT_ be escaped.

type HealthChecker 

type HealthChecker func(addr string) bool

HealthChecker is a callback that impements a service-specific health check It must return true if the service at the given `addr` is reachable, false otherwerise.

type LocalCluster 

type LocalCluster struct {
	Config

	// Env is the Environment which will be used for unning this local cluster.
	// It can be set by the user before calling Setup(). If not set, Setup() will
	// use the NewDefaultEnv callback to instantiate an environment with the system
	// default settings
	Env Environment
	// contains filtered or unexported fields
}

LocalCluster controls a local Vitess setup for testing, containing a MySQL instance and one or more vtgate-equivalent access points. To use, simply create a new LocalCluster instance and either pass in the desired Config, or manually set each field on the struct itself. Once the struct is configured, call LocalCluster.Setup() to instantiate the cluster. See: Config for configuration settings on the cluster

func (*LocalCluster) Execute 

func (db *LocalCluster) Execute(sql []string, dbname string) error

Execute runs a series of SQL statements on the MySQL instance backing this local cluster. This is provided for debug/introspection purposes; normal cluster access should be performed through the Vitess GRPC interface.

func (*LocalCluster) GrpcPort 

func (db *LocalCluster) GrpcPort() int

GrpcPort returns the grpc port used by vtcombo

func (*LocalCluster) JSONConfig 

func (db *LocalCluster) JSONConfig() any

JSONConfig returns a key/value object with the configuration settings for the local cluster. It should be serialized with `json.Marshal`

func (*LocalCluster) MySQLAppDebugConnParams 

func (db *LocalCluster) MySQLAppDebugConnParams() mysql.ConnParams

MySQLAppDebugConnParams returns a mysql.ConnParams struct that can be used to connect directly to the mysqld service in the self-contained cluster, using the appdebug user. It's valid only if you used MySQLOnly option.

func (*LocalCluster) MySQLConnParams 

func (db *LocalCluster) MySQLConnParams() mysql.ConnParams

MySQLConnParams returns a mysql.ConnParams struct that can be used to connect directly to the mysqld service in the self-contained cluster This connection should be used for debug/introspection purposes; normal cluster access should be performed through the vtgate port.

func (*LocalCluster) Query 

func (db *LocalCluster) Query(sql, dbname string, limit int) (*sqltypes.Result, error)

Query runs a SQL query on the MySQL instance backing this local cluster and returns its result. This is provided for debug/introspection purposes; normal cluster access should be performed through the Vitess GRPC interface.

func (*LocalCluster) Setup 

func (db *LocalCluster) Setup() error

Setup brings up the self-contained Vitess cluster by spinning up MySQL and Vitess instances. The spawned processes will be running until the TearDown() method is called. Please ensure to `defer db.TearDown()` after calling this method

func (*LocalCluster) TearDown 

func (db *LocalCluster) TearDown() error

TearDown shuts down all the processes in the local cluster and cleans up any temporary on-disk data. If an error is returned, some of the running processes may not have been shut down cleanly and may need manual cleanup.

type LocalTestEnv 

type LocalTestEnv struct {
	BasePort     int
	TmpPath      string
	DefaultMyCnf []string
	Env          []string
}

LocalTestEnv is an Environment implementation for local testing See: NewLocalTestEnv()

func NewLocalTestEnv 

func NewLocalTestEnv(flavor string, basePort int) (*LocalTestEnv, error)

NewLocalTestEnv returns an instance of the default test environment used for local testing Vitess. The defaults are as follows: - Directory() is a random temporary directory in VTDATAROOT, which is cleaned up when closing the Environment. - LogDirectory() is the `logs` subdir inside Directory() - The MySQL flavor is set to `flavor`. If the argument is not set, it will default to the value of MYSQL_FLAVOR, and if this variable is not set, to DefaultMySQLFlavor - PortForProtocol() will return ports based off the given basePort. If basePort is zero, a random port between 10000 and 20000 will be chosen. - DefaultProtocol() is always "grpc" - ProcessHealthCheck() performs no service-specific health checks - BinaryPath() will look up the default Vitess binaries in VTROOT - MySQLManager() will return a vttest.Mysqlctl instance, configured with the given MySQL flavor. This will use the `mysqlctl` command to initialize and teardown a single mysqld instance.

func NewLocalTestEnvWithDirectory 

func NewLocalTestEnvWithDirectory(flavor string, basePort int, directory string) (*LocalTestEnv, error)

NewLocalTestEnvWithDirectory returns a new instance of the default test environment with a directory explicitly specified.

func (*LocalTestEnv) BinaryPath 

func (env *LocalTestEnv) BinaryPath(binary string) string

BinaryPath implements BinaryPath for LocalTestEnv

func (*LocalTestEnv) DefaultProtocol 

func (env *LocalTestEnv) DefaultProtocol() string

DefaultProtocol implements DefaultProtocol for LocalTestEnv. It is always GRPC.

func (*LocalTestEnv) Directory 

func (env *LocalTestEnv) Directory() string

Directory implements Directory for LocalTestEnv.

func (*LocalTestEnv) EnvVars 

func (env *LocalTestEnv) EnvVars() []string

EnvVars implements EnvVars for LocalTestEnv

func (*LocalTestEnv) LogDirectory 

func (env *LocalTestEnv) LogDirectory() string

LogDirectory implements LogDirectory for LocalTestEnv.

func (*LocalTestEnv) MySQLManager 

func (env *LocalTestEnv) MySQLManager(mycnf []string, snapshot string) (MySQLManager, error)

MySQLManager implements MySQLManager for LocalTestEnv

func (*LocalTestEnv) PortForProtocol 

func (env *LocalTestEnv) PortForProtocol(name, proto string) int

PortForProtocol implements PortForProtocol for LocalTestEnv.

func (*LocalTestEnv) ProcessHealthCheck 

func (env *LocalTestEnv) ProcessHealthCheck(name string) HealthChecker

ProcessHealthCheck implements ProcessHealthCheck for LocalTestEnv. By default, it performs no service-specific health checks

func (*LocalTestEnv) TearDown 

func (env *LocalTestEnv) TearDown() error

TearDown implements TearDown for LocalTestEnv

func (*LocalTestEnv) TopoManager added in v0.13.0 

func (env *LocalTestEnv) TopoManager(topoImplementation, topoServerAddress, topoRoot string, topology *vttest.VTTestTopology) TopoManager

TopoManager implements TopoManager for LocalTestEnv

func (*LocalTestEnv) VtcomboArguments 

func (env *LocalTestEnv) VtcomboArguments() []string

VtcomboArguments implements VtcomboArguments for LocalTestEnv.

type MySQLManager 

type MySQLManager interface {
	Setup() error
	Start() error
	TearDown() error
	Auth() (string, string)
	Address() (string, int)
	UnixSocket() string
	TabletDir() string
	Params(dbname string) mysql.ConnParams
}

MySQLManager is an interface to a mysqld process manager, capable of starting/shutting down mysqld services and initializing them.

type Mysqlctl 

type Mysqlctl struct {
	Binary    string
	InitFile  string
	Directory string
	Port      int
	MyCnf     []string
	Env       []string
	UID       uint32
}

Mysqlctl implements MySQLManager through Vitess' mysqlctld tool

func (*Mysqlctl) Address 

func (ctl *Mysqlctl) Address() (string, int)

Address returns the hostname/tcp port pair required to connect to mysqld

func (*Mysqlctl) Auth 

func (ctl *Mysqlctl) Auth() (string, string)

Auth returns the username/password tuple required to log in to mysqld

func (*Mysqlctl) Params 

func (ctl *Mysqlctl) Params(dbname string) mysql.ConnParams

Params returns the mysql.ConnParams required to connect directly to mysqld using Vitess' mysql client.

func (*Mysqlctl) Setup 

func (ctl *Mysqlctl) Setup() error

Setup spawns a new mysqld service and initializes it with the defaults. The service is kept running in the background until TearDown() is called.

func (*Mysqlctl) Start added in v0.10.0 

func (ctl *Mysqlctl) Start() error

Start spawns a mysqld service for an existing data directory The service is kept running in the background until TearDown() is called.

func (*Mysqlctl) TabletDir added in v0.10.0 

func (ctl *Mysqlctl) TabletDir() string

TabletDir returns the path where data for this Tablet would be stored

func (*Mysqlctl) TearDown 

func (ctl *Mysqlctl) TearDown() error

TearDown shutdowns the running mysqld service

func (*Mysqlctl) UnixSocket 

func (ctl *Mysqlctl) UnixSocket() string

UnixSocket returns the path to the local Unix socket required to connect to mysqld

type SeedConfig 

type SeedConfig struct {
	// RngSeed is the seed uset to initialize the random number
	// generator that will be used to fill the database with
	// random data. Multiple runs with the same seed will result
	// in the same initial data
	RngSeed int

	// MinSize is the minimum number of initial rows in each tale shard
	MinSize int

	// MaxSize is the maximum number of initial rows in each table shard
	MaxSize int

	// NullProbability is the chance to initialize a field a NULL value.
	// Only applies to fields that can contain NULL values
	NullProbability float64

	// RandomField is a callback to generate the value of a random field
	RandomField FieldGenerator
}

SeedConfig are the settings to enable the initialization of the local cluster with random data. This struct must be set in Config before Setup() is called.

func SeedConfigDefaults 

func SeedConfigDefaults() *SeedConfig

SeedConfigDefaults returns the default values for SeedConfig

type TopoData added in v0.14.0 

type TopoData struct {
	// contains filtered or unexported fields
}

TopoData is a struct representing a test topology.

It implements pflag.Value and can be used as a destination command-line via pflag.Var or pflag.VarP.

func JSONTopoData added in v0.14.0 

func JSONTopoData(tpb *vttestpb.VTTestTopology) *TopoData

JSONTopoData returns a test TopoData that unmarshals using protojson.Unmarshal.

func TextTopoData added in v0.14.0 

func TextTopoData(tpb *vttestpb.VTTestTopology) *TopoData

TextTopoData returns a test TopoData that unmarshals using prototext.Unmarshal.

func (*TopoData) Set added in v0.14.0 

func (td *TopoData) Set(value string) error

Set is part of the pflag.Value interface.

func (*TopoData) String added in v0.14.0 

func (td *TopoData) String() string

String is part of the pflag.Value interface.

func (*TopoData) Type added in v0.15.0 

func (td *TopoData) Type() string

Type is part of the pflag.Value interface.

type TopoManager added in v0.13.0 

type TopoManager interface {
	Setup() error
}

TopoManager is an interface to manage things in a remote topology server for local cluster

type Topoctl added in v0.13.0 

type Topoctl struct {
	TopoImplementation      string
	TopoGlobalServerAddress string
	TopoGlobalRoot          string
	Topology                *vttest.VTTestTopology
}

Topoctl implements TopoManager. For now it only sets up cells in remote topology server if any of them doesn't exist.

func (*Topoctl) Setup added in v0.13.0 

func (ctl *Topoctl) Setup() error

type VtProcess 

type VtProcess struct {
	Name         string
	Directory    string
	LogDirectory string
	Binary       string
	ExtraArgs    []string
	Env          []string
	Port         int
	PortGrpc     int
	HealthCheck  HealthChecker
	// contains filtered or unexported fields
}

VtProcess is a generic handle for a running Vitess process. It can be spawned manually or through one of the available helper methods.

func VtcomboProcess 

func VtcomboProcess(environment Environment, args *Config, mysql MySQLManager) (*VtProcess, error)

VtcomboProcess returns a VtProcess handle for a local `vtcombo` service, configured with the given Config. The process must be manually started by calling WaitStart()

func (*VtProcess) Address 

func (vtp *VtProcess) Address() string

Address returns the main address for this Vitess process. This is usually the main HTTP endpoint for the service.

func (*VtProcess) IsHealthy 

func (vtp *VtProcess) IsHealthy() bool

IsHealthy returns whether the monitored Vitess process has started successfully.

func (*VtProcess) WaitStart 

func (vtp *VtProcess) WaitStart() (err error)

WaitStart spawns this Vitess process and waits for it to be up and running. The process is considered "up" when it starts serving its debug HTTP endpoint -- this means the process was successfully started. If the process is not healthy after 60s, this method will timeout and return an error.

func (*VtProcess) WaitTerminate 

func (vtp *VtProcess) WaitTerminate() error

WaitTerminate attempts to gracefully shutdown the Vitess process by sending a SIGTERM, then wait for up to 10s for it to exit. If the process hasn't exited cleanly after 10s, a SIGKILL is forced and the corresponding exit error is returned to the user

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.