Perf Test Agent
Table of contents generated with markdown-toc
Description
This tool can be used to compare timing data between arbitrary number of similar
services to evaluate differences in speed between a specified baseline backend
and the other backends.
Specifically, this tool is used here as a performance test agent deployed
alongside Secretless to compare the following scenarios:
Backend |
Baseline |
Compare With |
Connection Type |
MSSQL |
Direct connection |
Secretless (persistent connection) |
TCP port |
MySQL |
Direct connection |
Secretless (persistent connection) |
Unix socket |
MySQL |
Direct connection |
Secretless (persistent connection) |
TCP port |
Postgres |
Direct connection |
Secretless (persistent connection) |
Unix socket |
Postgres |
Direct connection |
Secretless (persistent connection) |
TCP port |
MSSQL |
Direct connection |
Secretless (recreated connection) |
TCP port |
MySQL |
Direct connection |
Secretless (recreated connection) |
Unix socket |
MySQL |
Direct connection |
Secretless (recreated connection) |
TCP port |
Postgres |
Direct connection |
Secretless (recreated connection) |
Unix socket |
Postgres |
Direct connection |
Secretless (recreated connection) |
TCP port |
All scenarios can be run with limits as either minimum number of rounds completed or a
time-limited duration.
We compare the following results:
- Returned values versus expected values
- Number of rounds (single-shot tests runs) completed
- Average/Min/Max single-shot test duration
- Error count, error messages, and percentage of errors
- Differences in timing (percentage-based) between runs temporaly close
- Percentage of single-shot runs that are above the specified baseline threshold
- 90% confidence interval (percentage-based) of test runs as compared to the
baseline backend.
Note: More comparison types may be added in the future.
Status: Alpha
Warning: Naming and APIs are still subject to breaking changes!
Running the code locally
You can run and test this code on your machine easily with a few things:
- Docker
- Golang 1.12 or higher
- Secretless-broker (optional)
- A writable folder for shared socket files (e.g.
/tmp
in these configurations) if you're using
socket files (optional)
Start the test databases
PG
docker run --name pg-test-db \
-p 5433:5432 \
-e POSTGRES_USER=myuser \
-e POSTGRES_PASSWORD=mypassword \
-e POSTGRES_DB=mydb \
-d \
postgres
MySQL
docker run --name mysql-test-db \
-p 3307:3306 \
-e MYSQL_ROOT_PASSWORD=myrootpassword \
-e MYSQL_USER=myuser \
-e MYSQL_PASSWORD=mypassword \
-e MYSQL_DATABASE=mydb \
-d \
mysql:5
MSSQL
docker run --name mssql-test-db \
-p 1434:1433 \
-e ACCEPT_EULA=Y \
-e SA_PASSWORD="MYp4ssword1" \
-d \
mcr.microsoft.com/mssql/server:2017-latest
Run Secretless Broker
Note: This step is optional but it is required for this specific example since it tests the broker's
performance.
- Clone the secretless-broker repository (
git clone https://github.com/cyberark/secretless-broker
).
- Create the following
secretless.yml
file in that folder:
secretless.yml
version: 2
services:
mysql-socket:
connector: mysql
listenOn: unix:///tmp/mysql
credentials:
username: myuser
password: mypassword
host: 127.0.0.1
port: 3307
sslmode: disable
pg-socket:
connector: pg
listenOn: unix:///tmp/.s.PGSQL.5432
credentials:
username: myuser
password: mypassword
host: 127.0.0.1
port: 5433
sslmode: disable
mssql-tcp:
connector: mssql
listenOn: tcp://0.0.0.0:1433
credentials:
username: sa
password: MYp4ssword1
host: 127.0.0.1
port: 1434
Finally, start Secretless Broker with:
go run cmd/secretless-broker/main.go -f secretless.yml
Run Juxtaposer
- Open a new terminal while leaving the Secretless Broker to run in the background.
- Go into the secretless-broker's
bin/juxtaposer
folder (cd bin/juxtaposer
).
- Create the following
juxtaposer.yml
file in that folder and make any relevant changes to it depening on what
database backend you want to test:
juxtaposer.yml
---
#driver: mssql
#driver: mysql-5.7
driver: postgres
comparison:
baselineBackend: pg_direct
# baselineBackend: mysql_direct
# baselineBackend: mssql_direct
# recreateConnections: true
# sqlStatementType: select
# rounds: 1000
# rounds: infinity
# baselineMaxThresholdPercent: 120
threads: 5
silent: false
formatters:
json:
outputFile: ./results.json
stdout:
backends:
pg_secretless:
host: /tmp
pg_direct:
host: localhost
port: 5433
username: myuser
password: mypassword
sslmode: disable
debug: false
mysql_secretless:
host: /tmp/mysql
ignore: true
mysql_direct:
host: localhost
port: 3307
username: myuser
password: mypassword
sslmode: disable
debug: false
ignore: true
mssql_secretless:
host: 127.0.0.1
ignore: true
mssql_direct:
host: 127.0.0.1
port: 1434
username: sa
password: MYp4ssword1
ignore: true
Note: Be sure to modify the value for ignore
for each backend
you wish to run against.
Finally, start Juxtaposer with:
go run main.go -f juxtaposer.yml
Running the code on OpenShift
Running the code on an OpenShift cluster is done through scripts in the deploy/
folder.
Ensure that you have:
- Databases that you are using for your tests already deployed
- Credentials for those databases
- Docker
oc
CLI tool
- Configured
oc
/kubectl
context
- A configured namespace (
project
in OpenShift terminology)
Juxtaposer is deployed on kubernetes as part of secretless xa tests. The following
diagram shows the architecture used:
Main configuration of the deployment scripts is done through the ./deploy/bootstrap.sh
environment variables. Make sure to change these according to your deployment and
source the file with:
source ./deploy/bootstrap.sh
Depending on the backend driver and database details, configure the appropriate
./deploy/juxtaposer_${CONFIG_TEMPLATE}.yml
file and add/replace any backend
details that are specific to your deployment. This usually includes changes to
majority of the backends
section of the file but depending on what you might
be testing, it could include other fields too.
With the configuration files ready, the last part is configuring the deployment
command for Juxtaposer in ./deploy/juxtaposer_deployment_template.yml
file. This will be influenced heavily by what type of testing you are doing on
the codebase and may be changed depending on your needs.
- If you are running round-based tests, uncomment the following line of Juxtaposer container:
...
args: ["-c", "-f", "/etc/${APP_NAME}/${APP_NAME}_${CONFIG_TEMPLATE}.yml"]
...
- If you are running time-based tests, uncomment the following line of Juxtaposer container:
...
args: ["-c", "-t", "${TEST_DURATION}", "-f", "/etc/${APP_NAME}/${APP_NAME}_${CONFIG_TEMPLATE}.yml"
...
- If you will be performing output parsing manually, uncomment the following line of Juxtaposer container:
...
command: ["sh", "-c", "juxtaposer -t ${TEST_DURATION} -f /etc/${APP_NAME}/${APP_NAME}_${CONFIG_TEMPLATE}.yml &> /tmp/output.txt && echo 'done' && ls -la /tmp/output.txt && sleep 999d"]
...
- If you are developing Juxtaposer code, uncomment the following line of Juxtaposer container:
...
command: [ "/bin/sleep", "999d" ]
...
Deploy the code to OpenShift
With all of the files fixed up and bootstrap.sh
sourced, you can then deploy
things to Openshift with:
./deploy/start
Results will be present either in Docker logs or on the filesystem depending on the
start command and formatter configuration. Containers will remain running until
they are manually torn down unless you remove the -c
flag from the startup command.
Cleaning up
To remove all deployments and clean up resources run:
./deploy/stop
CLI flags
-c
: Continue running after end of tests
Leaves the process running after the tests are complete and the results are
printed. This protects the data from container log reaping.
-f
: Path to configuration file
Overrides the default configuration file path, which is ./juxtaposer.yml
.
-t
: Run tests for this duration
Run tests for a specified time rather than a number of loops. Uses Golang's
ParseDuration format to specify
time (e.g. 10h5m3s
).
Configuration
Configuration is specified either with a default juxtaposer.yml
in the current
directory or overridden by -f
CLI flag.
A configuration file looks like this:
driver: <DRIVER_NAME>
comparison:
baselineBackend: backend1
formatters:
stdout:
backends:
backend1:
host: <HOST>
port: <PORT>
username: <USERNAME>
password: <USERNAME>
sslmode: disable
backend2:
host: <PATH_TO_SOCKET>
Supported Drivers:
comparison
options
-
baselineBackend
, string
This setting is linked to a named backend to indicate the baseline backend
against which all calculation will be compared.
-
(optional) recreateConnections
, bool
, default: false
This setting decides if the connections during testing will be recreated
on each test (true
) or will the initial connection be opened only once
and then subsequent tests run on that opened connection (false
).
-
(optional) sqlStatementType
, string
, default: select
This setting decides what SQL style of comparison will be run. Currently
only select
is supported.
-
(optional) rounds
, int
, default: 1000
This setting decides how many loops the main test run will iterate over all
the defined backends. This setting is ignored if time-based CLI flag is used.
This field also supports a special keyword infinity
that lets the tests run
forever or until the user sends an interrupt signal. Note that this indicates
only the minimum amount of runs that all backends will be iterated over.
-
(optional) threads
, int
, default: 1
This setting selects how many parallel routines (threads) will be used for each
backend testing. Increasing this setting will mean that there will be a proportional
load testing increase on the service per unit of time.
-
(optional) baselineMaxThresholdPercent
, int
, default: 120
This setting decides how slow (percentage-wise) can a non-baseline backend be
before it is counted as "exceeding threshold".
-
(optional) silent
, bool
, default: false
When this setting is turned on, only minimal test run messages are displayed during
runs.
This key-value map indicates what formatters are run at the end of tests to handle
aggregated results. Currently only json
and stdout
are supported. json
formatter
currently also supports outputFile
options which can be used to send the output data
from it to a file instead of stdout.
backends
options
This key-value map lists all backends that should be tested as part of the agent's
running.
Options are as follows:
host: Hostname (or socket address) that the driver will use. Note that for PG driver and sockets, this is the *folder* instead of the path to the socket file.
port: Port to use when connecting to the backend
username: Username to use for connecting to the backend. Some drivers do not need this setting.
password: Password to use for connecting to the backend. Some drivers do not need this setting.
sslmode: SSL mode to use for connecting to the backend. Some drivers do not need this setting.
debug: Print out verbose information about this backend test runs
ignore: If set to true, the agent will ignore this backend