README
¶
Local Vitess cluster using docker-compose
This directory has a docker-compose sample application. To understand it better, you can run it.
First you will need to install docker-compose.
Before you begin
You will need to create a docker-compose.yml file. There are 2 ways to do this.
- Run
go run vtcompose/vtcompose.go --args. Instructions can be found here - Use the
docker-compose.beginners.ymlto generate yourdocker-compose.ymlfile. Run:
vitess/examples/compose$ cp docker-compose.beginners.yml docker-compose.yml
Create your .env file
vitess/examples/compose$ cp template.env .env
You can then proceed to the instructions under Start the Cluster section.
Programatically create Vitess configuration for Docker
To create a configuration to your specifications, run vtcompose. Creates corresponding docker-compose file, vschema files per keyspace, and loads schemas.
vitess/examples/compose$ go run vtcompose/vtcompose.go --args(optional)
Use -h or --help to get list of flags with descriptions.
Flags available:
- baseDockerComposeFile - Specifies starting docker-compose yaml file.
- baseVschemaFile - Specifies starting vschema json file.
- topologyFlags - Specifies Vitess topology flags config
- webPort - Specifies web port to be used.
- gRpcPort - Specifies gRPC port to be used.
- mySqlPort - Specifies mySql port to be used.
- cell - `Specifies Vitess cell name to be used.
- keyspaceData - List of
keyspace_name:num_of_shards:num_of_replica_tablets:schema_file_names:<optional>lookup_keyspace_nameseparated by ';'.- This is where you specify most of the data for the program to build your vSchema and docker-compose files.
- Examples you can run;
- Default
go run vtcompose/vtcompose.go- Use
0fornum_of_shardsto specify an unsharded keyspace
go run vtcompose/vtcompose.go -keyspaceData="test_keyspace:0:2:create_messages.sql"- Multiple keyspaces with sharded test_keyspace
go run vtcompose/vtcompose.go -keyspaceData="test_keyspace:2:1:create_messages.sql,create_tokens.sql:lookup_keyspace;lookup_keyspace:1:1:create_tokens_token_lookup.sql,create_messages_message_lookup.sql" - externalDbData - Specifies which databases/keyspaces are external and provides data along with it to connect to the external db.
List of
<external_db_name>,<DB_HOST>,<DB_PORT>,<DB_USER>,<DB_PASS>,<DB_CHARSET>separated by ';'.
When using this, make sure to have the external_db_name/keyspace in thekeyspaceDataflag with no schema_file_names specified.go run vtcompose/vtcompose.go -keyspaceData="commerce:0:2::" -externalDbData="commerce:external_db_host:3306:external_db_user:external_db_password:"
Start the cluster
To start Consul(which saves the topology config), vtctld, vtgate, vtorc and a few vttablets with MySQL running on them.
vitess/examples/compose$ docker-compose up -d
Check cluster status
Check the status of the cluster.
vitess/examples/compose$ docker-compose ps
Name Command State Ports
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
compose_consul1_1 docker-entrypoint.sh agent ... Up 8300/tcp, 8301/tcp, 8301/udp, 8302/tcp, 8302/udp,
0.0.0.0:8400->8400/tcp,:::8400->8400/tcp, 0.0.0.0:8500->8500/tcp,:::8500->8500/tcp,
0.0.0.0:8600->8600/tcp,:::8600->8600/tcp, 8600/udp
compose_consul2_1 docker-entrypoint.sh agent ... Up 8300/tcp, 8301/tcp, 8301/udp, 8302/tcp, 8302/udp, 8400/tcp, 8500/tcp, 8600/tcp,
8600/udp
compose_consul3_1 docker-entrypoint.sh agent ... Up 8300/tcp, 8301/tcp, 8301/udp, 8302/tcp, 8302/udp, 8400/tcp, 8500/tcp, 8600/tcp,
8600/udp
compose_external_db_host_1 docker-entrypoint.sh --ser ... Up (healthy) 0.0.0.0:59816->3306/tcp, 33060/tcp
compose_schemaload_1 sh -c /script/schemaload.sh Exit 0
compose_set_keyspace_durability_policy_1 sh -c /script/set_keyspace ... Exit 0
compose_vreplication_1 sh -c [ $EXTERNAL_DB -eq 1 ... Exit 0
compose_vtctld_1 sh -c /vt/bin/vtctld -top ... Up 0.0.0.0:59823->15999/tcp, 0.0.0.0:15000->8080/tcp,:::15000->8080/tcp
compose_vtgate_1 sh -c /vt/bin/vtgate -topo ... Up 0.0.0.0:15306->15306/tcp,:::15306->15306/tcp, 0.0.0.0:59830->15999/tcp,
0.0.0.0:15099->8080/tcp,:::15099->8080/tcp
compose_vtorc_1 sh -c /script/vtorc-up.sh Up (healthy) 0.0.0.0:13000->3000/tcp,:::13000->3000/tcp
compose_vttablet100_1 sh -c [ $EXTERNAL_DB -eq 1 ... Exit 0
compose_vttablet101_1 sh -c /script/vttablet-up. ... Up (healthy) 0.0.0.0:59824->15999/tcp, 0.0.0.0:59825->3306/tcp,
0.0.0.0:15101->8080/tcp,:::15101->8080/tcp
compose_vttablet102_1 sh -c /script/vttablet-up. ... Up (healthy) 0.0.0.0:59826->15999/tcp, 0.0.0.0:59827->3306/tcp,
0.0.0.0:15102->8080/tcp,:::15102->8080/tcp
compose_vttablet103_1 sh -c /script/vttablet-up. ... Up (healthy) 0.0.0.0:65284->15999/tcp, 0.0.0.0:65285->3306/tcp,
0.0.0.0:15103->8080/tcp,:::15103->8080/tcp
Check the status of the containers
You can check the logs of the containers (vtgate, vttablet101, vttablet102, vttablet103) at any time. For example to check vtgate logs, run the following;
vitess/examples/compose$ docker-compose logs -f vtgate
Load the schema
This step is carried out by the schemaload container
We need to create a few tables into our new cluster. To do that, we can run the ApplySchema command.
vitess/examples/compose$ ./lvtctl.sh ApplySchema -- --sql "$(cat tables/create_messages.sql)" test_keyspace
Create Vschema
This step is carried out by the schemaload container
Create Vschema
vitess/examples/compose$ ./lvtctl.sh ApplyVschema -- --vschema '{"sharded": false }' test_keyspace
Connect to vgate and run queries
vtgate responds to the MySQL protocol, so we can connect to it using the default MySQL client command line.
vitess/examples/compose$ mysql --port=15306 --host=127.0.0.1
Note that you may need to replace 127.0.0.1 with docker ip or docker-machine ip
You can also use the ./lmysql.sh helper script.
vitess/examples/compose$ ./lmysql.sh --port=15306 --host=<DOCKER_HOST_IP>
where <DOCKER_HOST_IP> is docker-machine ip or external docker host ip addr
Play around with vtctl commands
vitess/examples/compose$ ./lvtctl.sh Help
Exploring
-
vtctld web ui: http://localhost:15000
-
vttablets web ui: http://localhost:15101/debug/status http://localhost:15102/debug/status http://localhost:15103/debug/status
-
vtgate web ui: http://localhost:15099/debug/status
-
vtorc ui: http://localhost:13000
-
Stream querylog
curl -S localhost:15099/debug/querylog
Note that you may need to replace localhost with docker ip or docker-machine ip
Troubleshooting
If the cluster gets in a bad state, you most likely will have to stop and kill the containers. Note: you will lose all the data.
vitess/examples/compose$ docker-compose kill
vitess/examples/compose$ docker-compose rm
Advanced Usage
External mysql instance
The compose example has the capability to run against an external mysql instance. Kindly take care to secure your connection to the database. To start vitess against unsharded external mysql, change the following variables in your .env file to match your external database;
KEYSPACE=external_db_name
DB=external_db_name
EXTERNAL_DB=1
DB_HOST=external_db_host
DB_PORT=external_db_port
DB_USER=external_db_user
DB_PASS=external_db_password
DB_CHARSET=CHARACTER SET utf8 COLLATE utf8_general_ci
Important Ensure you have log bin enabled on your external database. Refer to the .cnf files here
Start the cluster
To simulate running vitess against an external cluster, simply change
EXTERNAL_DB=1 when you copy your template.env then run the command below
vitess/examples/compose$ docker-compose up -d
Check cluster status
Once vitess starts, check cluster status
vitess/examples/compose$ docker-compose ps
Name Command State Ports
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
compose_consul1_1 docker-entrypoint.sh agent ... Up 8300/tcp, 8301/tcp, 8301/udp, 8302/tcp, 8302/udp, 0.0.0.0:8400->8400/tcp, 0.0.0.0:8500->8500/tcp, 0.0.0.0:8600->8600/tcp, 8600/udp
compose_consul2_1 docker-entrypoint.sh agent ... Up 8300/tcp, 8301/tcp, 8301/udp, 8302/tcp, 8302/udp, 8400/tcp, 8500/tcp, 8600/tcp, 8600/udp
compose_consul3_1 docker-entrypoint.sh agent ... Up 8300/tcp, 8301/tcp, 8301/udp, 8302/tcp, 8302/udp, 8400/tcp, 8500/tcp, 8600/tcp, 8600/udp
compose_external_db_host_1 docker-entrypoint.sh mysqld Up (healthy) 0.0.0.0:32846->3306/tcp, 33060/tcp
compose_schemaload_1 sh -c /script/schemaload.sh Exit 0
compose_vreplication_1 sh -c [ $EXTERNAL_DB -eq 1 ... Exit 0
compose_vtctld_1 sh -c /vt/bin/vtctld -top ... Up 0.0.0.0:32847->15999/tcp, 0.0.0.0:15000->8080/tcp
compose_vtgate_1 sh -c /vt/bin/vtgate -topo ... Up 0.0.0.0:15306->15306/tcp, 0.0.0.0:32856->15999/tcp, 0.0.0.0:15099->8080/tcp
compose_vtorc_1 sh -c /script/vtorc-up.sh Up (healthy) 0.0.0.0:13000->3000/tcp
compose_vttablet100_1 sh -c [ $EXTERNAL_DB -eq 1 ... Up (healthy) 0.0.0.0:32848->15999/tcp, 0.0.0.0:32849->3306/tcp, 0.0.0.0:15100->8080/tcp
compose_vttablet101_1 sh -c /script/vttablet-up. ... Up (healthy) 0.0.0.0:32850->15999/tcp, 0.0.0.0:32851->3306/tcp, 0.0.0.0:15101->8080/tcp
compose_vttablet102_1 sh -c /script/vttablet-up. ... Up (healthy) 0.0.0.0:32853->15999/tcp, 0.0.0.0:32855->3306/tcp, 0.0.0.0:15102->8080/tcp
compose_vttablet103_1 sh -c /script/vttablet-up. ... Up (healthy) 0.0.0.0:32852->15999/tcp, 0.0.0.0:32854->3306/tcp, 0.0.0.0:15103->8080/tcp
Check replication
The vreplication container included performs the following actions;
- Identifies the correct source and destination tablets for you
- Copies schema from external database to Vitess managed tablet
- Starts VReplication from external keyspace to managed keyspace
- Prints out helpful debug information for you.
vitess/examples/compose$ docker-compose logs -f vreplication
...
Connect to vgate and run queries
vtgate responds to the MySQL protocol, so we can connect to it using the default MySQL client command line. Verify that data was copied and is replicating successfully.
vitess/examples/compose$ ./lmysql --port=15306 --host=<host of machine containers are running in e.g. 127.0.0.1, docker-machine ip e.t.c>
mysql> show databases;
mysql> show databases;
+--------------+
| Databases |
+--------------+
| commerce |
| ext_commerce |
+--------------+
2 rows in set (0.01 sec)
mysql> use commerce@replica;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;
+--------------------+
| Tables_in_commerce |
+--------------------+
| users |
+--------------------+
1 row in set (0.00 sec)
mysql> select count(*) from users;
+----------+
| count(*) |
+----------+
| 1000 |
+----------+
1 row in set (0.00 sec)
Helper Scripts
The following helper scripts are included to help you perform various actions easily
- vitess/examples/compose/lvtctl.sh
- vitess/examples/compose/lmysql.sh
You may run them as below
vitess/examples/compose$ ./lvtctl.sh <args>
To run against a specific compose service/container, use the environment variable $CS
vitess/examples/compose$ (export CS=vttablet101; ./lvtctl.sh <args> )
Custom Image Tags
You may specify a custom vitess:lite image tag by setting the evnironment variable VITESS_TAG.
This is optional and defaults to the latest tag. Example;
- Set
VITESS_TAG=8.0.0in your.envbefore runningdocker-compose up -d - Run
VITESS_TAG=8.0.0; docker-compose up -d
Reference
Checkout this excellent post about The Life of a Vitess Cluster
Documentation
Source Files
Directories