cascatalog

README

• FireCamp Cassandra Internals
• Tutorials

FireCamp Cassandra Internals

The FireCamp Cassandra container is based on the official Cassandra image. Two volumes will be created, one for the commit log, the other for all other data. The commit log volume will be mounted to the /journal directory inside container. The data volume will be mounted to the /data directory inside container. Cassandra commit log is stored under /journal directory. All other Cassandra data, including data file, hints, and the key cache, will be stored under /data directory.

Topology

Cassandra Ec2Snitch could load the AWS region and availability zone from the EC2 API. But Ec2Snitch could not support other clouds, such as Google or Microsoft Cloud. So the GossipingPropertyFileSnitch is used to teach Cassandra about the topology. Every member will record the region as dc, and the availability zone as rack.

Only one region is supported currently. The multi-regions Cassandra will be supported in the future.

Scale

Scaling a FireCamp Cassandra service requires 4 steps. For example, scaling a 3 replicas Cassandra service to 6 replicas. Cassandra service name is "mycas". Cluster name is "t1". If you want to scale to a large number of replicas, please repeat below steps to scale 3 replicas at one time.

1. Scale the cluster nodes. For example, on AWS, go to the EC2 AutoScalingGroup console and scale the FireCamp cluster nodes.

2. Scale the replicas using firecamp cli.

firecamp-service-cli -op=scale-service -service-type=cassandra -region=us-east-1 -cluster=t1 -service-name=mycas -replicas=6

1. Use nodetool status to verify the new replicas are fully bootstrapped and all other nodes are up (UN).

nodetool -h mycas-3.t1-firecamp.com -u cassandrajmx -pw changeme status
nodetool -h mycas-4.t1-firecamp.com -u cassandrajmx -pw changeme status
nodetool -h mycas-5.t1-firecamp.com -u cassandrajmx -pw changeme status

1. Run nodetool cleanup on each of the previously existing nodes to remove the keys that no longer belong to those nodes. Wait for cleanup to complete on one node before running nodetool cleanup on the next node. Cleanup can be safely postponed for low-usage hours.

nodetool -h mycas-0.t1-firecamp.com -u cassandrajmx -pw changeme cleanup
nodetool -h mycas-1.t1-firecamp.com -u cassandrajmx -pw changeme cleanup
nodetool -h mycas-2.t1-firecamp.com -u cassandrajmx -pw changeme cleanup

Currently FireCamp only supports scaling from 3 replicas. Scaling from 1 replica is not supported yet. Scaling down the replicas are not supported as well. If you want to stop the service when idle, could run firecamp-service-cli -op=stop-service -service-name=mycas. This will stop the service containers, while the volumes are still there. Later could start the containers with firecamp-service-cli -op=start-service -service-name=mycas.

Troubleshooting

If the new replicas do not work, probably the new node is not correctly added to the cluster.

1. Check if the new node is added to the container orchestration framework.

For ECS, go to ECS console, click the right cluster, and check if the new nodes show up in the container instances. If not, find the corresponding EC2 instance and terminate it. The AutoScalingGroup will create and initialize a new EC2 instance automatically.

For Docker Swarm, login to swarm manager node, run docker node ls and check if the new node is added.

1. Check the firecamp volume plugin is correctly installed.

Login to the EC2 instance, run sudo docker plugin ls. If firecamp-volume plugin is not installed or not enabled, terminate the corresponding EC2 instance. The AutoScalingGroup will create and initialize a new EC2 instance automatically.

Security

The FireCamp Cassandra follows the official security guide. The Authentication and Authorization are enabled by default.

The TLS/SSL encryption will be supported in the future. This would not be a big security risk. The Cassandra cluster on the FireCamp platform could only be accessed by the nodes running on the AppAccessSecurityGroup and the same VPC. And both Authentication and Authorization are enabled.

After the Cassandra cluster is initialized, please login to create a new superuser and disable the default "cassandra" superuser. For example, cluster name is t1, cassandra service name is mycas. The steps are:

1. Login to the cluster: cqlsh mycas-0.t1-firecamp.com -u cassandra -p cassandra

2. Create a new superuser: CREATE ROLE newsuperuser WITH SUPERUSER = true AND LOGIN = true AND PASSWORD = 'super';

3. Log out.

4. Login to the cluster with the new superuser: cqlsh mycas-0.t1-firecamp.com -u newsuperuser -p super

5. Disable the default superuser: ALTER ROLE cassandra WITH SUPERUSER = false AND LOGIN = false;

6. Finally, set up the roles and credentials for your application users with CREATE ROLE statements.

Cache

The Cassandra key cache is enabled, and the row cache is disabled. The key cache size is automatically calculated based on the total memory of the node.

Note: the FireCamp Cassandra assumes the whole node is only used by Cassandra. We may consider to limit the max cpu and memory for Cassandra container in the future.

Logging

The Cassandra logs are sent to the Cloud Logs, such as AWS CloudWatch logs.

Configs

JMX

By default, JMX is enabled to allow tools such as nodetool to access the Cassandra replica remotely. You could specify the JMX user and password when creating the service. If you do not specify the JMX user and password, the default user is "cassandrajmx" and an UUID will be generated as the password.

JVM Configs

The default Java heap size, both Xmx and Xms, are set to 8GB. The max heap size is 14GB, as Cassandra recommends. If your wants other memory, you could specify the "cas-heap-size" when creating the Cassandra service by the firecamp-service-cli. Note: if the heap size is less than 1GB and you run some load, JVM may stall long time at GC.

Set JVM TTL for Java CQL driver

By default, JVM caches a successful DNS lookup forever. If you use Cassandra Java CQL driver, please set JVM TTL to a reasonable value such as 60 seconds. So when Cassandra container moves to another node, Java CQL driver could lookup the new address.

Refs:

[1] Cassandra on AWS White Paper

[2] Planning a Cassandra cluster on Amazon EC2

Tutorials

This is a simple tutorial about how to create a Cassandra service and how to use it. This tutorial assumes the cluster name is "t1", the AWS Region is "us-east-1", and the Cassandra service name is "mycas".

Create a Cassandra service

Follow the Installation guide to create a 3 nodes cluster across 3 availability zones. Create a Cassandra cluster:

firecamp-service-cli -op=create-service -service-type=cassandra -region=us-east-1 -cluster=t1 -service-name=mycas -replicas=3 -volume-size=100 -journal-volume-size=10

This creates a 3 replicas Cassandra on 3 availability zones. Each replica has 2 volumes, 10GB volume for journal and 100GB volume for data. The DNS names of the replicas would be: mycas-0.t1-firecamp.com, mycas-1.t1-firecamp.com, mycas-2.t1-firecamp.com. To reduce the heap size for the simple test, could set such as -cas-heap-size=512.

The Cassandra service creation steps:

1. Create the Volumes and persist the metadata to the FireCamp DB. This is usually very fast. But if AWS is slow at creating the Volume, this step will be slow as well.
2. Create the service on the container orchestration framework, such as AWS ECS, and wait for all service containers running. The speed depends on how fast the container orchestration framework could start all containers.
3. Wait some time (10 seconds) for Cassandra to stabilize.
4. Start the Cassandra initialization task. The speed also depends on how fast the container orchestration framework schedules the task.
5. The initialization task does:
   • Increase the replication factor for the system_auth keyspace to 3.
   • Notify the manage server to do the post initialization work.
6. The manage server will set the Cassandra service initialized.

In case the service creation fails, please simply retry it. Once the service is initialized, you could create a new super user.

Create the New Super User

Cassandra has a default super user, name "cassandra" and password "cassandra". After the Cassandra service is initialized, please login to create a new superuser and disable the default "cassandra" superuser. The steps are:

1. Login to the cluster: cqlsh mycas-0.t1-firecamp.com -u cassandra -p cassandra
2. Create a new superuser: CREATE ROLE newsuperuser WITH SUPERUSER = true AND LOGIN = true AND PASSWORD = 'super';
3. Log out.
4. Login to the cluster with the new superuser: cqlsh mycas-0.t1-firecamp.com -u newsuperuser -p super
5. Disable the default superuser: ALTER ROLE cassandra WITH SUPERUSER = false AND LOGIN = false;
6. Finally, set up the roles and credentials for your application users with CREATE ROLE statements.

Create Keyspace and User

1. Login with the new superuser: cqlsh mycas-0.t1-firecamp.com -u newsuperuser -p super
2. Create a keyspace and table

CREATE KEYSPACE test WITH REPLICATION = { 'class' : 'NetworkTopologyStrategy', 'us-east-1' : 3 };
use test;
CREATE TABLE users (userid text PRIMARY KEY, first_name text, last_name text);

1. Create role for the keyspace and table

CREATE ROLE supervisor;
GRANT MODIFY ON test.users TO supervisor;
GRANT SELECT ON test.users TO supervisor;
CREATE ROLE pam WITH PASSWORD = 'password' AND LOGIN = true;
GRANT supervisor TO pam;
LIST ALL PERMISSIONS OF pam;

1. Login as the new "pam" role: cqlsh mycas-2.t1-firecamp.com -u pam -p password
2. Insert new records:

use test;
insert into users (userid, first_name, last_name) values('user1', 'a1', 'b1');
select * from users;

Documentation

Index

• Constants
• func GenDefaultCreateServiceRequest(platform string, region string, azs []string, cluster string, service string, ...) (*manage.CreateServiceRequest, error)
• func GenDefaultInitTaskRequest(req *manage.ServiceCommonRequest, logConfig *cloudlog.LogConfig, ...) *containersvc.RunTaskOptions
• func GenInitTaskEnvKVPairs(region string, cluster string, service string, manageurl string) []*common.EnvKeyValuePair
• func GenReplicaConfigs(platform string, region string, cluster string, service string, azs []string, ...) []*manage.ReplicaConfig
• func IsJmxConfFile(filename string) bool
• func IsJvmConfFile(filename string) bool
• func NewJVMConfContent(heapSizeMB int64) string
• func NewJmxConfContent(jmxUser string, jmxPasswd string) string
• func ValidateRequest(req *manage.CatalogCreateCassandraRequest) error
• func ValidateUpdateRequest(req *manage.CatalogUpdateCassandraRequest) error

Constants

const (

	// ContainerImage is the main Cassandra running container.
	ContainerImage = common.ContainerNamePrefix + "cassandra:" + defaultVersion
	// InitContainerImage initializes the Cassandra cluster.
	InitContainerImage = common.ContainerNamePrefix + "cassandra-init:" + defaultVersion

	// DefaultHeapMB is the default Cassandra JVM heap size.
	// https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_tune_jvm_c.html
	DefaultHeapMB = 8192
	// MaxHeapMB is the max Cassandra JVM heap size. Cassandra recommends no more than 14GB
	MaxHeapMB = 14 * 1024
	// MinHeapMB is the minimal JVM heap size. If heap < 1024, jvm may stall long time at gc.
	MinHeapMB = 1024
	// DefaultJmxRemoteUser is the default jmx remote user.
	DefaultJmxRemoteUser = "cassandrajmx"
)

Functions

func GenDefaultCreateServiceRequest

func GenDefaultCreateServiceRequest(platform string, region string, azs []string,
	cluster string, service string, opts *manage.CatalogCassandraOptions, res *common.Resources) (*manage.CreateServiceRequest, error)

GenDefaultCreateServiceRequest returns the default service creation request.

func GenDefaultInitTaskRequest

func GenDefaultInitTaskRequest(req *manage.ServiceCommonRequest, logConfig *cloudlog.LogConfig,
	serviceUUID string, manageurl string) *containersvc.RunTaskOptions

GenDefaultInitTaskRequest returns the default Cassandra init task request.

func GenInitTaskEnvKVPairs

func GenInitTaskEnvKVPairs(region string, cluster string, service string, manageurl string) []*common.EnvKeyValuePair

GenInitTaskEnvKVPairs generates the environment key-values for the init task.

func GenReplicaConfigs

func GenReplicaConfigs(platform string, region string, cluster string, service string, azs []string, opts *manage.CatalogCassandraOptions) []*manage.ReplicaConfig

GenReplicaConfigs generates the replica configs.

func IsJmxConfFile

func IsJmxConfFile(filename string) bool

IsJmxConfFile checks if the file is jmx conf file

func IsJvmConfFile

func IsJvmConfFile(filename string) bool

IsJvmConfFile checks if the file is jvm conf file

func NewJVMConfContent

func NewJVMConfContent(heapSizeMB int64) string

NewJVMConfContent returns the new jvm.options file content

func NewJmxConfContent

func NewJmxConfContent(jmxUser string, jmxPasswd string) string

NewJmxConfContent returns the new jmxremote.password file content

func ValidateRequest

func ValidateRequest(req *manage.CatalogCreateCassandraRequest) error

ValidateRequest checks if the create request is valid

func ValidateUpdateRequest

func ValidateUpdateRequest(req *manage.CatalogUpdateCassandraRequest) error

ValidateUpdateRequest checks if the update request is valid