catalog

README

This doc discusses about the common mechanisms shared by all catalog services.

Monitoring

FireCamp integrates with Telegraf to monitor the stateful services. You could create one telegraf service for one stateful service. For more details, please refer toFireCamp Telegraf.

Max Memory and CPU

In general, you could set the reserved and max memory for the stateful service container. By default, the reserved memory is 256MB, and the max memory is unlimited. You could set the them when creating the service, "reserve-memory" and "max-memory". For the JVM based service such as Kafka, the reserved memory will be the max size of JVM heap size and reserved memory, max(JVM heap size, reserved memory).

You could also set the reserved and max cpu for the stateful service container, "reserve-cpuunits" and "max-cpuunits". AWS ECS assigns 1,024 cpu units for every CPU core. By default, the reserved cpu units is 256, and no limit for the max cpu units.

Note: AWS ECS does not support to limit the max cpu units. You could only set the reserved cpu units on ECS.

Logging

FireCamp integrates with cloud logs. On AWS, the service logs are sent to AWS CloudWatch.

AWS ECS and Docker Swarm have the same log group and stream names.

• Every stateful service will have its own log group: clustername-servicename-serviceUUID. Every stateful service member will get the stream name: membername/hostname/containerID.
• The stateful service task, such as the init task, will use the same log group with the stateful service. The log stream has the task type as prefix.
• The stateless service, such as the FireCamp management service, Kafka Manager service, etc, will have the same log group name with the stateful service. The log stream has the service name as prefix.

K8s will be slightly different. K8s does not support custom log driver like AWS ECS and Docker Swarm supports. K8s simply writes the log to the local files. The Fluentd DaemonSet is used to send the logs to CloudWatch. The log group and stream are different, as K8s has namespace and init container concepts.

K8s log group: clustername-namespace-servicename-serviceUUID. K8s log stream: membername/containername/hostname/containerID. FireCamp will set the init/stop container name with the init and stop prefix in the pod. Please refer to FireCamp K8s Fluentd ConfigMap for the format.

FireCamp System Configs

Transparent Huge Pages

The THP (Transparent Huge Pages) is set to madvise. THP might impact the performance for some services such as MongoDB and Redis. While, some service such as Cassandra could benefit from THP.

File Descriptors

The file descriptors are increased 100,000, as the stateful service such as Kafka/ElasticSearch may use a large number of files.

Number of Threads

The number of threads are increased to 64,000. So the stateful service could use as many threads as needed.

Tcp backlog

Both /proc/sys/net/core/somaxconn and /proc/sys/net/ipv4/tcp_max_syn_backlog are increased to 512.

On AWS ECS, the service container directly uses the host network stack for better network performance. So the service container could also see the somaxconn to be 512. While Docker Swarm service creation does not support host network. The docker run command supports modifying the sysctl options, but Docker Swarm does not support the sysctl options yet.

For Docker Swarm, we would have to use some workarounds before Docker Swarm supports the sysctl option. The workaround is: mount host /proc inside a container and configure network stack from within. Create the service with "-v /proc:/writable-proc". Then could change somaxconn inside container, "echo 512 > /writable-proc/sys/net/core/somaxconn", "echo 512 > /writable-proc/sys/net/ipv4/tcp_max_syn_backlog".

Tcp tuning

The idle connection timeout is adjusted. The socket memory is increased to handle thousands of concurrent connections. ''' net.ipv4.tcp_keepalive_time=60 net.ipv4.tcp_keepalive_probes=3 net.ipv4.tcp_keepalive_intvl=10 net.core.rmem_max=16777216 net.core.wmem_max=16777216 net.core.rmem_default=16777216 net.core.wmem_default=16777216 net.core.optmem_max=40960 net.ipv4.tcp_rmem=4096 87380 16777216 net.ipv4.tcp_wmem=4096 65536 16777216 '''

Memory Configs

swapping is bad for performance and stability. For example, if the JVM heap is swapped out to disk, the JVM garbage collections may last for minutes instead of milliseconds and can cause nodes to respond slowly or even to disconnect from the cluster. FireCamp sets the vm.swappiness to 1. This reduces the kernel’s tendency to swap and should not lead to swapping under normal circumstances, while still allowing the whole system to swap in emergency conditions. And the vm.overcommit_memory is disabled. If the system memory is not enough for the stateful service, it would be better to add more memory.

mmap may be used by some stateful services such as ElasticSearch, Cassandra. The vm.max_map_count is increased to 1048575.

memlock is set to unlimited. So the service such as Cassandra could lock the memory in RAM and never get moved to the swap disk.

Documentation

Index

• Constants
• func GenServiceMemberHosts(cluster string, service string, replicas int64) string
• func GenServiceMemberHostsWithPort(cluster string, service string, replicas int64, port int64) string
• func GenServiceMemberURIs(cluster string, service string, replicas int64, port int64) string
• func GenStatelessServiceReplicaConfigs(cluster string, service string, replicas int) []*manage.ReplicaConfig
• func IsMemberConfigFile(filename string) bool
• func IsServiceConfigFile(filename string) bool
• func UpdateServiceConfigHeapAndJMX(oldContent string, heapSizeMB int64, jmxUser string, jmxPasswd string) string
• func ValidateUpdateOptions(heapSizeMB int64, jmxUser string, jmxPasswd string) error

Constants

const (
	// The service configurable variables and some service specific variables
	SERVICE_FILE_NAME = "service.conf"
	// The specific variables for one service member
	MEMBER_FILE_NAME = "member.conf"

	BindAllIP = "0.0.0.0"

	// the default password is uuid for better security
	// every service will define its own jmx port
	JmxDefaultRemoteUser = "jmxuser"
	JmxReadOnlyAccess    = "readonly"
	JmxReadWriteAccess   = "readwrite"
)

Functions

func GenServiceMemberHosts

func GenServiceMemberHosts(cluster string, service string, replicas int64) string

GenServiceMemberHosts creates the hostname list of all service members, example: myzoo-0.t1-firecamp.com,myzoo-1.t1-firecamp.com Note: this currently works for the service that all members have a single name format, such as ZooKeeper, Kafka, etc. For service such as ElasticSearch, MongoDB, that has different name formats, this is not suitable.

func GenServiceMemberHostsWithPort

func GenServiceMemberHostsWithPort(cluster string, service string, replicas int64, port int64) string

GenServiceMemberHostsWithPort creates the hostname:port list of all service members, example: myzoo-0.t1-firecamp.com:2181,myzoo-1.t1-firecamp.com:2181 Note: this currently works for the service that all members have a single name format, such as ZooKeeper, Kafka, etc. For service such as ElasticSearch, MongoDB, that has different name formats, this is not suitable.

func GenServiceMemberURIs

func GenServiceMemberURIs(cluster string, service string, replicas int64, port int64) string

GenServiceMemberURIs creates the list of URIs for all service members, example: http://myes-0.t1-firecamp.com:9200,http://myes-1.t1-firecamp.com:9200

func GenStatelessServiceReplicaConfigs

func GenStatelessServiceReplicaConfigs(cluster string, service string, replicas int) []*manage.ReplicaConfig

GenStatelessServiceReplicaConfigs generates the replica configs for the stateless service.

func IsMemberConfigFile

func IsMemberConfigFile(filename string) bool

IsMemberConfigFile checks if the file is the member.conf file

func IsServiceConfigFile

func IsServiceConfigFile(filename string) bool

IsServiceConfigFile checks if the file is the service.conf file

func UpdateServiceConfigHeapAndJMX

func UpdateServiceConfigHeapAndJMX(oldContent string, heapSizeMB int64, jmxUser string, jmxPasswd string) string

UpdateServiceConfigHeapAndJMX updates the service.conf file content

func ValidateUpdateOptions

func ValidateUpdateOptions(heapSizeMB int64, jmxUser string, jmxPasswd string) error

ValidateUpdateOptions checks if the update options are valid