catalog

README

This doc discusses about the common mechanisms shared by all catalog services.

Monitoring

FireCamp integrates with Telegraf to monitor the stateful services. You could create one telegraf service for one stateful service. For more details, please refer toFireCamp Telegraf.

Max Memory and CPU

In general, you could set the reserved and max memory for the stateful service container. By default, the reserved memory is 256MB, and the max memory is unlimited. You could set the them when creating the service, "reserve-memory" and "max-memory". For the JVM based service such as Kafka, the reserved memory will be the max size of JVM heap size and reserved memory, max(JVM heap size, reserved memory).

You could also set the reserved and max cpu for the stateful service container, "reserve-cpuunits" and "max-cpuunits". AWS ECS assigns 1,024 cpu units for every CPU core. By default, the reserved cpu units is 256, and no limit for the max cpu units.

Note: AWS ECS does not support to limit the max cpu units. You could only set the reserved cpu units on ECS.

Logging

FireCamp integrates with cloud logs. On AWS, the service logs are sent to AWS CloudWatch.

AWS ECS and Docker Swarm have the same log group and stream names.

• Every stateful service will have its own log group: clustername-servicename-serviceUUID. Every stateful service member will get the stream name: membername/hostname/containerID.
• The stateful service task, such as the init task, will use the same log group with the stateful service. The log stream has the task type as prefix.
• The stateless service, such as the FireCamp management service, Kafka Manager service, etc, will have the same log group name with the stateful service. The log stream has the service name as prefix.

K8s will be slightly different. K8s does not support custom log driver like AWS ECS and Docker Swarm supports. K8s simply writes the log to the local files. The Fluentd DaemonSet is used to send the logs to CloudWatch. The log group and stream are different, as K8s has namespace and init container concepts.

K8s log group: clustername-namespace-servicename-serviceUUID. K8s log stream: membername/containername/hostname/containerID. FireCamp will set the init/stop container name with the init and stop prefix in the pod. Please refer to FireCamp K8s Fluentd ConfigMap for the format.

FireCamp System Configs

Transparent Huge Pages

The THP (Transparent Huge Pages) is set to madvise. THP might impact the performance for some services such as MongoDB and Redis. While, some service such as Cassandra could benefit from THP.

File Descriptors

The file descriptors are increased 100,000, as the stateful service such as Kafka/ElasticSearch may use a large number of files.

Number of Threads

The number of threads are increased to 64,000. So the stateful service could use as many threads as needed.

Tcp backlog

Both /proc/sys/net/core/somaxconn and /proc/sys/net/ipv4/tcp_max_syn_backlog are increased to 512.

On AWS ECS, the service container directly uses the host network stack for better network performance. So the service container could also see the somaxconn to be 512. While Docker Swarm service creation does not support host network. The docker run command supports modifying the sysctl options, but Docker Swarm does not support the sysctl options yet.

For Docker Swarm, we would have to use some workarounds before Docker Swarm supports the sysctl option. The workaround is: mount host /proc inside a container and configure network stack from within. Create the service with "-v /proc:/writable-proc". Then could change somaxconn inside container, "echo 512 > /writable-proc/sys/net/core/somaxconn", "echo 512 > /writable-proc/sys/net/ipv4/tcp_max_syn_backlog".

Tcp tuning

The idle connection timeout is adjusted. The socket memory is increased to handle thousands of concurrent connections. ''' net.ipv4.tcp_keepalive_time=60 net.ipv4.tcp_keepalive_probes=3 net.ipv4.tcp_keepalive_intvl=10 net.core.rmem_max=16777216 net.core.wmem_max=16777216 net.core.rmem_default=16777216 net.core.wmem_default=16777216 net.core.optmem_max=40960 net.ipv4.tcp_rmem=4096 87380 16777216 net.ipv4.tcp_wmem=4096 65536 16777216 '''

Memory Configs

swapping is bad for performance and stability. For example, if the JVM heap is swapped out to disk, the JVM garbage collections may last for minutes instead of milliseconds and can cause nodes to respond slowly or even to disconnect from the cluster. FireCamp sets the vm.swappiness to 1. This reduces the kernel’s tendency to swap and should not lead to swapping under normal circumstances, while still allowing the whole system to swap in emergency conditions. And the vm.overcommit_memory is disabled. If the system memory is not enough for the stateful service, it would be better to add more memory.

mmap may be used by some stateful services such as ElasticSearch, Cassandra. The vm.max_map_count is increased to 1048575.

memlock is set to unlimited. So the service such as Cassandra could lock the memory in RAM and never get moved to the swap disk.

Documentation

Index

• Constants
• func CreateJmxAccessConfFileContent(jmxUser string, accessPerm string) string
• func CreateJmxPasswdConfFileContent(jmxUser string, jmxPasswd string) string
• func CreateJmxRemoteAccessConfFile(jmxUser string, accessPerm string) *manage.ReplicaConfigFile
• func CreateJmxRemotePasswdConfFile(jmxUser string, jmxPasswd string) *manage.ReplicaConfigFile
• func CreateSysConfigFile(platform string, memberDNSName string) *manage.ReplicaConfigFile
• func GenServiceMemberHosts(cluster string, service string, replicas int64) string
• func GenServiceMemberHostsWithPort(cluster string, service string, replicas int64, port int64) string
• func GenServiceMemberURIs(cluster string, service string, replicas int64, port int64) string
• func GenStatelessServiceReplicaConfigs(platform string, cluster string, service string, replicas int) []*manage.ReplicaConfig
• func IsJmxAccessConfFile(filename string) bool
• func IsJmxPasswdConfFile(filename string) bool
• func MBToBytes(mb int64) int64
• func ReplaceJmxUserInAccessConfFile(content string, newUser string, oldUser string) string

Constants

const (
	// The system variables in the sys.conf file
	SYS_FILE_NAME = "sys.conf"

	BindAllIP = "0.0.0.0"

	JmxRemotePasswdConfFileName = "jmxremote.password"
	JmxRemoteAccessConfFileName = "jmxremote.access"
	JmxConfFileMode             = 0400
	// the default password is uuid for better security
	// every service will define its own jmx port
	JmxDefaultRemoteUser = "jmxuser"
	JmxReadOnlyAccess    = "readonly"
	JmxReadWriteAccess   = "readwrite"
)

Functions

func CreateJmxAccessConfFileContent

func CreateJmxAccessConfFileContent(jmxUser string, accessPerm string) string

CreateJmxAccessConfFileContent returns the jmxremote.access file content

func CreateJmxPasswdConfFileContent

func CreateJmxPasswdConfFileContent(jmxUser string, jmxPasswd string) string

CreateJmxPasswdConfFileContent returns the jmxremote.password file content

func CreateJmxRemoteAccessConfFile

func CreateJmxRemoteAccessConfFile(jmxUser string, accessPerm string) *manage.ReplicaConfigFile

CreateJmxRemoteAccessConfFile creates the jmx remote access file.

func CreateJmxRemotePasswdConfFile

func CreateJmxRemotePasswdConfFile(jmxUser string, jmxPasswd string) *manage.ReplicaConfigFile

CreateJmxRemotePasswdConfFile creates the jmx remote password file.

func CreateSysConfigFile

func CreateSysConfigFile(platform string, memberDNSName string) *manage.ReplicaConfigFile

CreateSysConfigFile creates the content for the sys.conf file. example:

PLATFORM=ecs
SERVICE_MEMBER=mycas-0.cluster-firecamp.com

func GenServiceMemberHosts

func GenServiceMemberHosts(cluster string, service string, replicas int64) string

GenServiceMemberHosts creates the hostname list of all service members, example: myzoo-0.t1-firecamp.com,myzoo-1.t1-firecamp.com Note: this currently works for the service that all members have a single name format, such as ZooKeeper, Kafka, etc. For service such as ElasticSearch, MongoDB, that has different name formats, this is not suitable.

func GenServiceMemberHostsWithPort

func GenServiceMemberHostsWithPort(cluster string, service string, replicas int64, port int64) string

GenServiceMemberHostsWithPort creates the hostname:port list of all service members, example: myzoo-0.t1-firecamp.com:2181,myzoo-1.t1-firecamp.com:2181 Note: this currently works for the service that all members have a single name format, such as ZooKeeper, Kafka, etc. For service such as ElasticSearch, MongoDB, that has different name formats, this is not suitable.

func GenServiceMemberURIs

func GenServiceMemberURIs(cluster string, service string, replicas int64, port int64) string

GenServiceMemberURIs creates the list of URIs for all service members, example: http://myes-0.t1-firecamp.com:9200,http://myes-1.t1-firecamp.com:9200

func GenStatelessServiceReplicaConfigs

func GenStatelessServiceReplicaConfigs(platform string, cluster string, service string, replicas int) []*manage.ReplicaConfig

GenStatelessServiceReplicaConfigs generates the replica configs for the stateless service.

func IsJmxAccessConfFile

func IsJmxAccessConfFile(filename string) bool

IsJmxAccessConfFile checks if the file is jmx access conf file

func IsJmxPasswdConfFile

func IsJmxPasswdConfFile(filename string) bool

IsJmxPasswdConfFile checks if the file is jmx password conf file

func MBToBytes

func MBToBytes(mb int64) int64

MBToBytes converts MB to bytes

func ReplaceJmxUserInAccessConfFile

func ReplaceJmxUserInAccessConfFile(content string, newUser string, oldUser string) string

ReplaceJmxUserInAccessConfFile replaces the old jmx user in the access conf file