kafkacatalog

README

The FireCamp Kafka container is based on openjdk. The data volume will be mounted to the /data directory inside container. The Kafka logs will be stored under /data/kafka.

Configs

The FireCamp Kafka configs follows Kafka Configs for the Production Environment.

Replication Factor The default Replication Factor for Kafka is set to 3. The production environment should have at least 3 nodes across 3 availability zones. If the number of nodes is less than 3, the replication factor will be automatically adjusted to the number of nodes. This is for the developing and testing environment with 1 or 2 nodes.

Topology The replicas of one topic are automatically distributed to the availability zones in the cluster. If the cluster has 3 availability zones, each zone will have one replica. The production cluster should have 3 availability zones, to tolerate the availability zone failure.

Minimum In-sync Replica The Minimum In-sync Replica is set to 2 by default. Please set the "acks" to "all" or "-1" in the producer configs. When a producer sets acks to "all" (or "-1"), the min.insync.replicas specifies the minimum number of replicas that must acknowledge a write for the write to be considered successful. This guarantees that the message is not lost unless both hosts crash. If the cluster has only 1 node for developing and testing, the min.insync.replicas will be automatically adjusted to 1.

Unclean Leader Election The Unclean Leader Election is disabled by default. If the unclean leader election is enabled, the out-of-sync replica is allowed to become the leader and messages that were not synced to the new leader are lost.

Auto Topic Creation The auto topic creation is enabled by default. The default max number of partitions is 8. If the cluster has more than 8 nodes, the auto-created topic will have 8 partitions. If the total nodes are less than 8, the partitions of the auto-created topic will be the same with the number of nodes.

JVM Configs Java 1.8 with G1 collector is used. The default Java heap size, both Xmx and Xms, are set to 6GB. If your Kafka wants other memory, you could specify the "reserve-memory" when creating the Kafka service by the firecamp-service-cli. The Java GC tuning also follow the recommendation.

Set JVM TTL for Kafka Java client By default, JVM caches a successful DNS lookup forever. Kafka Java client should set JVM TTL to a reasonable value such as 60 seconds. So when Kafka container moves to another node, JVM could lookup the new address.

Max Open Files The node's file descriptor count is increased to very large, 100,000, as Kafka uses a very large number of files.

Logging

The Kafka logs are sent to the Cloud Logs, such as AWS CloudWatch logs.

Security

The Kafka Security will be enabled in the coming releases.

Refs

1. Design and Deployment Considerations for Deploying Apache Kafka on AWS

Documentation

Index

• Constants
• func GenDefaultCreateServiceRequest(platform string, region string, azs []string, cluster string, service string, ...) *manage.CreateServiceRequest
• func GenReplicaConfigs(platform string, cluster string, service string, azs []string, replicas int64, ...) []*manage.ReplicaConfig

Constants

const (
	// ContainerImage is the main running container.
	ContainerImage = common.ContainerNamePrefix + "kafka:" + common.Version

	// DefaultHeapMB is the default kafka java heap size
	DefaultHeapMB = 6144
)

Functions

func GenDefaultCreateServiceRequest

func GenDefaultCreateServiceRequest(platform string, region string, azs []string,
	cluster string, service string, replicas int64, volSizeGB int64, res *common.Resources,
	allowTopicDel bool, retentionHours int64, zkattr *common.ServiceAttr) *manage.CreateServiceRequest

GenDefaultCreateServiceRequest returns the default service creation request.

func GenReplicaConfigs

func GenReplicaConfigs(platform string, cluster string, service string, azs []string, replicas int64,
	reserveMemMB int64, allowTopicDel bool, retentionHours int64, zkServers string) []*manage.ReplicaConfig

GenReplicaConfigs generates the replica configs.