Vault Secrets Kafka Plugin
Description
This is a vault secrets plugin for kafka.
This plugin integrates with kafka clusters with SASL_SSL SCRAM-SHA-256 and
SASL_SSL SCRAM-SHA-512 authentication mechanisms.
Installation
-
Download the plugin from the releases page:
curl -LO https://github.com/palkx/vault-plugin-secrets-kafka/releases/download/v0.0.4/vault-plugin-secrets-kafka_0.0.4_linux_amd64.zip
-
Unzip it: unzip vault-plugin-secrets-kafka_0.0.4_linux_amd64.zip
-
Copy kafka plugin to the vault plugins directory. For example if you set
plugin_directory = "/vault/plugins"
in you vault server config:
cp kafka /vault/plugins/
-
Start vault, unseal it and register the plugin:
vault plugin register -sha256="$(sha256sum /vault/plugins/kafka | awk '{print $1}')" secret kafka
-
Create new backend: vault secrets enable -path=kafka kafka
-
Backend configuration supports the following options:
username
: SASL username. This user will be used to create
credentials/ACLs in your kafka cluster.
password
: SASL password.
bootstrap_servers
: comma separated list of kafka brokers.
ca_bundle
(optional): Base64 encoded CA certificates. If present they're
will be used to verify the server certificate.
certificate
(optional): Base64 encoded client certificate. If specified
this certificate will be provided to authenticate in Kafka cluster. When
provided the certificate_key
must be provided as well.
certificate_key
(optional): Base64 encoded client certificate key. If
specified this key will be used to authenticate in Kafka cluster.
scram_sha_version
(optional): SCRAM-SHA version. The default is 512
.
Valid values are 256
and 512
.
-
Example backend configuration:
vault write kafka/config username=root password=rootpassword bootstrap_servers=kafka:29092 ca_bundle=$(cat /secrets/ca.pem|base64 -w0) certificate=$(cat /secrets/kafka.pem|base64 -w0) certificate_key=$(cat /secrets/kafka-key.pem|base64 -w0) scram_sha_version=512
-
Role configuration supports the following options:
username_prefix
(optional): prefix, which will be for credentials. If not
specified defaults to role name.
scram_sha_version
(optional): SCRAM-SHA version, which will be for
credentials. The default is 512
. Valid values are 256
and 512
.
resource_acls
(optional): JSON encoded array of ACL permissions to be
created for the role credentials. This set of ACLs is created for each
credential created and will be deleted on lease expire. By default it's
empty. JSON structure described below.
ttl
(optional): The TTL of the role credentials. Specified in seconds.
max_ttl
(optional): The Maximum TTL of the role credentials. Specified in
seconds.
-
Example role configuration:
vault write kafka/role/svc resource_acls='[{"ResourceType":"topic","ResourceName":"test-topic","ResourcePatternType":"literal","Acls":[{"Host":"*","Operation":"read","PermissionType":"allow"},{"Host":"*","Operation":"write","PermissionType":"allow"},{"Host":"*","Operation":"describeConfigs","PermissionType":"allow"}]}]'
-
Read role credentials: vault read kafka/creds/svc
. Example output:
vault read kafka/creds/svc
Key Value
--- -----
lease_id kafka/creds/svc/7RC40tu6fwd5ZzUlJsj63A0e
lease_duration 768h
lease_renewable true
password c70f0d84-ce17-4644-8e58-f842636b66c3
username svc-token-ae6e9d30-bdd1-4d3a-b609-c074600dede0
ACL JSON Structure
Example JSON structure:
[
{
"ResourceType": "Topic",
"ResourceName": "test-topic",
"ResourcePatternType": "Literal",
"Acls": [
{
"Host": "*",
"Operation": "Read",
"PermissionType": "Allow"
},
{
"Host": "*",
"Operation": "Write",
"PermissionType": "Allow"
},
{
"Host": "*",
"Operation": "DescribeConfigs",
"PermissionType": "Allow"
}
]
},
{
"ResourceType": "Topic",
"ResourceName": "test-topic2",
"ResourcePatternType": "Prefixed",
"Acls": [
{
"Host": "*",
"Operation": "All",
"PermissionType": "Allow"
}
]
}
]
For all available options and values check
kafka ACLs documentation
To compact this json you can use jq
. For example: jq -rcM '.' acls.json
Local development
Currently we're using go 1.22.x.
To perform local build and test run the following commands:
- Compile the plugin, launch kafka with SASL_SSL SCRAM-SHA-512 and run the
vault server:
docker compose up -d
- Exec into the vault container:
docker compose exec vault ash
- Enable our plugin in the vault:
vault secrets enable -path=kafka kafka
- Configure the plugin:
vault write kafka/config username=root password=rootpassword bootstrap_servers=kafka:29092 ca_bundle=$(cat /secrets/ca.pem|base64 -w0) certificate=$(cat /secrets/kafka.pem|base64 -w0) certificate_key=$(cat /secrets/kafka-key.pem|base64 -w0)
- Create a role:
vault write kafka/role/svc resource_acls='[{"ResourceType":"topic","ResourceName":"test-topic","ResourcePatternType":"literal","Acls":[{"Host":"*","Operation":"read","PermissionType":"allow"},{"Host":"*","Operation":"write","PermissionType":"allow"},{"Host":"*","Operation":"describeConfigs","PermissionType":"allow"}]},{"ResourceType":"topic","ResourceName":"test-topic2","ResourcePatternType":"prefixed","Acls":[{"Host":"*","Operation":"all","PermissionType":"allow"}]}]'
- Generate credentials:
vault read kafka/creds/svc
(should successfully
create a user in the kafka cluster)
Roadmap
- Initialize plugin skeleton
- Write first working prototype
- Remove dependency on confluent-kafka go plugin (it's using C dependencies
and because of this we can't perform statically linked builds)
- Configure basic CI/CD flow
- Revisit kafka client lifecycle
- Improve credential revocation during service disruption events
- Select SCRAM-SHA version on the plugin config/role level
- Specify TLS certificates and CA on the plugin config level
- Revisit tests
- Revisit acceptance tests
- Add ability to manage ACLs