kms-vault-operator

README

KMS Vault operator

• KMS Vault operator
  • Intro
  • Description
  • Configuration
    • AWS
    • Vault
      • Kubernetes authentication method (vaultAuthMethod: k8s)
      • Vault token authentication method (vaultAuthMethod: token)
      • Vault userpass authentication method (vaultAuthMethod: userpass)
    • Deploying the operator
    • Creating a secret
    • Partial secrets
  • For security nerds
    • Docker images are signed and published to Docker Hub's Notary server
    • Docker images are labeled with Git and GPG metadata
  • Important notes by this project
    • Kubernetes namespaces and Vault namespaces
    • Multiple secrets writing to the same location
    • No validation on target path
    • Removing secrets when a KMSVaultSecret is deleted.
    • Decryption or decoding errors are ignored
    • Support for K/V V2 is limited (as of this version)
    • Partial secrets don't validate keys
    • Partial secrets don't support finalizers (yet)
  • Help wanted!

Intro

We all know (or should know) that keeping secrets in plain text in source control is definitely a big no-no. That forces us to keep sensitive text in a different way like a password manager, or worse, in an encrypted file in the same repository (but where do we keep the password for that?!). So we either have a "first secret" problem, or a "source of truth" problem, or an automation problem.

KMS provides a nice solution to the encryption/decryption problem so we can keep encrypted secrets in source control (and thus closer to the code that depends on it). But more often than not, said secret is not very valuable by itself and needs to be decrypted and put where it can be consumed at runtime, e.g. in Vault.

One pattern that can accomplish this is KMS + Vault + Terraform. By creating a KMS key (which can be done with Terraform), we can encrypt a secret that can be safely stored in source control. It can then be decrypted by a Terraform aws_kms_secrets data source and passed to a vault_generic_secret resource to be written into Vault. Top it off with the inmem storage backend and the secret will never be in plaintext in durable storage.

While this gets the job done for storing secrets in a secure way, it doesn't lend itself to automation. Options like Terraform Enterprise are often prohibitively expensive, and ad-hoc solutions like running terraform apply on a cron job are fragile and insecure.

The goal of this operator is to minimize the security risk of secret exposure, while leveraging Kubernetes to provide the automation to minimize configuration drift.

Description

This operator manages KMSVaultSecret CRDs containing secrets encrypted with a KMS key (and base64-encoded) and stores them in Vault. This allows you to securely manage your Vault secrets in source control and because the decryption only happens at runtime, it's only in the operator memory, minimizing the exposure of the plain text secret. The operator resources and scaffolding code are managed and generated automatically with the operator-sdk framework.

The AWS credentials to do the decryption operation use aws-sdk-go and follows the default credential precedence order, so if you're going to inject environment variables or configuration files, you should do so on the operator's Deployment manifest (e.g. deploy/operator.yaml).

Each secret will define a path, a vaultAuthMethod, a set of kvSettings, and a list of secrets. Each secret will have a key and encryptedSecret field of type string, and a secretContext field that's an arbitrary set of key-value pairs corresponding to the encryption context with which the secret was encrypted.

This first version of the operator supports authenticating to Vault via the Kubernetes auth method (i.e. vaultAuthMethod: k8s), the Userpass authe method, or directly via a Vault token (i.e. vaultAuthMethod: token). Support for more authentication methods will be added in the future. Note that the configuration required for the operator to perform KMS and Vault operations is not done on the KMSVaultSecret CR but on the operator Deployment itself, and is documented below.

Configuration

AWS

As stated above, the required configuration to allow the operator to decrypt secrets should be injected via the operator Deployment manifest (as AWS_* environment variables or ~/.aws/credentials/~/.aws/config files). The aws-sdk-go library will also discover IAM roles at runtime if the operator is running on an EC2 instance with an instance role, in which case no additional configuration should be required on the operator.

Vault

In addition to the AWS configuration, Vault configuration should be injected too. The common VAULT_* environment variables will be read and used by the client. The variables that would need to be set will vary depending on your environment, but you'll typically want to at least set VAULT_ADDR, and either VAULT_CACERT/VAULT_CAPATH (pointing to a correspoinging mounted file or directory) or VAULT_SKIP_VERIFY. This is all done on the operator Deployment manifest.

The following sections document the environment variables used by each authentication method and their defaults. They are used in addition to the AWS and base Vault variables described before.

Kubernetes authentication method (vaultAuthMethod: k8s)

Environment variable	Required?	Default	Description
VAULT_K8S_ROLE	N	kms-vault-operator	The Vault role that the client should authenticate as on the Kubernetes login endpoint.
VAULT_K8S_LOGIN_ENDPOINT	N	auth/kubernetes/login	The Kubernetes authentication endpoint in Vault

Vault token authentication method (vaultAuthMethod: token)

This method simply follows the convention and uses the VAULT_TOKEN environment variable to authenticate.

Environment variable	Required?	Default	Description
VAULT_TOKEN	Y		The Vault token used to perform operations on Vault.

Vault userpass authentication method (vaultAuthMethod: userpass)

Environment variable	Required?	Default	Description
VAULT_USERNAME	Y		The Vault username used to authenticate
VAULT_PASSWORD	Y		The password corresponding to VAULT_USERNAME

Deploying the operator

The deploy/ directory has some statically defined manifests that you can modify to configure your own keys and deploy. However, this repo also provides a Helm template in the helm/kms-vault-operator directory, as well as a sample Helm values file for each authentication method above under deploy/helm-values. Please note that some of the values assume you have previously created the Kubernetes secrets or other configuration they rely on.

Tiller is not required to deploy this chart. It can be deployed by applying the charts rendered by helm template directly. For example, to deploy a chart with the deploy/helm-values/vault-token-operator-values.yaml values file, you can run:

helm template -n default -f deploy/vault-token-operator-values.yaml helm/kms-vault-operator/ | kubectl apply -f -

Creating a secret

Setting up the AWS resources required to encrypt/decrypt a KMS secret are outside of the scope of this documentation (but you can read about KMS here), so we're going to assume that the KMS key already exists and that the credentials used to encrypt the secret offline, as well as those use to decrypt it at runtime have the required permissions to perform those actions. If you're Terraform-oriented, you might find these modules useful for dealing with KMS.

From the command line, you can use the aws kms encrypt command, e.g.

aws kms encrypt --key-id <key-id-or-alias> --plaintext "Hello world" --output text --query CiphertextBlob

Then take that output and set it as an item the spec.secrets array of your KMSVaultSecret resource, e.g.

apiVersion: k8s.patoarvizu.dev/v1alpha1
kind: KMSVaultSecret
metadata:
  name: example-kmsvaultsecret
  namesace: default
spec:
  path: secret/test/kms-vault-secret
  vaultAuthMethod: <auth-method>
  kvSettings:
    engineVersion: v1
  secrets:
    - key: test
      encryptedSecret: <kms-encrypted-secret>

Make sure you also set the appropriate vaultAuthMethod based on your setup. After that, assuming the resource is in deploy/example-kms-vault-secret.yaml, just run:

kubectl apply -f deploy/example-kms-vault-secret.yaml

Partial secrets

In addition to managing KMSVaultSecret custom resources, this operator also handles a second type of resource called PartialKMSVaultSecret. This CRD is similar to KMSVaultSecret but only supports the secrets field, and doesn't have its own controller. Instead, the purpose of this resource is to hold secrets that can be included in a KMSVaultSecret, via the includeSecrets field. The single kmsvaultsecret_controller.go will aggregate the included secrets along with those of the resource itself and write them all together as a single item in Vault. To keep things as simple as possible, the first iteration of this feature won't support nesting PartialKMSVaultSecrets (e.g. by including PartialKMSVaultSecrets in other PartialKMSVaultSecrets). Rather, the way to include multiple partial secrets is to just list them all in the includeSecrets field of the KMSVaultSecret resource.

Because of their abstract nature, PartialKMSVaultSecrets don't have a path, Vault authenticating method, or KV settings, but they do support the KMS secret encryption context, which is passed down to the concrete KMSVaultSecret object.

For security nerds

Docker images are signed and published to Docker Hub's Notary server

The Notary project is a CNCF incubating project that aims to provide trust and security to software distribution. Docker Hub runs a Notary server at https://notary.docker.io for the repositories it hosts.

Docker Content Trust is the mechanism used to verify digital signatures and enforce security by adding a validating layer.

You can inspect the signed tags for this project by doing docker trust inspect --pretty docker.io/patoarvizu/kms-vault-operator, or (if you already have notary installed) notary -d ~/.docker/trust/ -s https://notary.docker.io list docker.io/patoarvizu/kms-vault-operator.

If you run docker pull with DOCKER_CONTENT_TRUST=1, the Docker client will only pull images that come from registries that have a Notary server attached (like Docker Hub).

Docker images are labeled with Git and GPG metadata

In addition to the digital validation done by Docker on the image itself, you can do your own human validation by making sure the image's content matches the Git commit information (including tags if there are any) and that the GPG signature on the commit matches the key on the commit on github.com.

For example, if you run docker pull patoarvizu/kms-vault-operator:898c158e9c3c313984d9a0c7947f0f3178501cf2 to pull the image tagged with that commit id, then run docker inspect patoarvizu/kms-vault-operator:898c158e9c3c313984d9a0c7947f0f3178501cf2 | jq -r '.[0].ContainerConfig.Labels' (assuming you have jq installed) you should see that the GIT_COMMIT label matches the tag on the image. Furthermore, if you go to https://github.com/patoarvizu/kms-vault-operator/commit/898c158e9c3c313984d9a0c7947f0f3178501cf2 (notice the matching commit id), and click on the Verified button, you should be able to confirm that the GPG key ID used to match this commit matches the value of the SIGNATURE_KEY label, and that the key belongs to the AUTHOR_EMAIL label. When an image belongs to a commit that was tagged, it'll also include a GIT_TAG label, to further validate that the image matches the content.

Keep in mind that this isn't tamper-proof. A malicious actor with access to publish images can create one with malicious content but with values for the labels matching those of a valid commit id. However, when combined with Docker Content Trust, the certainty of using a legitimate image is increased because the chances of a bad actor having both the credentials for publishing images, as well as Notary signing credentials are significantly lower and even in that scenario, compromised signing keys can be revoked or rotated.

Here's the list of included Docker labels:

• AUTHOR_EMAIL
• COMMIT_TIMESTAMP
• GIT_COMMIT
• GIT_TAG
• SIGNATURE_KEY

Important notes by this project

Kubernetes namespaces and Vault namespaces

The KMSVaultSecret CRD is a namespaced resource, but please note that the Kubernetes namespace doesn't map to a Vault namespace. Support for Vault namespaces is outside of the scope of this project, since they're only available in Vault enterprise for now.

Additionally, the PartialKMSVaultSecret CRD is also namespaced, but the includeSecrets field on a KMSVaultSecret object will only discover partial secrets within the same namespace. Support for referencing secrets from another namespace will be added in a future version.

Multiple secrets writing to the same location

Also, the operator doesn't make any guarantees or checks about KMSVaultSecrets in different namespaces writing to the same Vault paths. The operator is designed to continuously write the secret, so if two or more resources are pointing to the same location, the operator will constantly overwrite them.

No validation on target path

Because the controller is designed to write the secret to Vault continuously, it doesn't perform any validation on what may exist on the configured path before writing to it. Be careful when deploying a KMSVaultSecret to make sure you don't overwrite your existing secrets.

Removing secrets when a KMSVaultSecret is deleted.

The kms-vault-operator controller supports removing secrets from Vault by setting delete.k8s.patoarvizu.dev as a Kubernetes finalizer. Support for this for K/V V1 is simple since secrets are not versioned, but when the secret is for K/V V2, deleting a KMSVaultSecret object will delete ALL of its versions and metadata from Vault, so handle it with care. If the secret is V2, the path for the DELETE operation is the same as the input one, replacing secret/data/ with secret/metadata/. There is currently no support for removing a single version of a K/V V2 secret.

Decryption or decoding errors are ignored

If a secret is incorrectly encoded or encrypted (including if the encryption context doesn't match the secret), the operator will skip those secrets and will continue writing the rest. This applies to individual items in the secrets list, i.e. the controller will still apply other secrets within the same KMSVaultSecret even if one of them fails.

Support for K/V V2 is limited (as of this version)

The KMSVaultSecret CRD supports specifying kvSettings.engineVersion: v2 and a check-and-set index with kvSettings.casIndex but support for it is limited. For example, the operator doesn't doesn't enforce or validate that the path is V2-friendly, and no metadata operations are available.

Partial secrets don't validate keys

The operator doesn't do any validation in regards to keys included both in the KMSVaultSecret object and in any included PartialKMSVaultSecrets (or if they exist in multiple included PartialKMSVaultSecrets), and it also doesn't provide any guarantees regarding the order of precedence of secrets. Because of this, you should make sure that your secret aggregation avoids these overlaps. Support for validating this can be added in a future version.

Partial secrets don't support finalizers (yet)

Because PartialKMSVaultSecrets don't have their own controller (as of the latest version), it's not possible to handle finalizers on them (specifically the delete.k8s.patoarvizu.dev finalizer). That means that the presence of finalizers on those objects won't have the same effect as setting them on KMSVaultSecret objects. If a PartialKMSVaultSecret object is deleted, the direct effect is that any KMSVaultSecret that includes a deleted PartialKMSVaultSecret could see the included keys deleted from Vault! This would only happen if the Vault KV secret backend is v1, since to update v2 secrets you need to update the casIndex field. Supporting finalizers for partial secrets could be added to a future version.

Help wanted!

I (the author of this operator) am not a "real" golang developer and the code probably shows it. It's also my first venture into writing a Kubernetes operator. I'd welcome any Issues or PRs on this repo, including (and specially) support for additional Vault authentication methods.