README ΒΆ
confidential (working title)
Export parameters from AWS Systems Manager Parameters as environment variables.
See some examples of common use cases.
Why I wrote this?
Configuration management, specifically secrets management tends to get complicated. After having been through several projects, both in my spare time and at work, using solutions such as Ansible Vault, private AWS CodeCommit repositories or Amazon KMS encrypted configuration files in Amazon S3, I was looking for something simpler, while still maintaining a high level of security.
I deemed self-hosted solution, such as Hashicorp Vault (and the other solutions listed on this Hashicorp Vault vs. Other Software page) too time consuming to set up and maintain.
Luckily, Amazon Web Services have been busy improving their Amazon EC2 Systems Manager Parameter Store in 2017 and it now supports both seamless Amazon KMS encryption and versioning of parameters.
Getting Started
These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.
Prerequisites
Confidential is written in Go and can be run as a single binary, no language specific requirements are needed.
It is designed to run on the GNU/Linux, macOS, and Windows operating systems. Other operating systems will probably work as long as you can compile a Go binary on them.
Installing
Make sure you have Go installed and that the $GOPATH
is set correctly.
Build binary
go get github.com/nlindblad/confidential/apps/confidential
cd $GOPATH/src/github.com/nlindblad/confidential/apps/confidential
go build
Or if you have cloned this repository:
make PLATFORM
where PLATFORM
is one of linux
, darwin
or windows
.
Run
./confidential --help
And you should see:
NAME:
confidential - Export parameters from AWS Systems Manager Parameters as environment variables
USAGE:
confidential [global options] command [command options] [arguments...]
VERSION:
0.0.0
AUTHOR:
Niklas Lindblad <niklas@lindblad.info>
COMMANDS:
exec, e retrieve environment variables and execute command with an updated environment
output, o retrieve and atomically output environment variables to a file
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--forwarded-profile value AWS profile to forward credentials for in the created environment [$AWS_FORWARDED_PROFILE]
--prefix value Amazon SSM parameter prefix
--profile value AWS profile to use when calling Amazon SSM [$AWS_PROFILE]
--region value AWS region e.g. eu-west-1 [$AWS_REGION]
--help, -h show help
--version, -v print the version
Running the tests
Tests use the excellent Go stretchr/testify package.
go test -v ./...
Deployment
TODO
Examples
The machine needs to have the following AWS IAM permissions:
kms:Decrypt
on the relevant Amazon KMS key used to encrypt sensitive parameters.ssm:GetParametersByPath
on the relevant resource:arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/<PREFIX>
(note there should be no trailing slash or wildcards)
AWS Systems Manager Parameters names are translated into environment variable compatible names using the following logic:
- All disallowed characters are omitted. Allowed characters are
-
,.
,_
,a-z
,A-Z
and0-9
. - Any non-alphanumerical character (not
a-z
,A-Z
or0-9
) is converted to an underscore (_
).
For example:
/my-prefix/database.password
becomes DATABASE_PASSWORD
(as would /my-prefix/database/password
).
/my-prefix/bar
simply becomes BAR
π Use with Docker and systemd services:
A handy way of running Docker containers supervised by systemd is to create a unit (service) using the systemd-docker
wrapper:
[Unit]
Description=My service
Requires=docker.service
After=docker.service
[Service]
TimeoutStartSec=0
ExecStartPre=/usr/local/bin/confidential --region eu-west-1 --prefix /my-service/prod output --env-file /etc/my-service/prod.env
ExecStartPre=/usr/bin/docker pull username/image-name:latest
ExecStart=/usr/local/bin/systemd-docker --cgroups name=systemd run \
--name %n \
--env-file /etc/my-service/prod.env \
... Add other Docker run flags here ...
-d username/image-name:latest
ExecStop=/usr/bin/docker stop %n
ExecStopPost=/usr/bin/docker rm -f %n
Restart=always
RestartSec=10s
Type=notify
NotifyAccess=all
[Install]
WantedBy=default.target
The following service will run username/image-name
as a service which will get restarted if it falls over.
Every time the service is started/restarted, it runs the two ExecStartPre
steps:
-
Uses confidential to get the latest environment variables from AWS Systems Manager Parameters in the
eu-west-1
AWS Region and writes them to the file/etc/my-service/prod.env
in a format that Docker understands -
Pulls down the latest version of the
username/image-name
Docker image
This ensures that the service is always running using the latest published Docker image and that any configuration changes are picked up automatically.
Managing the environment variables for the service is now done within the /my-service/prod
namespace in AWS Systems Manager Parameters.
π΄ Use with generic systemd services:
The EnvironmentFile
directive can be used to expose the retrieved environment variable to any kind of executable running as a systemd service:
[Unit]
Description=Service
After=syslog.target network.target remote-fs.target nss-lookup.target
[Service]
Type=forking
PIDFile=/run/my-service.pid
ExecStartPre=/usr/local/bin/confidential --region eu-west-1 --prefix /my-service/prod output --env-file /etc/my-service/prod.env
EnvironmentFile=-/etc/my-service/prod.env
ExecStart=/usr/local/bin/my-service --flag=something --foo=bar
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true
[Install]
WantedBy=multi-user.target
Every time the service is started/restarted, it runs the ExecStartPre
steps and populates /etc/my-service/prod.env
and includes it using the EnvironmentFile
directive (note the -
before the filename).
Managing the environment variables for the service is now done within the /my-service/prod
namespace in AWS Systems Manager Parameters in the eu-west-1
AWS Region.
β Give EC2 hosts permissions to access specific parameters:
See full CloudFormation template: examples/cloudformation/example-3-cloudformation.yml
π Create an IAM role with permissions to access specific parameters:
See full CloudFormation template: examples/cloudformation/example-4-cloudformation.yml
Creates a dedicated IAM user and access keys that is allowed to decrypt and retrieve parameters with a specific prefix.
Note: Some other tools using AWS Systems Manager Parameters use a mix of ssm:DescribeParameters
and ssm:GetParameters
, which makes it hard to create fine grained acess control, especially when iterating parameters requires permissions to describe all parameters.
β Create an IAM role with permissions to set specific parameters:
See full CloudFormation template: examples/cloudformation/example-5-cloudformation.yml
Creates a dedicated IAM user and access keys that is allowed to encrypt and set parameters with a specific prefix, but not retrieve or decrypt.
Example usage:
aws --profile <PROFILE> ssm put-parameter --name '<PREFIX>/<PARAMETER NAME>' --type "SecureString" --value '<VALUE>'
π Run arbitrary executable with an environment populated by confidential:
The simplest example of this is running the /usr/bin/env
utility and print out the environment variables that are accessible to the newly invoked process:
/usr/local/bin/confidential --region eu-west-1 --prefix /my-service/prod exec -- env
π Use with supervisord:
[program:my-service]
command=/usr/local/bin/confidential --region eu-west-1 --prefix /my-service/prod exec -- /usr/local/bin/my-service --flag=something --foo=bar
directory=/tmp
autostart=true
autorestart=true
startretries=3
stdout_logfile=/tmp/my-service.log
stderr_logfile=/tmp/my-service.err.log
user=username
π Use specific AWS profile from ~/.aws/credentials
By default, the AWS SDK for Go will automatically look for AWS credentials in a couple of pre-defined places.
If you are using the standard ~/.aws/credentials
(used by the standard AWS CLI tool), you can specify multiple sections with different credentials:
[default]
aws_access_key_id = AKIAPEIPJKJSOJ267
aws_secret_access_key = XXXXXXXXXXXXXXXXXXX
[parameters-read]
aws_access_key_id = AKIABCDEFGH12345
aws_secret_access_key = XXXXXXXXXXXXXXXXXXX
Using the --profile
flag, you can specify that you want to use the parameters-read
profile instead of the default
one (which would get picked up by the AWS SDK for Go):
/usr/local/bin/confidential --profile parameters-read --region eu-west-1 --prefix /my-service/prod output --env-file /etc/my-service/prod.env
β© Forward AWS credentials from ~/.aws/credentials
to new environment
It is possible to forward AWS credentials from ~/.aws/credentials
for a given profile to the new enviromment using the --forwarded-profile
flag.
Given a ~/.aws/credentials
file:
[default]
aws_access_key_id = AKIAPEIPJKJSOJ267
aws_secret_access_key = XXXXXXXXXXXXXXXXXXX
[parameters-read]
aws_access_key_id = AKIABCDEFGH12345
aws_secret_access_key = XXXXXXXXXXXXXXXXXXX
[my-service]
aws_access_key_id = AKIAHIHIIW233445
aws_secret_access_key = XXXXXXXXXXXXXXXXXXX
You can use the the AWS credentials for the parameters-read
profile to retrieve the parameters from AWS Systems Manager Parameters and forward the AWS credentials for the my-service
profile using:
/usr/local/bin/confidential --profile parameters-read --forwarded-profile my-service --region eu-west-1 --prefix /my-service/prod output --env-file /etc/my-service/prod.env
In the above example, /etc/my-service/prod.env
would contain all parameters retrieved from AWS Systems Manager Parameters in the eu-west-1
AWS Region in addition to:
AWS_ACCESS_KEY_ID=AKIAHIHIIW233445
AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXX
AWS_SESSION_TOKEN=
Built With
- aws/aws-sdk-go - AWS SDK for Go
- urfave/cli - Command line interface helpers
- dchest/safefile - Safe "atomic" saving of files
Versioning
Uses SemVer for versioning. For the versions available, see the tags on this repository.
Authors
- Niklas Lindblad - Initial work
License
This project is licensed under the MIT License - see the LICENSE.md file for details
Acknowledgments
- Sjeanpierre/param_api provided a great starting point for using the Amazon SSM API in Go
- gurusi/systemd-make-environment for mentioning some common gotchas with
EnvironmentFile
andExecStartPre
withsystemd
- Storing Secrets with AWS ParameterStore provided a great starting point for the CloudFormation templates
- segmentio/chamber for a nice way of implementing the
exec
command