ecfg

README

ecfg

ecfg is nearly complete, but there are still some bugs. None of the known ones will compromise your data, as the encryption stuff is all basically identical to ejson. However, it might write out improperly formatted data, fail completely, etc. We originally intended to replace ejson with ecfg, but ended up never getting around to it. Caveat Implementor

ecfg is a utility for managing a collection of secrets, generally with the intent of committing them to source control. The secrets are encrypted using public key, elliptic curve cryptography (NaCl Box: Curve25519 + Salsa20 + Poly1305-AES). Secrets are collected in a JSON, YAML, or TOML file, in which all the string values are encrypted. Public keys are embedded in the file, and the decrypter looks up the corresponding private key from its local filesystem or the process environment.

The main benefits provided by ecfg are:

• Secrets can be safely stored in a git repo.
• Changes to secrets are auditable on a line-by-line basis with git blame.
• Anyone with git commit access has access to write new secrets.
• Decryption access can easily be locked down to production servers only.
• Secrets change synchronously with application source (as opposed to secrets provisioned by Configuration Management).
• Simple, well-tested, easily-auditable source.

See the manpages for more technical documentation.

ecfg is written in lowercase, and normally pronounced "e-config", like "email".

Differences from EJSON 1.0

• Supports YAML and TOML
• Expected filename changes from *.ejson to *.ecfg.<format> (e.g. .ecfg.json)
• encrypt and decrypt can now both receive data on stdin, and will emit output to stdout in this case.
• Added --type/-t flag to encrypt and decrypt commands. If the filename is *{json,yaml,yml,toml}, this flag is optional, but when the type can't be inferred because of a different filename or when reading from stdin, it must be specified. (note that this recognizes *.ejson as json automatically)
• All references to EJSON have changed to ECFG, including ECFG_KEYDIR.
• Key lookup has been changed to allow a greater degree of control:
• ECFG_PRIVATE_KEY may be set as an environment variable. This can be used to preempt key selection logic during decrypt. If present, ecfg will blindly attempt to decrypt any encrypted values using this key, instead of trying to find the matching key in the keydir. This is useful for deploying to Heroku.
• Unless ECFG_KEYDIR is set, ecfg will now be look in four directories (in order of decreasing precedence):
  • $XDG_CONFIG_HOME/ecfg/keys (if set, and non-root)
  • $HOME/.ecfg/keys (if non-root)
  • /etc/ecfg/keys
  • /opt/ejson/keys
• If ECFG_KEYDIR is set, then ecfg will look in $ECFG_KEYDIR/keys
• Overhauled build process

Installation

You can download the .deb package from Github Releases.

On macOS, you can install ecfg using brew install shopify/shopify/ecfg.

You can also install ecfg as a gem using gem install ecfg or adding it to your Gemfile.

Workflow

1: Create the Keydir

By default, ecfg looks for keys in /etc/ecfg/keys. You can change this by setting ECFG_KEYDIR or passing the --keydir option.

$ mkdir -p /etc/ecfg/keys

2: Generate a keypair

When called with -w, ecfg keygen will write the keypair into the keydir and print the public key. Without -w, it will print both keys to stdout. This is useful if you have to distribute the key to multiple servers via configuration management, etc.

ecfg keygen will write the keypair to the first directory on the keypath that's writable:

1. $XDG_CONFIG_HOME/ecfg/keys (if non-root and $XDG_CONFIG_HOME is set)
2. $HOME/.ecfg/keys (if non-root)
3. /etc/ecfg/keys
4. /opt/ejson/keys

$ ecfg keygen
Public Key:
63ccf05a9492e68e12eeb1c705888aebdcc0080af7e594fc402beb24cce9d14f
Private Key:
75b80b4a693156eb435f4ed2fe397e583f461f09fd99ec2bd1bdef0a56cf6e64

$ ecfg keygen -w
53393332c6c7c474af603c078f5696c8fe16677a09a711bba299a6c1c1676a59
$ cat ~/.ecfg/keys/5339*
888a4291bef9135729357b8c70e5a62b0bbe104a679d829cdbe56d46a4481aaf

$ sudo ecfg keygen -w
53393332c6c7c474af603c078f5696c8fe16677a09a711bba299a6c1c1676a59
$ cat /etc/ecfg/keys/5339*
888a4291bef9135729357b8c70e5a62b0bbe104a679d829cdbe56d46a4481aaf

3: Create an ecfg file

The format is described in more detail later on. For now, create a file that looks something like this. Fill in the <key> with whatever you got back in step 2.

Create this file as test.ecfg.json:

{
  "_public_key": "<key>",
  "database_password": "1234password"
}

4: Encrypt the file

Running ecfg encrypt test.ecfg.json will encrypt any new plaintext keys in the file, and leave any existing encrypted keys untouched:

{
  "_public_key": "63ccf05a9492e68e12eeb1c705888aebdcc0080af7e594fc402beb24cce9d14f",
  "database_password": "EJ[1:WGj2t4znULHT1IRveMEdvvNXqZzNBNMsJ5iZVy6Dvxs=:kA6ekF8ViYR5ZLeSmMXWsdLfWr7wn9qS:fcHQtdt6nqcNOXa97/M278RX6w==]"
}

Try adding another plaintext secret to the file and run ecfg encrypt test.ecfg.json again. The database_password field will not be changed, but the new secret will be encrypted.

5: Decrypt the file

To decrypt the file, you must have a file present in the keydir whose name is the 64-byte hex-encoded public key exactly as embedded in the ecfg document. The contents of that file must be the similarly-encoded private key. If you used ecfg keygen -w, you've already got this covered.

Alternatively, in some environments, it may be easier to pass the private key via ECFG_PRIVATE_KEY, which preempts the keydir lookup.

Unlike ecfg encrypt, which overwrites the specified files, ecfg decrypt only takes one file parameter, and prints the output to stdout:

$ ecfg decrypt foo.ecfg.json
{
  "_public_key": "63ccf05a9492e68e12eeb1c705888aebdcc0080af7e594fc402beb24cce9d14f",
  "database_password": "1234password"
}

Format

The ecfg.json document format is simple, but there are a few points to be aware of:

1. It's just JSON (or YAML or TOML, in the case of ecfg.yaml and ecfg.toml)
2. There must be a key at the top level named _public_key, whose value is a 32-byte hex-encoded (i.e. 64 ASCII byte) public key as generated by ecfg keygen.
3. Any string literal that isn't an object key will be encrypted by default (ie. in {"a": "b"}, "b" will be encrypted, but "a" will not.
4. Non-string data types aren't encrypted in json or toml, but are in yaml, simply because it makes the transformer code simpler.
5. If a key begins with an underscore, its corresponding value will not be encrypted. This is used to prevent the _public_key field from being encrypted, and is useful for implementing metadata schemes.
6. Underscores do not propagate downward. For example, in {"_a": {"b": "c"}}, "c" will be encrypted.

Building ecfg

If you work at Shopify, just run dev up && dev build; otherwise:

1. Install ruby (the system one will do on OS X, or you can brew install ruby. hpricot doesn't seem to want to build with 2.3.x)
2. Install bundler (gem install bundler)
3. bundle install
4. Install Go (brew install go)
5. Configure your $GOPATH and make sure this is on it at github.com/Shopify/ecfg.
6. make

Documentation

Overview

Package ecfg implements the primary interface to interact with ecfg documents and keypairs. The CLI implemented by cmd/ecfg is a fairly thin wrapper around this package.

Index

• Constants
• func DecryptData(data []byte, keypath []string, fileType FileType) ([]byte, error)
• func DecryptFile(filePath string, keypath []string, fileType FileType) ([]byte, error)
• func DefaultKeypath() (keypath []string)
• func EncryptData(data []byte, fileType FileType) ([]byte, error)
• func EncryptFileInPlace(filePath string, fileType FileType) (int, error)
• func GenerateKeypair() (pub string, priv string, err error)
• func SystemKeypath() (keypath []string)
• func UserKeypath() (keypath []string)
• type FileType

Constants

const (
	FileTypeJSON = iota
	FileTypeYAML
	FileTypeTOML
)

Functions

func DecryptData

func DecryptData(data []byte, keypath []string, fileType FileType) ([]byte, error)

DecryptData takes a an encrypted ecfg document and returns the same document, decrypted. The public key used to encrypt the values is embedded in the document, and the matching private key is searched for in keypath. There must exist a file in at least one of the keypath entries whose name is the public key from the ecfg document, and whose contents are the corresponding private key. See README.md for more details on this.

func DecryptFile

func DecryptFile(filePath string, keypath []string, fileType FileType) ([]byte, error)

DecryptFile takes a path to an encrypted ecfg file and returns the data decrypted. The public key used to encrypt the values is embedded in the referenced document, and the matching private key is searched for in keypath. There must exist a file in at least one of the keypath entries whose name is the public key from the ecfg document, and whose contents are the corresponding private key. See README.md for more details on this.

func DefaultKeypath

func DefaultKeypath() (keypath []string)

DefaultKeypath is UserKeypath prefixed to SystemKeypath. For root, this will be equal to SystemKeypath, and for other users, this will cause key lookups to first try their own local keys, falling back to system keys if that fails.

func EncryptData

func EncryptData(data []byte, fileType FileType) ([]byte, error)

func EncryptFileInPlace

func EncryptFileInPlace(filePath string, fileType FileType) (int, error)

EncryptFileInPlace takes a path to a file on disk, which must be a valid ecfg file (see README.md for more on what constitutes a valid ecfg file). Any encryptable-but-unencrypted fields in the file will be encrypted using the public key embdded in the file, and the resulting text will be written over the file present on disk.

func GenerateKeypair

func GenerateKeypair() (pub string, priv string, err error)

GenerateKeypair is used to create a new ecfg keypair. It returns the keys as hex-encoded strings, suitable for printing to the screen. hex.DecodeString can be used to load the true representation if necessary.

func SystemKeypath

func SystemKeypath() (keypath []string)

SystemKeypath returns the default system-wide locations at which to search for ecfg keys. /opt/ejson/keys is provided for backwards-compatibility with ejson.

func UserKeypath

func UserKeypath() (keypath []string)

UserKeypath returns the user-specific locations at which to search for ecfg keys. In most cases, this is empty for root, and ~/.ecfg/keys in other cases. If XDG_CONFIG_HOME is set, $XDG_CONFIG_HOME/ecfg/keys is highest priority.

Types

type FileType

type FileType int