safeadmin

README

Problem Statement

After seeing yet another data breach due to inadvertent SQL dumps being left on the wrong server (or at least a similar cause) the author made the following observations:

Many organizations often dump data from databases using pipelines such as:

mysqldump mydb | bzip2 -c > dump_mydb_2016-11-01.sql.bz2

And restore in similar pipelines such as:

bzcat dump_mydb_2016-11-01.sql.bz2 | mysql mydb

The business reasons for needing to do these types of operations vary, but, with the exception of genuine backups, are often to facilitate ad-hoc operations such as restoring staging/test environment to a snapshot of production etc.

Since many of these operations are ad-hoc, often various data dumps are forgotten about, and eventually over time the environments are re-purposed and data becomes increasing likely to accidentally leak.

Introducing safedump / saferestore

Add the following to the pipeline:

mysqldump mydb | bzip2 -c | safedump -for 24h > dump_mydb_2016-11-01.sql.bz2

And on the other end:

saferestore < dump_mydb_2016-11-01.sql.bz2 | bzcat | mysql mydb

safedump encrypts the data such that it can be decrypted for the specified time period only (it defaults to 1 day).

saferestore is its counterpart, that is able to decrypt the file, but only if we are inside the window specified when it was encrypted.

Quickstart - using SafeDump Public Key Server

1. Install golang, and ensure $GOPATH/bin is in your $PATH

2. Install the safedump and saferestore binaries:

   go get -u github.com/continusec/safeadmin/cmd/{safedump,saferestore}
   

3. Done! Verify it works end-to-end:

   echo Test | safedump -for 10s | saferestore
   

safedump options

Will choose server to connect to based on ~/.safedump_config - see below.

Will cache the current certificate for a server inside of ~/.safedump_cache/.

$ safedump -help
Usage of safedump:
  -chunks
    	If set, base64 and put in a chunk suitable for use in a larger file such as a log
  -for duration
    	Duration for which this data may be restored (default 24h0m0s)

saferestore options

Will choose server to connect to based on ~/.safedump_config - see below.

$ saferestore -help
Usage of saferestore:
  -chunks
    	If set, look for chunks and decode them rather than entire file
  -skip-local-check
    	If set, skip client date check and send to server anyway

Note: -skip-local-check is a "feel-good" option to help verify that the server will reject a decryption request for an object that is past TTL (without setting this, the client won't even try to contact the server in such a case).

Protocol details

For safedump and saferestore to operate, they require connectivity to a SafeDump Key Server that supports the following two functions.

(By default, a public SafeDump Key Server is used. It is recommended to run your own Key Server, an implentation is provided)

Get Current Certificate

Returns a short-lived self-signed certificate that includes an RSA public-key and a validity period. The validity period (a typical validity period may be from now until 7 days in the future) signifies what time period the server will accept decryption requests for objects encrypted with this key.

It additionally returns the time when the server will next generate a new certificate. Typical clients will cache the current certificate until this time.

When safedump runs, it checks to see if it has a currently valid certificate in ~/.safedump_cache/ for the configured server, which allows encryption up to the date specified by the invocation, and if not it contacts the configured server (per ~/.safedump_config, defaulting to the public key server) to fetch an updated certificate.

Once a valid certificate is found, safedump then generates locally a once-off 256-bit AES key, and encrypts this using OAEP with the public key from the certificate using the requested TTL (now + 24 hours in our example) as the label input.

A header including the SPKI SHA256 fingerprint of the certificate that contained the public key used to encrypt the private key, the TTL and the OAEP result is written to stdout, followed by the rest of stdin encrypted using AES CTR with the generated AES key.

Decrypt Private Key

Takes as input the header with the SPKI fingerprint, TTL and encrypted OAEP result. The server validates that the current time is before the TTL, and if so, proceeds to load the private key associated with the SPKI fingerprint, checks that the TTL is within the validity period for the associated certificate, and if so, decrypts the OAEP data (with the TTL encoded as a label). The decrypted key is returned to the client.

When saferestore runs, it reads from stdin the header as written by safedump, sends this header only to the server as configured in ~/.safedump_config (defaulting to the public key server, gets the private key used to encrypt back, and then uses this to decrypt the data stream which it writes to stdout.

Running your own server

Two server implementations are provided. The recommended option for private servers is to use the gRPC server.

gRPC Server (recommended)

The gRPC server implements the SafeDump protocol. It accepts connections over the gRPC protocol, and writes certificates/keys to a specified directory.

Unlike the public key server, the gRPC server does not automatically purge old private keys. This means that administrators with direct access to this directory can choose to override the TTL specified in encrypted files by using the breakglassrestore tool.

Starting the server (servesafedump)

1. Grab the binary:

   go get -u github.com/continusec/safeadmin/cmd/servesafedump
   

2. Create a server configuration file:

   # Address (host:port) to listen for gRPC requests on (HTTP/2).
   listen_bind: ":10001"
   
   # tcp4 or tcp6
   listen_protocol: "tcp4"
   
   # TLS cert / key to use for gRPC server. The following will generate a self-signed certificate:
   # openssl req -x509 -newkey rsa:4096 -keyout grpc-key.pem -out grpc-cert.pem -days 3600 -nodes -subj '/CN=localhost' -batch
   server_cert_path: "grpc-cert.pem"
   server_key_path: "grpc-key.pem"
   
   # Where keys and certificates will be stored:
   archived_keys_dir: "/path/to/archived_keys_dir"
   
   # What is the maximum duration that a client can encrypt data for?
   # Note that due to certificate rotation periods, the actual max for a client will be between this and this + the rotation period)
   max_decryption_period: "168h"
   
   # How often will the server rotate the current certificate?
   certificate_rotation_period: "24h"
   
   # Should the server purge old keys?
   purge_old_keys: true
   
   # How long, after the normal key expiration, should old keys be kept for breakglass usage?
   key_retention_period: "720h" # 30 days
   

3. Start the server:

   servesafedump /path/to/config
   

4. Remember to configure your clients to use this server (by populating ~/.safedump_config).

Out-of-band decryption (breakglassrestore)

Sometimes it may become necessary for an administrator to decrypt a file that the server refuses to decrypt (e.g. the TTL has passed for a file).

Use of this tool requires direct server access, specifically read access to the directory that servesafedump is configured to write certificates to.

To use this tool (which, unlike the client tools, ignores any client configuration):

breakglassrestore -keys /path/to/archived_keys_dir < encryptedfile

Google App Engine server (used by Public Key Server)

The Google App Engine server is very similar to the gRPC server, however instead of persisting data to a filesystem directory, it writes to Google Cloud Datastore. It is configured to purge old keys, and to do so with no retention period, meaning it is not possible to provide breakglassrestore access to data encrypted using this server.

The following assumes the SDK for Google App Engine (Go) has already been downloaded and installed.

To run the local Google App Engine development server:

goapp serve cmd/gaesafedumpserver

To deploy the GAE app:

goapp deploy cmd/gaesafedumpserver

Client Configuration

(Skip this section unless you are running your own server)

The client tools will search for configuration in the following order:

1. If ~/.safedump_config exists, then this file is used.
2. If /etc/safedump_config exists, then this file is used.
3. Fall back to default configuration, which is to use the SafeDump Public Key Server.

The configuration file format is a text protobuf format, and should contain the following content:

gRPC protocol

For gRPC protocol (recommended for private servers):

protocol: GRPC_PROTOCOL

# host:port of grpc server
grpc_server: "safedump.example.com:10001"

# One of the following 3 options should be used:

# Option 1: If your gRPC certificate is signed by a CA trusted by the clients operating system, set this:
use_system_ca_for_grpc: true

# Option 2: If using a self-signed gRPC certificate, then include the full PEM encoded certificate for the gRPC server here
grpc_cert: "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----\n"

# Option 3: Disable host validation. Should only be used by lazy debuggers.
no_grpc_security: true

HTTP(S) protocol

For HTTP(S) protocol (as currently used by the SafeDump Public Key Server) you need the following configuration:

protocol: HTTP_PROTOCOL

# includes protocol, no trailing slash
http_base_url: "https://safedump-public-key-server.appspot.com"

Note that if no configuration file is found, the above values are used by default.

Side-note: Why two transport protocols?

gRPC is the preferred transport protocol, however support for gRPC within Google App Engine (which currently hosts the SafeDump Public Key Server) does not appear stable yet. Hence we include a simpler HTTP(S) protocol, even though we consider it born deprecated.

Developer Notes

To rebuild the protobuf files:

rm -rf pb
mkdir pb
protoc --go_out=plugins=grpc:pb safedump.proto

To rebuild all non-GAE binaries and compile the GAE code:

go install github.com/continusec/safeadmin/cmd/{safedump,saferestore,servesafedump,breakglassrestore,gaesafedumpserver}

FAQ

How does safedump and saferestore authenticate to the server?

They don't. The aim of this tool is to be simple to roll out, and an improvement on the status quo (unencrypted files on disk). It is not intended to be the only security mechanism for protecting dump files.

How can I decrypt a file after the expiry time?

When using the Public Key Server, this is not possible. If you are running your own private key server, and have direct access to the server, use the breakglassrestore tool (provided that the keys are not yet been purged).

Is all my data being sent into the cloud?

No. Even when using the public key server, the only data sent to the public key server is the encrypted header, which the SPKI hash of the certificate used to encrypt the file, the expiration date for the encryption on the file, and the encrypted key that was generated for this file (total a couple of KB). The server decryptes the encrypted key and returns it, allowing the saferestore tool to decrypt the remainder of the file.

Should I run my own private key server?

Yes. In most instances you will want to do so, as it provides lower latency, and means that an someone with receipt of an encrypted file will also need access to your private key server in order to decrypt it. It also allows your administrators the ability to recover encrypted files that are past their TTL, but prior to key purging, which, by design, the public key server will not do.

The public key server is primarily intended for demonstrative purposes and we make no guarantees or warranties about any aspect of its operation.

What files can I clean up?

safedump will write cached public certificates to the ~/.safedump_cache/ directory. If you only over connect to one server, then there should be only one small file in there. This directory can be safely deleted at any time, however is retained for convenience, because if a valid certificate is found there, running safedump can be achieved without needing to contact the server at all.

Contact information

If you find these tools useful, please drop us a note telling us that you're using it.

Feedback and pull-requests welcome!

Adam Eijdenberg adam@continusec.com

Documentation

Index

• Variables
• type FilesystemPersistence
  • func (f *FilesystemPersistence) Load(ctx context.Context, key []byte) ([]byte, error)
  • func (f *FilesystemPersistence) Purge(ctx context.Context, now time.Time) error
  • func (f *FilesystemPersistence) Save(ctx context.Context, key, value []byte, ttl time.Time) error
• type SafeDumpClient
  • func CreateClientFromConfiguration() (*SafeDumpClient, error)
  • func (c *SafeDumpClient) Close() error
  • func (c *SafeDumpClient) DecryptWithTTL(in io.Reader, out io.Writer, chunks bool) error
  • func (c *SafeDumpClient) EncryptWithTTL(ttl time.Time, in io.Reader, out io.Writer, chunk bool) error
• type SafeDumpPersistence
• type SafeDumpServer
  • func (s *SafeDumpServer) BeginPurgeCron()
  • func (s *SafeDumpServer) Close() error
  • func (s *SafeDumpServer) CronPurge(ctx context.Context) error
  • func (s *SafeDumpServer) DecryptSecret(ctx context.Context, req *pb.DecryptSecretRequest) (*pb.DecryptSecretResponse, error)
  • func (s *SafeDumpServer) GetPublicCert(ctx context.Context, req *pb.GetPublicCertRequest) (*pb.GetPublicCertResponse, error)
  • func (s *SafeDumpServer) SourceName() string
• type SafeDumpServiceClient

Variables

var (
	// ErrStorageKeyNotFound returned when object not found in storage
	ErrStorageKeyNotFound = errors.New("ErrStorageKeyNotFound")

	// ErrInvalidDate returned when the date is a reason why we won't decrypt
	ErrInvalidDate = errors.New("ErrInvalidDate")

	// ErrInvalidRequest returned when an invalid request is received.
	ErrInvalidRequest = errors.New("ErrInvalidRequest")

	// ErrInternalError means that an unexpected error occurred.
	ErrInternalError = errors.New("ErrInternalError")

	// ErrInvalidConfig means the configuration is not supported.
	ErrInvalidConfig = errors.New("ErrInvalidConfig")
)

Types

type FilesystemPersistence

type FilesystemPersistence struct {
	// Dir is a path to the directory that contains the data
	Dir string

	// Immutable means be read-only
	Immutable bool
}

FilesystemPersistence write key/values to the specified directory.

func (*FilesystemPersistence) Load

func (f *FilesystemPersistence) Load(ctx context.Context, key []byte) ([]byte, error)

Load returns value if found, nil otherwise. It should ignore the TTL

func (*FilesystemPersistence) Purge

func (f *FilesystemPersistence) Purge(ctx context.Context, now time.Time) error

Purge removes data whose TTL is older than now

func (*FilesystemPersistence) Save

func (f *FilesystemPersistence) Save(ctx context.Context, key, value []byte, ttl time.Time) error

Save sets value, as atomically as we can. File will be saved with default umask This persistence layer saves the TTL as a prefix to each file

type SafeDumpClient

type SafeDumpClient struct {
	// Server is the server that we talk to
	Server SafeDumpServiceClient

	// Storage is used to cache public certs
	Storage SafeDumpPersistence

	// SendKnownBadDateToServer allows us to request decryption of files that we know have the wrong TTL. Useful for breakglass utilities.
	SendKnownBadDateToServer bool
}

SafeDumpClient is used to interact with a given server

func CreateClientFromConfiguration

func CreateClientFromConfiguration() (*SafeDumpClient, error)

CreateClientFromConfiguration loads the client configuration file from the standard location, "~/.safedump_config". If no file exists at that location, it will fall back to "/etc/safedump_config", and it that does not exist, we fall back to using a public key server

func (*SafeDumpClient) Close

func (c *SafeDumpClient) Close() error

Close should be called to close all underlying connections

func (*SafeDumpClient) DecryptWithTTL

func (c *SafeDumpClient) DecryptWithTTL(in io.Reader, out io.Writer, chunks bool) error

DecryptWithTTL reads an encrypted header, then decrypts the stream, copying to out. Note that since we don't MAC, we make no guarantees about integrity or authentication. If chunks is set, then look for chunks to decode as part of a larger file, instead of the whole stream. For now, if chunks is set, then the entire input stream is read before processing commences.

func (*SafeDumpClient) EncryptWithTTL

func (c *SafeDumpClient) EncryptWithTTL(ttl time.Time, in io.Reader, out io.Writer, chunk bool) error

EncryptWithTTL will generate a symmetric key, then encrypt this using the specified public key and TTL, write this out, and then apply a stream cipher to in, copying to out. The number of bytes written to out will be the same as those written to in with a small constant number of bytes added to it. If chunk is set, then output is suitable for embedded in a larger file (e.g. log file)

type SafeDumpPersistence

type SafeDumpPersistence interface {
	// Load returns value if found, nil otherwise
	Load(ctx context.Context, key []byte) ([]byte, error)

	// Save sets value
	// The TTL is a suggestion - and is used for the cleanup function
	// It should not affect loads.
	Save(ctx context.Context, key, value []byte, ttl time.Time) error

	// Delete all content with TTL before this time.
	Purge(ctx context.Context, now time.Time) error
}

SafeDumpPersistence is an abstraction for a persistence layer

type SafeDumpServer

type SafeDumpServer struct {
	// Storage is a dumb layer that can store and load stuff
	Storage SafeDumpPersistence

	// MaxDecryptionPeriod is the maximum length of time the server will commit to being able to decrypt an object encrypted with it's certicates
	MaxDecryptionPeriod time.Duration

	// CertificationRotationPeriod is how often a fresh certificate is issued
	CertificateRotationPeriod time.Duration

	// OverrideDateChecks, if set, will skip date checks on TTL. This should only be used with breakglass tools that operate on the server directly
	OverrideDateChecks bool

	// PurgeOldKeys deletes old keys automatically (assuming cron is started)
	PurgeOldKeys bool

	// KeyRetentionPeriod is the period after a key expires that it will be kept anyway
	// Only used if PurgeOldKeys is set
	KeyRetentionPeriod time.Duration
	// contains filtered or unexported fields
}

SafeDumpServer is the main server object that can handle the business logic, for the defined gRPC service, regardless of actually protocol and persistence layer

func (*SafeDumpServer) BeginPurgeCron

func (s *SafeDumpServer) BeginPurgeCron()

BeginPurgeCron will start go funcs the purge cron job We run as often as the CertificateRotationPeriod

func (*SafeDumpServer) Close

func (s *SafeDumpServer) Close() error

Close is a no-op for the server, but is required per the interface definition

func (*SafeDumpServer) CronPurge

func (s *SafeDumpServer) CronPurge(ctx context.Context) error

CronPurge should be called regularly

func (*SafeDumpServer) DecryptSecret

func (s *SafeDumpServer) DecryptSecret(ctx context.Context, req *pb.DecryptSecretRequest) (*pb.DecryptSecretResponse, error)

DecryptSecret finds the associated encryption key, and if all matches, decrypts it. Errors returned are ErrInvalidRequest, ErrInvalidDate, ErrInternalError, nil

func (*SafeDumpServer) GetPublicCert

func (s *SafeDumpServer) GetPublicCert(ctx context.Context, req *pb.GetPublicCertRequest) (*pb.GetPublicCertResponse, error)

GetPublicCert is part of the service definition, it returns the current public certificate, rotating if necessary

func (*SafeDumpServer) SourceName

func (s *SafeDumpServer) SourceName() string

SourceName is used to key cache data. Ought not be called in this context

type SafeDumpServiceClient

type SafeDumpServiceClient interface {
	pb.SafeDumpServiceServer
	io.Closer

	// SourceName returns the name of the serving we are connected to. Used to key cached certs.
	SourceName() string
}

SafeDumpServiceClient is to combine a Closer with a Server (which is bascially the same as client)