sio

README

Secure IO

Go implementation of the Data At Rest Encryption (DARE) format.

Introduction

It is a common problem to store data securely - especially on untrusted remote storage. One solution to this problem is cryptography. Before data is stored it is encrypted to ensure that the data is confidential. Unfortunately encrypting data is not enough to prevent more sophisticated attacks. Anyone who has access to the stored data can try to manipulate the data - even if the data is encrypted.

To prevent these kinds of attacks the data must be encrypted in a tamper-resistant way. This means an attacker should not be able to:

• Read the stored data - this is achieved by modern encryption algorithms.
• Modify the data by changing parts of the encrypted data.
• Rearrange or reorder parts of the encrypted data.

Authenticated encryption schemes (AE) - like AES-GCM or ChaCha20-Poly1305 - encrypt and authenticate data. Any modification to the encrypted data (ciphertext) is detected while decrypting the data. But even an AE scheme alone is not sufficiently enough to prevent all kinds of data manipulation.

All modern AE schemes produce an authentication tag which is verified after the ciphertext is decrypted. If a large amount of data is decrypted it is not always possible to buffer all decrypted data until the authentication tag is verified. Returning unauthenticated data has the same issues like encrypting data without authentication.

Splitting the data into small chunks fixes the problem of deferred authentication checks but introduces a new one. The chunks can be reordered - e.g. exchanging chunk 1 and 2 - because every chunk is encrypted separately. Therefore the order of the chunks must be encoded somehow into the chunks itself to be able to detect rearranging any number of chunks.

This project specifies a format for en/decrypting an arbitrary data stream and gives some recommendations about how to use and implement data at rest encryption (DARE). Additionally this project provides a reference implementation in Go.

Applications

DARE is designed with simplicity and efficiency in mind. It combines modern AE schemes with a very simple reorder protection mechanism to build a tamper-resistant encryption scheme. DARE can be used to encrypt files, backups and even large object storage systems.

Its main properties are:

• Security and high performance by relying on modern AEAD ciphers
• Small overhead - encryption increases the amount of data by ~0.05%
• Support for long data streams - up to 256 TB under the same key
• Random access - arbitrary sequences / ranges can be decrypted independently

Install: go get -u github.com/minio/sio

DARE and github.com/minio/sio are finalized and can be used in production.

We also provide a CLI tool to en/decrypt arbitrary data streams directly from your command line:

Install ncrypt: go get -u github.com/minio/sio/cmd/ncrypt && ncrypt -h

Performance

Cipher	8 KB	64 KB	512 KB	1 MB
AES_256_GCM	90 MB/s	1.96 GB/s	2.64 GB/s	2.83 GB/s
CHACHA20_POLY1305	97 MB/s	1.23 GB/s	1.54 GB/s	1.57 GB/s

On i7-6500U 2 x 2.5 GHz | Linux 4.10.0-32-generic | Go 1.8.3 | AES-NI & AVX2

Documentation

Overview

Package sio implements the DARE format. It provides an API for secure en/decrypting IO operations using io.Reader and io.Writer.

Index

• Constants
• func Decrypt(dst io.Writer, src io.Reader, config Config) (n int64, err error)
• func DecryptBuffer(dst, src []byte, config Config) (output []byte, err error)
• func DecryptReader(src io.Reader, config Config) (io.Reader, error)
• func DecryptReaderAt(src io.ReaderAt, config Config) (io.ReaderAt, error)
• func DecryptWriter(dst io.Writer, config Config) (io.WriteCloser, error)
• func DecryptedSize(size uint64) (uint64, error)
• func Encrypt(dst io.Writer, src io.Reader, config Config) (n int64, err error)
• func EncryptReader(src io.Reader, config Config) (io.Reader, error)
• func EncryptWriter(dst io.Writer, config Config) (io.WriteCloser, error)
• func EncryptedSize(size uint64) (uint64, error)
• type Config
• type Error
  • func (e Error) Error() string

Examples

• Decrypt
• Encrypt
• EncryptReader
• EncryptWriter

Constants

const (
	// Version20 specifies version 2.0
	Version20 byte = 0x20
	// Version10 specifies version 1.0
	Version10 byte = 0x10
)

const (
	// AES_256_GCM specifies the cipher suite AES-GCM with 256 bit keys.
	AES_256_GCM byte = iota
	// CHACHA20_POLY1305 specifies the cipher suite ChaCha20Poly1305 with 256 bit keys.
	CHACHA20_POLY1305
)

Functions

func Decrypt

func Decrypt(dst io.Writer, src io.Reader, config Config) (n int64, err error)

Decrypt reads from src until it encounters an io.EOF and decrypts all received data. The decrypted data is written to dst. It returns the number of bytes decrypted and the first error encountered while decrypting, if any.

Decrypt returns the number of bytes written to dst. Decrypt only writes data to dst if the data was decrypted successfully. It returns an error of type sio.Error if decryption fails.

Example

// the master key used to derive encryption keys
masterkey, err := hex.DecodeString("000102030405060708090A0B0C0D0E0FF0E0D0C0B0A090807060504030201000") // use your own key here
if err != nil {
	fmt.Printf("Cannot decode hex key: %v", err) // add error handling
	return
}

// the nonce used to derive the encryption key
nonce, err := hex.DecodeString("0000000000000000000000000000000000000000000000000000000000000001") // use your generated nonce here
if err != nil {
	fmt.Printf("Cannot decode hex key: %v", err) // add error handling
	return
}

// derive the encryption key from the master key and the nonce
var key [32]byte
kdf := hkdf.New(sha256.New, masterkey, nonce, nil)
if _, err = io.ReadFull(kdf, key[:]); err != nil {
	fmt.Printf("Failed to derive encryption key: %v", err) // add error handling
	return
}

input := os.Stdin   // customize for your needs - the encrypted data
output := os.Stdout // customize from your needs - the decrypted output

if _, err = Decrypt(output, input, Config{Key: key[:]}); err != nil {
	if _, ok := err.(Error); ok {
		fmt.Printf("Malformed encrypted data: %v", err) // add error handling - here we know that the data is malformed/not authentic.
		return
	}
	fmt.Printf("Failed to decrypt data: %v", err) // add error handling
	return
}

func DecryptBuffer

func DecryptBuffer(dst, src []byte, config Config) (output []byte, err error)

DecryptBuffer decrypts all received data in src. The decrypted data is appended to dst. If the number of output bytes is unknown, making a dst with capacity of len(src) is reasonable.

DecryptBuffer only returns data to if the data was decrypted successfully. It returns an error of type sio.Error if decryption fails.

func DecryptReader

func DecryptReader(src io.Reader, config Config) (io.Reader, error)

DecryptReader wraps the given src and returns an io.Reader which decrypts all received data. DecryptReader returns an error if the provided decryption configuration is invalid. The returned io.Reader returns an error of type sio.Error if the decryption fails.

func DecryptReaderAt

func DecryptReaderAt(src io.ReaderAt, config Config) (io.ReaderAt, error)

DecryptReaderAt wraps the given src and returns an io.ReaderAt which decrypts all received data. DecryptReaderAt returns an error if the provided decryption configuration is invalid. The returned io.ReaderAt returns an error of type sio.Error if the decryption fails.

func DecryptWriter

func DecryptWriter(dst io.Writer, config Config) (io.WriteCloser, error)

DecryptWriter wraps the given dst and returns an io.WriteCloser which decrypts all data written to it. DecryptWriter returns an error if the provided decryption configuration is invalid.

The returned io.WriteCloser must be closed successfully to finalize the decryption process. The returned io.WriteCloser returns an error of type sio.Error if the decryption fails.

func DecryptedSize

func DecryptedSize(size uint64) (uint64, error)

DecryptedSize computes the size of a decrypted data stream from the encrypted stream size. It is the inverse of EncryptedSize().

DecryptedSize returns an error if the provided size is to large or if the provided size is an invalid encrypted stream size.

func Encrypt

func Encrypt(dst io.Writer, src io.Reader, config Config) (n int64, err error)

Encrypt reads from src until it encounters an io.EOF and encrypts all received data. The encrypted data is written to dst. It returns the number of bytes encrypted and the first error encountered while encrypting, if any.

Encrypt returns the number of bytes written to dst.

Example

// the master key used to derive encryption keys
// this key must be keep secret
masterkey, err := hex.DecodeString("000102030405060708090A0B0C0D0E0FF0E0D0C0B0A090807060504030201000") // use your own key here
if err != nil {
	fmt.Printf("Cannot decode hex key: %v", err) // add error handling
	return
}

// generate a random nonce to derive an encryption key from the master key
// this nonce must be saved to be able to decrypt the data again - it is not
// required to keep it secret
var nonce [32]byte
if _, err = io.ReadFull(rand.Reader, nonce[:]); err != nil {
	fmt.Printf("Failed to read random data: %v", err) // add error handling
	return
}

// derive an encryption key from the master key and the nonce
var key [32]byte
kdf := hkdf.New(sha256.New, masterkey, nonce[:], nil)
if _, err = io.ReadFull(kdf, key[:]); err != nil {
	fmt.Printf("Failed to derive encryption key: %v", err) // add error handling
	return
}

input := os.Stdin   // customize for your needs - the plaintext
output := os.Stdout // customize from your needs - the decrypted output

if _, err = Encrypt(output, input, Config{Key: key[:]}); err != nil {
	fmt.Printf("Failed to encrypt data: %v", err) // add error handling
	return
}

func EncryptReader

func EncryptReader(src io.Reader, config Config) (io.Reader, error)

EncryptReader wraps the given src and returns an io.Reader which encrypts all received data. EncryptReader returns an error if the provided encryption configuration is invalid.

Example

// the master key used to derive encryption keys
// this key must be keep secret
masterkey, err := hex.DecodeString("000102030405060708090A0B0C0D0E0FF0E0D0C0B0A090807060504030201000") // use your own key here
if err != nil {
	fmt.Printf("Cannot decode hex key: %v", err) // add error handling
	return
}

// generate a random nonce to derive an encryption key from the master key
// this nonce must be saved to be able to decrypt the data again - it is not
// required to keep it secret
var nonce [32]byte
if _, err = io.ReadFull(rand.Reader, nonce[:]); err != nil {
	fmt.Printf("Failed to read random data: %v", err) // add error handling
	return
}

// derive an encryption key from the master key and the nonce
var key [32]byte
kdf := hkdf.New(sha256.New, masterkey, nonce[:], nil)
if _, err = io.ReadFull(kdf, key[:]); err != nil {
	fmt.Printf("Failed to derive encryption key: %v", err) // add error handling
	return
}

input := os.Stdin // customize for your needs - the plaintext input
encrypted, err := EncryptReader(input, Config{Key: key[:]})
if err != nil {
	fmt.Printf("Failed to encrypted reader: %v", err) // add error handling
	return
}

// the encrypted io.Reader can be used like every other reader - e.g. for copying
if _, err := io.Copy(os.Stdout, encrypted); err != nil {
	fmt.Printf("Failed to copy data: %v", err) // add error handling
	return
}

func EncryptWriter

func EncryptWriter(dst io.Writer, config Config) (io.WriteCloser, error)

EncryptWriter wraps the given dst and returns an io.WriteCloser which encrypts all data written to it. EncryptWriter returns an error if the provided decryption configuration is invalid.

The returned io.WriteCloser must be closed successfully to finalize the encryption process.

Example

// the master key used to derive encryption keys
// this key must be keep secret
masterkey, err := hex.DecodeString("000102030405060708090A0B0C0D0E0FF0E0D0C0B0A090807060504030201000") // use your own key here
if err != nil {
	fmt.Printf("Cannot decode hex key: %v", err) // add error handling
	return
}

// generate a random nonce to derive an encryption key from the master key
// this nonce must be saved to be able to decrypt the data again - it is not
// required to keep it secret
var nonce [32]byte
if _, err = io.ReadFull(rand.Reader, nonce[:]); err != nil {
	fmt.Printf("Failed to read random data: %v", err) // add error handling
	return
}

// derive an encryption key from the master key and the nonce
var key [32]byte
kdf := hkdf.New(sha256.New, masterkey, nonce[:], nil)
if _, err = io.ReadFull(kdf, key[:]); err != nil {
	fmt.Printf("Failed to derive encryption key: %v", err) // add error handling
	return
}

output := os.Stdout // customize for your needs - the encrypted output
encrypted, err := EncryptWriter(output, Config{Key: key[:]})
if err != nil {
	fmt.Printf("Failed to encrypted writer: %v", err) // add error handling
	return
}

// the encrypted io.Writer can be used now but it MUST be closed at the end to
// finalize the encryption.
if _, err = io.Copy(encrypted, os.Stdin); err != nil {
	fmt.Printf("Failed to copy data: %v", err) // add error handling
	return
}
if err = encrypted.Close(); err != nil {
	fmt.Printf("Failed to finalize encryption: %v", err) // add error handling
	return
}

func EncryptedSize

func EncryptedSize(size uint64) (uint64, error)

EncryptedSize computes the size of an encrypted data stream from the plaintext size. It is the inverse of DecryptedSize().

EncryptedSize returns an error if the provided size is to large.

Types

type Config

type Config struct {
	// The minimal supported version of the format. If
	// not set the default value - Version10 - is used.
	MinVersion byte

	// The highest supported version of the format. If
	// not set the default value - Version20 - is used.
	MaxVersion byte

	// A list of supported cipher suites. If not set the
	// default value is used.
	CipherSuites []byte

	// The secret encryption key. It must be 32 bytes long.
	Key []byte

	// The first expected sequence number. It should only
	// be set manually when decrypting a range within a
	// stream.
	SequenceNumber uint32

	// The RNG used to generate random values. If not set
	// the default value (crypto/rand.Reader) is used.
	Rand io.Reader

	// Nonce will override the nonce if set non-nil.
	// V2 will use all 12 bytes, V1 first 8 bytes.
	Nonce *[12]byte

	// The size of the encrypted payload in bytes. The
	// default value is 64KB. It should be used to restrict
	// the size of encrypted packages. The payload size
	// must be between 1 and 64 KB.
	//
	// This field is specific for version 1.0 and is
	// deprecated.
	PayloadSize int
}

Config contains the format configuration. The only field which must always be set manually is the secret key.

type Error

type Error struct {
	// contains filtered or unexported fields
}

Error is the error returned by an io.Reader or io.Writer if the encrypted data cannot be decrypted because it is malformed or not authentic.

func (Error) Error

func (e Error) Error() string