estream

README

Encrypted Stream

This package provides a flexible way to merge multiple streams with controlled encryption.

The stream is stateful and allows to send individually encrypted streams.

Features

• Allows encrypted and unencrypted streams.
• Any number of keys can be used on streams.
• Each key can be encrypted by a (different) public key.
• Each stream is identified by a string "name".
• A stream has optional (unencrypted) metadata slice.
• Keys can be re-encrypted with another public key without needing data re-encryption given the private key.
• Streams are checksummed.
• Streams cannot be truncated by early EOF.
• Format is extensible with skippable blocks.
• Allows signaling errors while writing streams.
• Nonce per stream (of course).
• Messagepack for platform independent type safety.

Usage

Create a writer that will write the stream.

You must provide an io.Writer to which the output is written. Once all streams have been written it should be closed to indicate end of payload.

    w := estream.NewWriter(output)
    defer w.Close()

It is possible to signal an error to the receiver using w.AddError(msg string). This will return the error to the receiver.

Adding keys

Keys for streams must be added. The keys themselves are 32 bytes of random data, but it must be specified how they are stored.

They can be added as plain text, which isn't secure, but allows later encryption using a public key. To add a key without encryption use w.AddKeyPlain() which will add the keys to the stream.

To add an encrypted key provide a 2048 bit public RSA key. Use w.AddKeyEncrypted(publicKey) to add a key to the stream.

Once a key has been sent on the stream it will be used for all subsequent encrypted streams. This means that different keys with different private/public keys can be sent for different streams.

Sending streams

Streams are added using either w.AddEncryptedStream or w.AddUnencryptedStream.

A string identifier can be used to identify each stream when reading. An optional byte block can also be sent.

Note that neither the name nor the byte block is encrypted, so they should not contain sensitive data.

The functions above return an io.WriteCloser. Data for this stream should be written to this interface and Close() should be called before another stream can be added.

Note that en-uncrypted streams are unbuffered, so it may be a benefit to insert a bufio.Writer to avoid very small packets. Encrypted streams are buffered since sio collects block before sending.

Reading Streams

To read back data r, err := estream.NewReader(input) can be used for create a Reader.

To set a private key, use r.SetPrivateKey(key) to set a single private key.

For multiple keys a key provider can be made to return the appropriate key:

    var key1, key2 *rsa.PrivateKey
    // (read keys)
    r.PrivateKeyProvider(func(key *rsa.PublicKey) *rsa.PrivateKey {
        if key.Equal(&key1.PublicKey) {
            return key1
        }
        if key.Equal(&key2.PublicKey) {
            return key2
        }
        // Unknown key :(
        return nil
    })

It is possible to skip streams that cannot be decrypted using r.SkipEncrypted(true).

A simple for loop can be used to get all streams:

    for {
        stream, err := r.NextStream()
        if err == io.EOF {
            // All streams read
            break
        }
        // Metadata:
        fmt.Println(stream.Name)
        fmt.Println(stream.Extra)
		
        // Stream content is a standard io.Reader
        io.Copy(os.StdOut, stream)
    }

Replacing keys

It is possible to replace public keys needed for decryption using estream.ReplaceKeys().

For encrypted keys the private key must be provided and optionally unencrypted keys can also be encrypted using a public key.

Format

Header

Format starts with 2 version bytes.

Field	Type
Major Version	1 byte
Minor Version	1 byte

Unknown major versions should be rejected by the decoder, however minor versions are assumed to be compatible, but may contain data that will be ignored by older versions.

Blocks

Field	Type	Contents
id	integer	Block ID
length	unsigned int	Length of block in bytes

Each block is preceded by a messagepack encoded int8 indicating the block type.

Positive types must be parsed by the decoder. Negative types are skippable blocks, so unknown skippable blocks can be ignored.

Blocks have their length encoded as a messagepack unsigned integer following the block ID. This indicates the number of bytes to skip after the length to reach the next block ID.

Maximum block size is 2^32-1 (4294967295) bytes.

All block content is messagepack encoded.

id 1: Plain Key

This block contains an unencrypted key that is used for all following streams.

Multiple keys can be sent, but only the latest key should be used to decrypt a stream.

Field	Type	Contents
Stream Key	bin array	32 byte key

id 2: RSA Encrypted Key

This block contains an RSA encrypted key that is used for all following streams.

Multiple keys can be sent, but only the latest key should be used to decrypt a stream.

Field	Type	Contents
Public Key	bin array	RSA public key to PKCS #1 in ASN.1 DER form
Cipher Key	bin array	32 byte key encrypted with public key above

The cipher key is encrypted with RSA-OAEP using SHA-512.

id 3: SIO Encrypted Stream

Start of stream encrypted using sio-go.

Stream will be encrypted using AES_256_GCM using the last key provided on stream.

Field	Type	Contents
Name	string	Identifier of the stream
Extra	bin array	Optional extra data
Checksum	uint8	Checksum type used for stream
Nonce	bin array	8 byte nonce used for stream

The stream consists of all data blocks following until "End Of Stream" block is sent.

Checksum is of encrypted data. There is no checksum for decrypted data.

id 4: Plain Stream

Start of unencrypted stream.

Field	Type	Contents
Name	string	Identifier of the stream
Extra	bin array	Optional extra data
Checksum	uint8	Checksum type used for stream

The stream consists of all data blocks following until "End Of Stream" block is sent.

id 5: Data Block

Data contains a data block.

Field	Type	Contents
Data	bin array	Data to append to stream

If block is part of an encrypted stream it should be sent to the stream decrypter as is.

id 6: End Of Stream

Indicates successful end of individual stream.

Field	Type	Contents
Checksum	bin array	Checksum

No more data blocks should be expected before new stream information is sent.

id 7: EOF

Indicates successful end of all streams.

id 8: Error

An error block can be sent to indicate an error occurred while generating the stream.

It is expected that the parser returns the message and stops processing.

Field	Type	Contents
Message	string	Error message that will be returned

Checksum types

ID	Type	Bytes
0	No checksum	(ignored)
1	64 bit xxhash (XXH64) (Big Endian)	8

Version History

Major	Minor	Changes
2	1	Initial Version

Documentation

Index

• Variables
• func ReplaceKeys(w io.Writer, r io.Reader, replace ReplaceFn, o ReplaceKeysOptions) error
• type Reader
  • func NewReader(r io.Reader) (*Reader, error)
  • func (r *Reader) DebugStream(w io.Writer) error
  • func (r *Reader) NextStream() (*Stream, error)
  • func (r *Reader) PrivateKeyProvider(fn func(key *rsa.PublicKey) *rsa.PrivateKey)
  • func (r *Reader) ReturnNonDecryptable(b bool)
  • func (r *Reader) SetPrivateKey(k *rsa.PrivateKey)
  • func (r *Reader) SkipEncrypted(b bool)
• type ReplaceFn
• type ReplaceKeysOptions
• type Stream
  • func (s *Stream) Skip() error
• type Writer
  • func NewWriter(w io.Writer) *Writer
  • func (w *Writer) AddEncryptedStream(name string, extra []byte) (io.WriteCloser, error)
  • func (w *Writer) AddError(msg string) error
  • func (w *Writer) AddKeyEncrypted(publicKey *rsa.PublicKey) error
  • func (w *Writer) AddKeyPlain() error
  • func (w *Writer) AddUnencryptedStream(name string, extra []byte) (io.WriteCloser, error)
  • func (w *Writer) Close() error

Variables

var ErrNoKey = errors.New("no valid private key found")

ErrNoKey is returned when a stream cannot be decrypted. The Skip function on the stream can be called to skip to the next.

Functions

func ReplaceKeys

func ReplaceKeys(w io.Writer, r io.Reader, replace ReplaceFn, o ReplaceKeysOptions) error

ReplaceKeys will replace the keys in a stream.

A replace function must be provided. See ReplaceFn for functionality. If encryptAll is set.

Types

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

func NewReader

func NewReader(r io.Reader) (*Reader, error)

NewReader will return a Reader that will split streams.

func (*Reader) DebugStream

func (r *Reader) DebugStream(w io.Writer) error

DebugStream will print stream block information to w.

func (*Reader) NextStream

func (r *Reader) NextStream() (*Stream, error)

NextStream will return the next stream. Before calling this the previous stream must be read until EOF, or Skip() should have been called. Will return nil, io.EOF when there are no more streams.

func (*Reader) PrivateKeyProvider

func (r *Reader) PrivateKeyProvider(fn func(key *rsa.PublicKey) *rsa.PrivateKey)

PrivateKeyProvider will ask for a private key matching the public key. If the function returns a nil private key the stream key will not be decrypted and if SkipEncrypted has been set any streams with this key will be silently skipped. This overrides any key set by SetPrivateKey.

func (*Reader) ReturnNonDecryptable

func (r *Reader) ReturnNonDecryptable(b bool)

ReturnNonDecryptable will return non-decryptable stream headers. Streams are returned with ErrNoKey error. Streams with this error cannot be read, but the Skip function can be invoked. SkipEncrypted overrides this.

func (*Reader) SetPrivateKey

func (r *Reader) SetPrivateKey(k *rsa.PrivateKey)

SetPrivateKey will set the private key to allow stream decryption. This overrides any function set by PrivateKeyProvider.

func (*Reader) SkipEncrypted

func (r *Reader) SkipEncrypted(b bool)

SkipEncrypted will skip encrypted streams if no private key has been set.

type ReplaceFn

type ReplaceFn func(key *rsa.PublicKey) (*rsa.PrivateKey, *rsa.PublicKey)

ReplaceFn provides key replacement.

When a key is found on stream, the function is called with the public key. The function must then return a private key to decrypt matching the key sent. The public key must then be specified that should be used to re-encrypt the stream.

If no private key is sent and the public key matches the one sent to the function the key will be kept as is. Other returned values will cause an error.

For encrypting unencrypted keys on stream a nil key will be sent. If a public key is returned the key will be encrypted with the public key. No private key should be returned for this.

type ReplaceKeysOptions

type ReplaceKeysOptions struct {
	// If EncryptAll set all unencrypted keys will be encrypted.
	EncryptAll bool

	// PassErrors will pass through error an error packet,
	// and not return an error.
	PassErrors bool
}

ReplaceKeysOptions allows passing additional options to ReplaceKeys.

type Stream

type Stream struct {
	io.Reader
	Name          string
	Extra         []byte
	SentEncrypted bool
	// contains filtered or unexported fields
}

Stream returns the next stream.

func (*Stream) Skip

func (s *Stream) Skip() error

Skip the remainder of the stream.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer provides a stream writer. Streams can optionally be encrypted. All streams have checksum verification.

func NewWriter

func NewWriter(w io.Writer) *Writer

NewWriter will return a writer that allows to add encrypted and non-encrypted data streams.

func (*Writer) AddEncryptedStream

func (w *Writer) AddEncryptedStream(name string, extra []byte) (io.WriteCloser, error)

AddEncryptedStream adds a named encrypted stream. AddKeyEncrypted must have been called before this, but multiple streams can safely use the same key. Extra data can be added, which is added without encryption or checksums.

func (*Writer) AddError

func (w *Writer) AddError(msg string) error

AddError will indicate the writer encountered an error and the reader should abort the stream. The message will be returned as an error.

func (*Writer) AddKeyEncrypted

func (w *Writer) AddKeyEncrypted(publicKey *rsa.PublicKey) error

AddKeyEncrypted will create a new encryption key and add it to the stream. The key will be encrypted with the public key provided. All following files will be encrypted with this key.

func (*Writer) AddKeyPlain

func (w *Writer) AddKeyPlain() error

AddKeyPlain will create a new encryption key and add it to the stream. The key will be stored without any encryption. All calls to AddEncryptedStream will use this key

func (*Writer) AddUnencryptedStream

func (w *Writer) AddUnencryptedStream(name string, extra []byte) (io.WriteCloser, error)

AddUnencryptedStream adds a named stream. Extra data can be added, which is added without encryption or checksums.

func (*Writer) Close

func (w *Writer) Close() error

Close will flush and close the output stream.