hoard

README

Hoard

Hoard is a stateless, deterministically encrypted, content-addressed object store.

Introduction

It convergently encrypts an object using its (SHA256) hash as the secret key (which can than be shared as a 'grant'). The address is then deterministically generated from the encrypted object's (SHA256) digest and allocated to the configured storage back-end:

• In-Memory
• Filesystem
• AWS
• GCP
• Azure
• IPFS

Installing

Hoard should be go-gettable with:

# Install Hoard-Daemon:
go get github.com/monax/hoard/cmd/hoard

# Install Hoard-Control:
go get github.com/monax/hoard/cmd/hoarctl

Usage

Hoard runs as a daemon providing a GRPC service to other clients including the command line client hoarctl. The purpose of the daemon is to read local secrets (such as PGP or other keys) and to configure itself to use a particular storage backend. You can run the daemon with:

# Run the daemon
hoard

# or with logging
hoard --logging

You can initialise a Hoard config by running one of:

# Initialise Hoard with memory backend
hoard config --init memory

# Initialise Hoard with filesystem backend
hoard config --init filesystem

# Initialise Hoard with AWS (S3) backend
hoard config --init aws

# Initialise Hoard with Azure backend
hoard config --init azure

# Initialise Hoard with GCP backend
hoard config --init gcp

# Initialise Hoard with IPFS backend
hoard config --init ipfs

These will provide base configurations you can configure to meet your needs. The config is located by default in $HOME/.config/hoard.conf but you can specify a file with hoard -c /path/to/config. The XDG base directory specification is used to search for config.

You can interact with Hoard using the go client hoarctl:

# Store an object:
ref=$(echo bar | hoarctl put)

# Retrieve 'bar' from its (deterministic) reference
echo $ref | hoarctl get

# Or get information about the object without decrypting
echo $ref | hoarctl stat

# This one-liner exercises the entire API:
echo foo | hoarctl put | hoarctl get | hoarctl putseal | hoarctl unsealget | hoarctl encrypt | hoarctl insert | hoarctl stat | hoarctl cat | hoarctl decrypt -k tbudgBSg+bHWHiHnlteNzN8TUvI80ygS9IULh4rklEw= | hoarctl ref | hoarctl seal | hoarctl reseal | hoarctl unseal | hoarctl get

You can chop off segments of the final command to see the output of each intermediate command. It is contrived so that the outputs can be used as inputs for the next pipeline step. hoarctl either returns JSON references or raw bytes depending on the command. You may find the excellent jq useful for working with single-line JSON files on the command line.

Config

Using the filesystem storage backend as an example (generated with hoard init -o- fs) you can configure Hoard with a file like:

# The listen address, also supported is "unix:///tmp/hoard.socket" for a unix domain socket
ListenAddress = "tcp://localhost:53431"

[Storage]
  StorageType = "filesystem"
  # One of: base64, base32, or hex (base 16)
  AddressEncoding = "base64"
  RootDirectory = "/home/user/.local/share/hoard"

[Logging]
  LoggingType = "logfmt"
  # Removing "trace" from this array will reduce log output
  Channels = ["info", "trace"]

The default directory is $HOME/.config/hoard.toml or you can pass the file with hoard -c.

Specification

See hoard.proto for the protobuf3 definition of the API. Hoard uses GRPC for its API for which there is a wide range of client libraries available. You should be able to set up a client in any GRPC supported language with relative ease. Also see hoarctl <CMD> -h for full help on each sub-command.

For more information on the design of Hoard please checkout our documentation.

Building

To build Hoard you will need to have the following installed:

• Go (Version >= 1.11) (with $GOPATH/bin in $PATH)
• GNU make
• Protocol Buffers 3

Then, from the project root run:

# Install protobuf GRPC plugin, glide, and glide dependencies
make protobuf_deps
# Run checks, tests, and build binaries
make build && make install

Javascript Client

A Javascript client library can be found here: hoard-js.

Hoard-js is a fairly lightweight wrapper around the Hoard GRPC API. It mainly serves to abstract over the dynamic protobuf library and the static protobuf generation.

Usage

First we need to have Hoard running. For development purposes this can be accomplished by:

go get github.com/monax/hoard/cmd/hoard 
# Run Hoard with logging
hoard --logging

Hoard will run with an in-memory store by default that will be discarded when it is shutdown, but will expose the same interface as when using remote storage backends.

To interact with Hoard from Node see example.js for a self-contained example of how to use every method of the API. To run use:

# Get dependencies
npm install
# Run example
node example.js

Documentation

Index

• Constants
• func ReadStream(receiver func() (interface{}, error)) (interface{}, error)
• func ReceiveAllAddresses(cli interface{ ... }) ([]*api.Address, error)
• func ReceiveAllPlaintexts(cli interface{ ... }) (*api.Plaintext, error)
• func ReceiveAllReferences(cli interface{ ... }) (reference.Refs, error)
• func SendPlaintext(data []byte, chunkSize int, srv PlaintextSender, version int32) error
• func StreamFileFrom(reader io.Reader, chunkSize int, sender func(chunk []byte) error) error
• func StreamFileTo(writer io.Writer, receiver func() ([]byte, error)) error
• type EncryptionService
• type GrantService
• type Hoard
  • func NewHoard(store stores.NamedStore, secrets config.SecretsManager, logger log.Logger) *Hoard
  • func (hrd *Hoard) Decrypt(ref *reference.Ref, encryptedData []byte) ([]byte, error)
  • func (hrd *Hoard) Delete(address []byte) error
  • func (hrd *Hoard) Encrypt(data, salt []byte) (*reference.Ref, []byte, error)
  • func (hrd *Hoard) Get(ref *reference.Ref) ([]byte, error)
  • func (hrd *Hoard) Name() string
  • func (hrd *Hoard) Put(data, salt []byte) (*reference.Ref, error)
  • func (hrd *Hoard) Seal(refs reference.Refs, spec *grant.Spec) (*grant.Grant, error)
  • func (hrd *Hoard) Store() stores.ContentAddressedStore
  • func (hrd *Hoard) Unseal(grt *grant.Grant) (reference.Refs, error)
• type ObjectService
• type PlaintextSender
• type Service
  • func NewService(grantService GrantService, chunkSize int) *Service
  • func (service *Service) Decrypt(srv api.Encryption_DecryptServer) error
  • func (service *Service) Delete(ctx context.Context, address *api.Address) (*api.Address, error)
  • func (service *Service) Encrypt(srv api.Encryption_EncryptServer) error
  • func (service *Service) Get(srv api.Cleartext_GetServer) error
  • func (service *Service) Pull(srv api.Storage_PullServer) error
  • func (service *Service) Push(srv api.Storage_PushServer) error
  • func (service *Service) Put(srv api.Cleartext_PutServer) error
  • func (service *Service) PutSeal(srv api.Grant_PutSealServer) error
  • func (service *Service) Reseal(ctx context.Context, arg *api.GrantAndGrantSpec) (*grant.Grant, error)
  • func (service *Service) Seal(srv api.Grant_SealServer) error
  • func (service *Service) Stat(ctx context.Context, address *api.Address) (*stores.StatInfo, error)
  • func (service *Service) Unseal(grt *grant.Grant, srv api.Grant_UnsealServer) error
  • func (service *Service) UnsealDelete(grt *grant.Grant, srv api.Grant_UnsealDeleteServer) error
  • func (service *Service) UnsealGet(grt *grant.Grant, srv api.Grant_UnsealGetServer) error

Constants

const MaxChunkSize = 1 << 20

MaxChunkSize = 1MiB

Functions

func ReadStream

func ReadStream(receiver func() (interface{}, error)) (interface{}, error)

ReadStream returns an interface when it is non-nil

func ReceiveAllAddresses

func ReceiveAllAddresses(cli interface {
	Recv() (*api.Address, error)
}) ([]*api.Address, error)

func ReceiveAllPlaintexts

func ReceiveAllPlaintexts(cli interface {
	Recv() (*api.Plaintext, error)
}) (*api.Plaintext, error)

func ReceiveAllReferences

func ReceiveAllReferences(cli interface {
	Recv() (*reference.Ref, error)
}) (reference.Refs, error)

func SendPlaintext

func SendPlaintext(data []byte, chunkSize int, srv PlaintextSender, version int32) error

SendPlaintext gets the plaintext for a given reference and sends it to the client

func StreamFileFrom

func StreamFileFrom(reader io.Reader, chunkSize int, sender func(chunk []byte) error) error

StreamFileFrom provides a convenience wrapper over an io.Reader

func StreamFileTo

func StreamFileTo(writer io.Writer, receiver func() ([]byte, error)) error

StreamFileTo provides a convenience wrapper over an io.Writer

Types

type EncryptionService

type EncryptionService interface {
	// Encrypt data and return it along with reference
	Encrypt(data, salt []byte) (ref *reference.Ref, encryptedData []byte, err error)
	// Encrypt data and return it along with reference
	Decrypt(ref *reference.Ref, encryptedData []byte) (data []byte, err error)
}

type GrantService

type GrantService interface {
	ObjectService
	// Seal a reference by encrypting it according to a grant spec
	Seal(refs reference.Refs, spec *grant.Spec) (*grant.Grant, error)
	// Unseal a grant by decrypting it and returning the reference
	Unseal(grt *grant.Grant) (reference.Refs, error)
}

type Hoard

type Hoard struct {
	// contains filtered or unexported fields
}

This is our top level API object providing library acting as a deterministic encrypted store and a grant issuer. It can be consumed as a Go library or as a GRPC service through grpcService which just plumbs this object into the hoard.proto interface.

func NewHoard

func NewHoard(store stores.NamedStore, secrets config.SecretsManager, logger log.Logger) *Hoard

func (*Hoard) Decrypt

func (hrd *Hoard) Decrypt(ref *reference.Ref, encryptedData []byte) ([]byte, error)

Decrypt data using reference

func (*Hoard) Delete

func (hrd *Hoard) Delete(address []byte) error

func (*Hoard) Encrypt

func (hrd *Hoard) Encrypt(data, salt []byte) (*reference.Ref, []byte, error)

Encrypt data and get reference

func (*Hoard) Get

func (hrd *Hoard) Get(ref *reference.Ref) ([]byte, error)

Gets encrypted blob

func (*Hoard) Name

func (hrd *Hoard) Name() string

func (*Hoard) Put

func (hrd *Hoard) Put(data, salt []byte) (*reference.Ref, error)

Encrypts data and storage it in underlying store and returns the address

func (*Hoard) Seal

func (hrd *Hoard) Seal(refs reference.Refs, spec *grant.Spec) (*grant.Grant, error)

func (*Hoard) Store

func (hrd *Hoard) Store() stores.ContentAddressedStore

func (*Hoard) Unseal

func (hrd *Hoard) Unseal(grt *grant.Grant) (reference.Refs, error)

type ObjectService

type ObjectService interface {
	EncryptionService
	// Get encrypted data from underlying storage at address and decrypt it
	Get(ref *reference.Ref) (data []byte, err error)
	// Encrypt data and put it in underlying storage
	Put(data, salt []byte) (*reference.Ref, error)
	// Delete underlying data obtained by address
	Delete(address []byte) error
	// Get the underlying ContentAddressedStore
	Store() stores.ContentAddressedStore
}

type PlaintextSender

type PlaintextSender interface {
	Send(*api.Plaintext) error
}

type Service

type Service struct {
	// contains filtered or unexported fields
}

Service implements the GRPC Hoard service. It should mostly be plumbing to a DeterministicEncryptedStore (for which hoard.hoard is the canonical example) and also to Grants.

func NewService

func NewService(grantService GrantService, chunkSize int) *Service

func (*Service) Decrypt

func (service *Service) Decrypt(srv api.Encryption_DecryptServer) error

Decrypt ciphertext and return plaintext

func (*Service) Delete

func (service *Service) Delete(ctx context.Context, address *api.Address) (*api.Address, error)

Delete removes the data located at the address

func (*Service) Encrypt

func (service *Service) Encrypt(srv api.Encryption_EncryptServer) error

Encrypt data and return ciphertext

func (*Service) Get

func (service *Service) Get(srv api.Cleartext_GetServer) error

Get decrypted data from the store

func (*Service) Pull

func (service *Service) Pull(srv api.Storage_PullServer) error

Pull gets ciphertext directly from the store

func (*Service) Push

func (service *Service) Push(srv api.Storage_PushServer) error

Push ciphertext directly to store

func (*Service) Put

func (service *Service) Put(srv api.Cleartext_PutServer) error

Put encrypted data in the store

func (*Service) PutSeal

func (service *Service) PutSeal(srv api.Grant_PutSealServer) error

PutSeal encrypts and seals plaintext

func (*Service) Reseal

func (service *Service) Reseal(ctx context.Context, arg *api.GrantAndGrantSpec) (*grant.Grant, error)

Reseal changes how the references in a grant are stored

func (*Service) Seal

func (service *Service) Seal(srv api.Grant_SealServer) error

Seal puts refs in a shareable grant

func (*Service) Stat

func (service *Service) Stat(ctx context.Context, address *api.Address) (*stores.StatInfo, error)

Stat checks the data stored at the given address

func (*Service) Unseal

func (service *Service) Unseal(grt *grant.Grant, srv api.Grant_UnsealServer) error

Unseal gets the refs stored in a grant

func (*Service) UnsealDelete

func (service *Service) UnsealDelete(grt *grant.Grant, srv api.Grant_UnsealDeleteServer) error

UnsealDelete gets the references stored in a grant and deletes them

func (*Service) UnsealGet

func (service *Service) UnsealGet(grt *grant.Grant, srv api.Grant_UnsealGetServer) error

UnsealGet decrypts and gets plaintext associated with a grant