persistent

package
v0.0.0-...-a227d8c Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 20, 2020 License: BSD-3-Clause Imports: 46 Imported by: 0

Documentation

Overview

Package persistent implements several compatible object-storage backends, and additional functionality that can be layered upon them.

Index

Constants

This section is empty.

Variables

View Source 
var AppStorageCommits = prometheus.NewCounter(prometheus.CounterOpts{
	Name: "app_storage_commits",
	Help: "The number of successful app storage transactions committed.",
})
View Source 
var (
	B2Ops = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Name: "b2_ops",
			Help: "The number of operations against a B2 backend.",
		},
		[]string{"operation", "success"},
	)
)
View Source 
var DiskCacheSize = prometheus.NewGaugeVec(
	prometheus.GaugeOpts{
		Name: "disk_cache_size",
		Help: "The number of entries in the on-disk cache.",
	},
	[]string{"path"},
)
View Source 
var (
	ErrObjectNotFound = errors.New("object not found")
)
View Source 
var (
	GCSOps = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Name: "gcs_ops",
			Help: "The number of operations against a GCS backend.",
		},
		[]string{"operation", "success"},
	)
)
View Source 
var LocalWALSize = prometheus.NewGaugeVec(
	prometheus.GaugeOpts{
		Name: "local_wal_size",
		Help: "The number of entries in the local WAL.",
	},
	[]string{"path"},
)
View Source 
var S3Ops = prometheus.NewCounterVec(
	prometheus.CounterOpts{
		Name: "s3_ops",
		Help: "The number of operations against an S3 backend.",
	},
	[]string{"operation", "success"},
)

Functions

func NewRemoteServer 

func NewRemoteServer(base ReliableStorage, transportKey string, oram bool) (*http.Server, error)

NewRemoteServer wraps a ReliableStorage implementation in an HTTP handler, allowing remote clients to make requests to it.

The corresponding client implementation is in NewRemoteClient.

Types

type AppStorage 

type AppStorage struct {
	// contains filtered or unexported fields
}

AppStorage is an extension of the BlockStorage interface that provides shared state.

func NewAppStorage 

func NewAppStorage(base BlockStorage) *AppStorage

func (*AppStorage) Commit 

func (as *AppStorage) Commit(ctx context.Context) error

func (*AppStorage) Get 

func (as *AppStorage) Get(ctx context.Context, ptr uint64) ([]byte, error)

func (*AppStorage) GetMany 

func (as *AppStorage) GetMany(ctx context.Context, ptrs []uint64) (map[uint64][]byte, error)

func (*AppStorage) Rollback 

func (as *AppStorage) Rollback(ctx context.Context)

func (*AppStorage) Set 

func (as *AppStorage) Set(ctx context.Context, ptr uint64, data []byte, dt DataType) error

func (*AppStorage) Start 

func (as *AppStorage) Start(ctx context.Context) error

func (*AppStorage) State 

func (as *AppStorage) State(ctx context.Context) (*State, error)

State returns a struct of shared global state. Consumers may modify the returned struct, and these modifications will be persisted after Commit is called.

type BlockStorage 

type BlockStorage interface {
	Start(ctx context.Context, prefetch []uint64) (data map[uint64][]byte, err error)

	Get(ctx context.Context, ptr uint64) (data []byte, err error)
	GetMany(ctx context.Context, ptrs []uint64) (data map[uint64][]byte, err error)
	Set(ctx context.Context, ptr uint64, data []byte, dt DataType) (err error)

	Commit(ctx context.Context) error
	Rollback(ctx context.Context)
}

BlockStorage is a derivative of ObjectStorage that uses uint64 pointers as keys instead of strings. It is meant to help make implementing ORAM easier.

func NewBlockMemory 

func NewBlockMemory() BlockStorage

NewBlockMemory returns an implementation of BlockStorage that simply stores data in-memory.

func NewBufferedStorage 

func NewBufferedStorage(base ReliableStorage) BlockStorage

func WithEncryption 

func WithEncryption(base BlockStorage, password string) BlockStorage

WithEncryption wraps a BlockStorage implementation and makes sure that all values are encrypted with AES-GCM before being processed further.

The encryption key is derived with Argon2 from `password`.

func WithIntegrity 

func WithIntegrity(base BlockStorage, password, pinFile string) (BlockStorage, error)

WithIntegrity wraps a BlockStorage implementation and builds a Merkle tree over the data stored.

The root of the Merkle tree is authenticated by `password`, and a copy of the root and other metadata is kept in `pinFile`.

func WithORAM 

func WithORAM(base BlockStorage, store ObliviousStorage, maxSize int64) (BlockStorage, error)

WithORAM wraps a BlockStorage implementation and prevents outsiders from seeing the user's access pattern. This includes which data a user is accessing and whether the user is reading or writing, but does not include the total amount of accesses or when.

A small amount of local data is stored in `store`. `maxSize` is the maximum size of a block of data.

type BufferedStorage 

type BufferedStorage struct {
	// contains filtered or unexported fields
}

BufferedStorage is an extension of the ReliableStorage interface that will buffer many changes and then commit them all at once.

func (*BufferedStorage) Commit 

func (bs *BufferedStorage) Commit(ctx context.Context) error

Commit persists any changes made to the backend.

func (*BufferedStorage) Delete 

func (bs *BufferedStorage) Delete(ctx context.Context, key uint64) error

func (*BufferedStorage) Get 

func (bs *BufferedStorage) Get(ctx context.Context, key uint64) ([]byte, error)

func (*BufferedStorage) GetMany 

func (bs *BufferedStorage) GetMany(ctx context.Context, keys []uint64) (map[uint64][]byte, error)

func (*BufferedStorage) Rollback 

func (bs *BufferedStorage) Rollback(ctx context.Context)

Rollback discards all changes made in this transaction.

func (*BufferedStorage) Set 

func (bs *BufferedStorage) Set(ctx context.Context, key uint64, data []byte, dt DataType) error

func (*BufferedStorage) Start 

func (bs *BufferedStorage) Start(ctx context.Context, prefetch []uint64) (map[uint64][]byte, error)

type DataType 

type DataType int

DataType represents the semantics of the data in a Set operation. It helps layers lower in the stack make smarter caching / performance decisions. 

const (
	Unknown DataType = iota
	Metadata
	Content
)

type MapMutex 

type MapMutex struct {
	// contains filtered or unexported fields
}

MapMutex implements the ability to lock and unlock specific keys of a map.

func NewMapMutex 

func NewMapMutex() MapMutex

func (MapMutex) Lock 

func (mm MapMutex) Lock(key interface{})

func (MapMutex) Unlock 

func (mm MapMutex) Unlock(key interface{})

type ObjectStorage 

type ObjectStorage interface {
	// Get returns the data corresponding to the given key, or ErrObjectNotFound
	// if no object with that key exists.
	Get(ctx context.Context, key string) (data []byte, err error)
	// Set updates the object with the given key or creates the object if it
	// does not exist.
	Set(ctx context.Context, key string, data []byte, dt DataType) (err error)
	// Delete removes the object with the given key.
	Delete(ctx context.Context, key string) (err error)
}

ObjectStorage defines the minimal interface that's implemented by a remote object storage provider.

func NewB2 

func NewB2(acctId, appKey, bucketName, url string) (ObjectStorage, error)

NewB2 returns object storage backed by Backblaze B2. `acctId` and `appKey` are the Account ID and Application Key of a B2 bucket. `bucketName` is the name of the bucket. `url` is the URL to use to download data.

func NewDisk 

func NewDisk(loc string) (ObjectStorage, error)

NewDisk returns object storage backed by an on-disk database stored at `loc`.

func NewDiskCache 

func NewDiskCache(base ObjectStorage, loc string, size int64, exclude []DataType) (ObjectStorage, error)

NewDiskCache wraps a base object storage backend with a large on-disk cache stored at `loc`.

func NewGCS 

func NewGCS(bucketName, credentialsPath string) (ObjectStorage, error)

NewGCS returns object storage backed by Google Compute Storage. `bucketName` is the name of the bucket to use. Authentication credentials should be stored in a file, and the path to that file is `credentialsPath`.

func NewMemory 

func NewMemory() ObjectStorage

NewMemory returns an object storage backend that simply stores data in-memory.

func NewPrefix 

func NewPrefix(base ObjectStorage, p string) ObjectStorage

NewPrefix wraps a base object storage backend, and ensures a user-provided prefix is added to all keys.

func NewRetry 

func NewRetry(base ObjectStorage, attempts int) (ObjectStorage, error)

NewRetry wraps a base object storage backend, and will retry if requests fail.

func NewS3 

func NewS3(appId, appKey, bucket, url, region string) (ObjectStorage, error)

NewS3 returns object storage backed by AWS S3 or a compatible service like Wasabi. `appId` and `appKey` are the static credentials. `bucket` is the name of the bucket. `url` and `region` are the location of the S3 cluster.

func NewTieredCache 

func NewTieredCache(special DataType, high, base ObjectStorage) ObjectStorage

NewTieredCache returns a cache-like object storage implementation, where objects matching the `special` data type are stored in both `high` and `base`, while all other objects are stored only in `base`.

type ObliviousStorage 

type ObliviousStorage interface {
	// Start begins a new transaction. It returns the number of blocks and the
	// current stash.
	Start(ctx context.Context, version uint64) (stash map[uint64][]byte, size uint64, err error)

	// Lookup returns a map from each requested pointer, to the leaf that the
	// pointer is assigned to.
	Lookup(ctx context.Context, ptr []uint64) (map[uint64]uint64, error)

	// Commit ends the transaction, replacing the stash with the given map and
	// making the requested pointer-to-leaf assignments.
	Commit(ctx context.Context, version uint64, stash map[uint64][]byte, assignments map[uint64]uint64) error

	// Rollback aborts the transaction without attempting to make any changes.
	Rollback(ctx context.Context)
}

ObliviousStorage defines the interface an ORAM implementation would use to access and store sensitive data.

func NewLocalOblivious 

func NewLocalOblivious(loc string) (ObliviousStorage, error)

NewLocalOblivious returns an implementation of the ObliviousStorage interface, used for storing temporary ORAM data, that's backed by an on-disk database at `loc`.

type ReliableStorage 

type ReliableStorage interface {
	// Start begins a new transaction. The methods below will not work until
	// this is called, and will stop working again after Commit is called.
	//
	// Transactions are isolated and atomic.
	Start(ctx context.Context, prefetch []uint64) (data map[uint64][]byte, err error)

	Get(ctx context.Context, key uint64) (data []byte, err error)
	GetMany(ctx context.Context, keys []uint64) (data map[uint64][]byte, err error)

	// Commit persists the changes in `writes` to the backend, atomically. If
	// the value of a key is nil, then that key is deleted.
	Commit(ctx context.Context, writes map[uint64]WriteData) error
}

ReliableStorage is an extension of the ObjectStorage interface that provides distributed locking (if necessary) and atomic transactions.

func NewBlockReliable 

func NewBlockReliable(base BlockStorage) ReliableStorage

NewBlockReliable returns a ReliableStorage implementation based on a BlockStorage implementation.

func NewCache 

func NewCache(base ReliableStorage, size int) ReliableStorage

NewCache wraps a base object storage backend with an LRU cache of the requested size.

func NewLocalWAL 

func NewLocalWAL(base ObjectStorage, loc string, maxSize, parallelism int) (ReliableStorage, error)

NewLocalWAL returns a ReliableStorage implementation that achieves reliable writes over a base object storage provider by buffering writes in a Write-Ahead Log (WAL) stored at `loc`.

The WAL may have at least `maxSize` buffered entries before new writes start blocking on old writes being flushed.

func NewRemoteClient 

func NewRemoteClient(transportKey, serverUrl string, oram bool) (ReliableStorage, error)

NewRemoteClient returns a ReliableStorage implementation that defers reads and writes to a remote server.

The corresponding server implementation is in NewRemoteServer.

func NewSimpleReliable 

func NewSimpleReliable(base ObjectStorage) ReliableStorage

NewSimpleReliable returns a ReliableStorage implementation, intended for testing. It simply panics if the atomicity of a transaction is broken.

type State 

type State struct {
	// RootPtr points to the root inode of the filesystem.
	RootPtr uint64

	// Blocks that were previously allocated but are now un-used are kept in a
	// linked list. TrashPtr points to the head of this list.
	TrashPtr uint64
	// NextPtr will be the pointer of the next block which is allocated.
	NextPtr uint64
}

State contains all of the shared global state of a deployment.

func NewState 

func NewState() *State

func (*State) Clone 

func (s *State) Clone() *State

type WriteData 

type WriteData struct {
	Data []byte
	Type DataType
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.