storage

README

Storage

This go package is used to store data in encrypted files. It is designed for convenience and correctness, and may not be suitable for high volume or high throughput applications. It was refactored from internal c2fmzq-server code, and re-licenced, so that other projects can use it.

GO objects can be saved, loaded, and atomically updated.

mk, err := crypto.CreateMasterKey([]byte("<passphrase>"))
if err != nil {
    panic(err)
}
store := storage.New("<data dir>", mk)

var obj1 Object
// Populate obj1
if err := store.SaveDataFile("<relative filename>", &obj1); err != nil {
    panic(err)
}

var obj2 Object
if err := store.ReadDataFile("<relative filename>", &obj2); err != nil {
    panic(err)
}
// obj1 and obj2 have the same value

To update objects atomically:

func foo() (retErr error) {
    var obj Object
    commit, err := store.OpenForUpdate("<relative filename>", &obj)
    if err != nil {
        panic(err)
    }
    defer commit(false, &retErr) // rollback unless first committed.
    // modify obj
    obj.Bar = X
    return commit(true, nil) // commit
}

Multiple objects can be updated atomically with OpenManyForUpdate().

Developers can also use OpenBlobRead() and OpenBlobWrite() to read and write encrypted BLOBs with a streaming API.

Documentation

Overview

Package storage stores arbitrary data in encrypted files.

Index

• Variables
• func AddPadding(w io.Writer, max int) error
• func SkipPadding(r io.ReadSeeker) error
• type Storage
  • func New(dir string, masterKey crypto.EncryptionKey) *Storage
  • func (s *Storage) CreateEmptyFile(filename string, empty interface{}) error
  • func (s *Storage) Dir() string
  • func (s *Storage) EditDataFile(filename string, obj interface{}) (retErr error)
  • func (s *Storage) HashString(str string) string
  • func (s *Storage) Lock(fn string) error
  • func (s *Storage) LockMany(filenames []string) error
  • func (s *Storage) Logger() crypto.Logger
  • func (s *Storage) OpenBlobRead(filename string) (stream io.ReadSeekCloser, retErr error)
  • func (s *Storage) OpenBlobWrite(writeFileName, finalFileName string) (io.WriteCloser, error)
  • func (s *Storage) OpenForUpdate(f string, obj interface{}) (func(commit bool, errp *error) error, error)
  • func (s *Storage) OpenManyForUpdate(files []string, objects interface{}) (func(commit bool, errp *error) error, error)
  • func (s *Storage) ReadDataFile(filename string, obj interface{}) error
  • func (s *Storage) SaveDataFile(filename string, obj interface{}) error
  • func (s *Storage) Unlock(fn string) error
  • func (s *Storage) UnlockMany(filenames []string) error

Variables

var (
	// Indicates that the update was successfully rolled back.
	ErrRolledBack = errors.New("rolled back")
	// Indicates that the update was already rolled back by a previous call.
	ErrAlreadyRolledBack = errors.New("already rolled back")
	// Indicates that the update was already committed by a previous call.
	ErrAlreadyCommitted = errors.New("already committed")
)

Functions

func AddPadding

func AddPadding(w io.Writer, max int) error

AddPadding writes a random-sized padding in the range [0,max[ at the current write position.

func SkipPadding

func SkipPadding(r io.ReadSeeker) error

SkipPadding skips the random-sized padding starting at the current read position.

Types

type Storage

type Storage struct {
	// contains filtered or unexported fields
}

Storage offers the API to atomically read, write, and update encrypted files.

func New

func New(dir string, masterKey crypto.EncryptionKey) *Storage

New returns a new Storage rooted at dir. The caller must provide an EncryptionKey that will be used to encrypt and decrypt per-file encryption keys.

func (*Storage) CreateEmptyFile

func (s *Storage) CreateEmptyFile(filename string, empty interface{}) error

CreateEmptyFile creates an empty file.

func (*Storage) Dir

func (s *Storage) Dir() string

Dir returns the root directory of the storage.

func (*Storage) EditDataFile

func (s *Storage) EditDataFile(filename string, obj interface{}) (retErr error)

EditDataFile opens a file in a text editor.

func (*Storage) HashString

func (s *Storage) HashString(str string) string

HashString returns a cryptographically secure hash of a string.

func (*Storage) Lock

func (s *Storage) Lock(fn string) error

Lock atomically creates a lock file for the given filename. When this function returns without error, the lock is acquired and nobody else can acquire it until it is released.

There is logic in place to remove stale locks after a while.

func (*Storage) LockMany

func (s *Storage) LockMany(filenames []string) error

LockMany locks multiple files such that if the exact same files are locked concurrently, there won't be any deadlock.

When the function returns successfully, all the files are locked.

func (*Storage) Logger

func (s *Storage) Logger() crypto.Logger

Logger returns the logger associated with the storage's master key.

func (*Storage) OpenBlobRead

func (s *Storage) OpenBlobRead(filename string) (stream io.ReadSeekCloser, retErr error)

OpenBlobRead opens a blob file for reading.

func (*Storage) OpenBlobWrite

func (s *Storage) OpenBlobWrite(writeFileName, finalFileName string) (io.WriteCloser, error)

OpenBlobWrite opens a blob file for writing. writeFileName is the name of the file where to write the data. finalFileName is the final name of the file. The caller is expected to rename the file to that name when it is done with writing.

func (*Storage) OpenForUpdate

func (s *Storage) OpenForUpdate(f string, obj interface{}) (func(commit bool, errp *error) error, error)

OpenForUpdate opens a file with the expectation that the object will be modified and then saved again.

Example:

 func foo() (retErr error) {
   var foo FooStruct
   commit, err := s.OpenForUpdate(filename, &foo)
   if err != nil {
     panic(err)
   }
   defer commit(false, &retErr) // rollback unless first committed.
   // modify foo
   foo.Bar = X
   return commit(true, nil) // commit
}

func (*Storage) OpenManyForUpdate

func (s *Storage) OpenManyForUpdate(files []string, objects interface{}) (func(commit bool, errp *error) error, error)

OpenManyForUpdate is like OpenForUpdate, but for multiple files.

Example:

 func foo() (retErr error) {
   file1, file2 := "file1", "file2"
   var foo FooStruct
   var bar BarStruct
   // foo is read from file1, bar is read from file2.
   commit, err := s.OpenManyForUpdate([]string{file1, file2}, []interface{}{&foo, &bar})
   if err != nil {
     panic(err)
   }
   defer commit(false, &retErr) // rollback unless first committed.
   // modify foo and bar
   foo.X = "new X"
   bar.Y = "new Y"
   return commit(true, nil) // commit
}

func (*Storage) ReadDataFile

func (s *Storage) ReadDataFile(filename string, obj interface{}) error

ReadDataFile reads an object from a file.

func (*Storage) SaveDataFile

func (s *Storage) SaveDataFile(filename string, obj interface{}) error

SaveDataFile atomically replace an object in a file.

func (*Storage) Unlock

func (s *Storage) Unlock(fn string) error

Unlock released the lock file for the given filename.

func (*Storage) UnlockMany

func (s *Storage) UnlockMany(filenames []string) error

UnlockMany unlocks multiples files locked by LockMany().