stateio

Documentation

Overview

Package stateio implements persistent state mechanism based on log files that interleave indexed state snapshots with state updates. Users maintain state by interaction with Reader and Writer objects. A typical application should reconcile to the current state before writing new log entries. New log entries should be written only after they are known to apply cleanly. (In the following examples, error handling is left as an exercise to the reader):

file, err := os.OpenFile(filename, os.O_RDWR|os.O_CREATE)
state, epoch, updates, err := stateio.RestoreFile(file)
application.Reset(state)
for {
	entry, err := updates.Read()
	if err == io.EOF {
		break
	}
	application.Update(entry)
}

w, err := stateio.NewFileWriter(file)

// Apply new state updates:
var update []byte = ...
application.Update(update)
err := w.Update(update)

// Produce a new snapshot:
var snap []byte = application.Snapshot()
w.Snapshot(snap)

Data format

State files maintained by package stateio builds on package logio. The log file contains a sequence of epochs, each beginning with a state snapshot (with the exception of the first epoch, which does not need a state snapshot). Each entry is prefixed with the type of the entry as well as the the epoch to which the entry belongs. Snapshot entries are are prefixed with the previous epoch, so that log files can efficiently rewound.

TODO(marius): support log file truncation

Index

• Variables
• type Reader
  • func Restore(r io.ReaderAt, limit int64) (state []byte, epoch uint64, updates *Reader, err error)
  • func RestoreFile(file *os.File) (state []byte, epoch uint64, updates *Reader, err error)
  • func (r *Reader) Read() ([]byte, error)
• type Writer
  • func NewFileWriter(file *os.File) (*Writer, error)
  • func NewWriter(w io.Writer, off int64, epoch uint64) *Writer
  • func (w *Writer) Snapshot(snap []byte) error
  • func (w *Writer) Update(update []byte) error

Variables

var ErrCorrupt = errors.New("corrupt state entry")

ErrCorrupt is returned when a corrupted log is encountered.

Types

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader reads a single epoch state updates.

func Restore

func Restore(r io.ReaderAt, limit int64) (state []byte, epoch uint64, updates *Reader, err error)

Restore restores the state from the last epoch in the state log read by the provided reader and the given limit. The returned state may be nil if no snapshot was defined for the epoch.

func RestoreFile

func RestoreFile(file *os.File) (state []byte, epoch uint64, updates *Reader, err error)

RestoreFile is a convenience function that restores the file from the provided os file.

func (*Reader) Read

func (r *Reader) Read() ([]byte, error)

Read returns the next state update entry. Read returns ErrCorrupt if a corrupted log entry was encountered, or logio.ErrCorrupt is a corrupt log file was encountered. In the latter case, the user may skip the corrupted entry by issuing another read.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer writes snapshots and update entries to an underlying log stream.

func NewFileWriter

func NewFileWriter(file *os.File) (*Writer, error)

NewFileWriter initializes a state log writer from the provided os file. The file's contents is committed to stable storage after each log write.

func NewWriter

func NewWriter(w io.Writer, off int64, epoch uint64) *Writer

NewWriter initializes and returns a new state log writer which writes to the stream w, which is positioned at the provided offset. The provided epoch must be the current epoch of the log file. If the provided io.Writer is also a syncer:

type Syncer interface {
	Sync() error
}

Then Sync() is called (and errors returned) after each log entry has been written.

func (*Writer) Snapshot

func (w *Writer) Snapshot(snap []byte) error

Snapshot writes a new snapshot to the state log. Subsequent updates are based on this snapshot.

func (*Writer) Update

func (w *Writer) Update(update []byte) error

Update writes a new state update to the log. The update refers to the last snapshot written.