wal

Documentation

Index

• Constants
• Variables
• func Repair(logger api.Logger, dirPath string) error
• type LogRecordCRC
• type LogRecordHeader
• type LogRecordLength
• type LogRecordReader
  • func NewLogRecordReader(logger api.Logger, fileName string) (*LogRecordReader, error)
  • func (r *LogRecordReader) CRC() uint32
  • func (r *LogRecordReader) Close() error
  • func (r *LogRecordReader) Read() (*protos.LogRecord, error)
• type Options
  • func DefaultOptions() *Options
  • func (o *Options) String() string
• type WriteAheadLogFile
  • func Create(logger api.Logger, dirPath string, options *Options) (*WriteAheadLogFile, error)
  • func InitializeAndReadAll(logger api.Logger, walDir string, options *Options) (writeAheadLog *WriteAheadLogFile, initialState [][]byte, err error)
  • func Open(logger api.Logger, dirPath string, options *Options) (*WriteAheadLogFile, error)
  • func (w *WriteAheadLogFile) Append(data []byte, truncateTo bool) error
  • func (w *WriteAheadLogFile) CRC() uint32
  • func (w *WriteAheadLogFile) Close() error
  • func (w *WriteAheadLogFile) ReadAll() ([][]byte, error)
  • func (w *WriteAheadLogFile) TruncateTo() error

Constants

const (
	FileSizeBytesDefault   int64 = 64 * 1024 * 1024 // 64MB
	BufferSizeBytesDefault int64 = 1024 * 1024      // 1MB
)

Variables

var (
	ErrCRC       = errors.New("wal: crc verification failed")
	ErrWriteOnly = errors.New("wal: in WRITE mode")
	ErrReadOnly  = errors.New("wal: in READ mode")

	ErrWALAlreadyExists = errors.New("wal: is already exists")
)

Functions

func Repair

func Repair(logger api.Logger, dirPath string) error

Repair tries to repair the last file of a WAL, in case an Open() comes back with a io.ErrUnexpectedEOF, which indicates the possibility to fix the WAL.

After a crash, the last log file in the WAL may be left in a state where an attempt tp reopen will result in the last Read() returning an error. This is because of several reasons: - we use pre-allocated / recycled files, the files have a "garbage" tail - the last write may have been torn - the failure might have occurred when the log file was being prepared (no anchor)

The Repair() tries to repair the last file of the wal by truncating after the last good record. Before doing so it copies the bad file to a side location for later analysis by operators.

logger: reference to a Logger implementation. dirPath: directory path of the WAL. return: an error if repair was not successful.

Types

type LogRecordCRC

type LogRecordCRC uint32

type LogRecordHeader

type LogRecordHeader uint64

LogRecordHeader contains the LogRecordLength (lower 32 bits) and LogRecordCRC (upper 32 bits).

type LogRecordLength

type LogRecordLength uint32

type LogRecordReader

type LogRecordReader struct {
	// contains filtered or unexported fields
}

func NewLogRecordReader

func NewLogRecordReader(logger api.Logger, fileName string) (*LogRecordReader, error)

func (*LogRecordReader) CRC

func (r *LogRecordReader) CRC() uint32

func (*LogRecordReader) Close

func (r *LogRecordReader) Close() error

func (*LogRecordReader) Read

func (r *LogRecordReader) Read() (*protos.LogRecord, error)

type Options

type Options struct {
	FileSizeBytes   int64
	BufferSizeBytes int64
}

func DefaultOptions

func DefaultOptions() *Options

DefaultOptions returns the set of default options.

func (*Options) String

func (o *Options) String() string

type WriteAheadLogFile

type WriteAheadLogFile struct {
	// contains filtered or unexported fields
}

WriteAheadLogFile is a simple implementation of a write ahead log (WAL).

The WAL is composed of a sequence of frames. Each frame contains: - a header (uint64) - payload: a record of type LogRecord, marshaled to bytes, and padded with zeros to 8B boundary.

The 64 bit header is made of two parts: - length of the marshaled LogRecord (not including pad bytes), in the lower 32 bits. - a crc32 of the data: marshaled record bytes + pad bytes, in the upper 32 bits.

The WAL is written to a sequence of files: <index>.wal, where index uint64=1,2,3...; represented in fixed-width hex format, e.g. 0000000000000001.wal

The WAL has two modes: append, and read.

When a WAL is first created, it is in append mode. When an existing WAL is opened, it is in read mode, and will change to append mode only after ReadAll() is invoked.

In append mode the WAL can accept Append() and TruncateTo() calls. The WAL must be closed after use to release all resources.

func Create

func Create(logger api.Logger, dirPath string, options *Options) (*WriteAheadLogFile, error)

Create will create a new WAL, if it does not exist, or an error if it already exists.

logger: reference to a Logger implementation. dirPath: directory path of the WAL. options: a structure containing Options, or nil, for default options.

return: pointer to a WAL, ErrWALAlreadyExists if WAL already exists or other errors

func InitializeAndReadAll

func InitializeAndReadAll(logger api.Logger, walDir string, options *Options) (writeAheadLog *WriteAheadLogFile, initialState [][]byte, err error)

func Open

func Open(logger api.Logger, dirPath string, options *Options) (*WriteAheadLogFile, error)

Open will open an existing WAL, if it exists, or an error if it does not exist.

After opening, the WAL is in read mode, and expects a call to ReadAll(). An attempt to write (e.g. Append, TruncateTo) will result in an error.

logger: reference to a Logger implementation. dirPath: directory path of the WAL. options: a structure containing Options, or nil, for default options.

return: pointer to a WAL, or an error

func (*WriteAheadLogFile) Append

func (w *WriteAheadLogFile) Append(data []byte, truncateTo bool) error

Append a data item to the end of the WAL and indicate whether this entry is a truncation point.

The data item will be added to the log, and internally marked with a flag that indicates whether it is a truncation point. The log implementation may truncate all preceding data items, not including this one.

data: the data to be appended to the log. Cannot be nil or empty. truncateTo: whether all records preceding this one, but not including it, can be truncated from the log.

func (*WriteAheadLogFile) CRC

func (w *WriteAheadLogFile) CRC() uint32

CRC returns the last CRC written to the log file.

func (*WriteAheadLogFile) Close

func (w *WriteAheadLogFile) Close() error

Close the files and directory of the WAL, and release all resources.

func (*WriteAheadLogFile) ReadAll

func (w *WriteAheadLogFile) ReadAll() ([][]byte, error)

ReadAll the data items from the latest truncation point to the end of the log. This method can be called only at the beginning of the WAL lifecycle, right after Open(). After a successful invocation the WAL moves to write mode, and is ready to Append().

In case of failure:

• an error of type io.ErrUnexpectedEOF is returned when the WAL can possibly be repaired by truncating the last log file after the last good record.
• all other errors indicate that the WAL is either
• is closed, or
• is in write mode, or
• is corrupted beyond the simple repair measure described above.

func (*WriteAheadLogFile) TruncateTo

func (w *WriteAheadLogFile) TruncateTo() error

TruncateTo appends a control record in which the TruncateTo flag is true. This marks that every record prior to this one can be safely truncated from the log.