Documentation
¶
Index ¶
- Constants
- Variables
- func WriteFileHeader(sig []byte, header FileHeader, writer io.Writer) (int, error)
- type BasicFileHeader
- type BasicIndexRecord
- type BasicLogRecord
- type BasicSnapshot
- type Config
- type FileHeader
- type IndexFactory
- type IndexRecord
- type IndexRecordDecoder
- type IndexRecordEncoder
- type IndexSlice
- type LogCursor
- type LogIndex
- type LogRecord
- type LogRecordDecoder
- type LogRecordEncoder
- type Metadata
- type Snapshot
- type State
- type WriteAheadLog
Constants ¶
const ( // - `DefaultIndexFlags` is the default boolean flags for an index file. DefaultIndexFlags = 0 // - `DefaultRecordFlags` represents the default boolean flags for each log record. DefaultRecordFlags uint32 = 0 // - `DefaultMaxRecordSize` is the default maximum size of a log record. DefaultMaxRecordSize = 0xffff // - `LogHeaderSize` is the size of the file header. LogHeaderSize = 8 // - `MaximumIndexSlice` is the maximum number of index records to be read at // one time MaximumIndexSlice = 32000 )
Variables ¶
var ( // LogFileSignature is the first 3 bytes of a log file - `LOG` LogFileSignature = []byte("LOG") // IndexFileSignature represents the first 3 bytes of an index file - `IDX` IndexFileSignature = []byte("IDX") )
## **Log Variables**
var ( // ErrReadFileHeader occurs when the file header cannot be read ErrReadFileHeader = errors.New("failed to read file header") // - `ErrReadIndexHeader` occurs when the index header cannot be read ErrReadIndexHeader = errors.New("failed to read index header") // - `ErrWriteIndexHeader` occurs when the index header cannot be written ErrWriteIndexHeader = errors.New("failed to write index header") // - `ErrReadLogHeader` occurs when the log header cannot be read ErrReadLogHeader = errors.New("failed to read log header") // - `ErrWriteLogHeader` occurs when the log header cannot be written ErrWriteLogHeader = errors.New("failed to write log header") // - `ErrSliceOutOfBounds` occurs when index.Slice is called and the offset // is larger than the size of the index. ErrSliceOutOfBounds = errors.New("read offset out of bounds") // - `ErrReadIndex` occurs when index.Slice fails to read the records ErrReadIndex = errors.New("failed to read index records") // - `ErrConfigRequired` occurs when no log config is given when creating // or opening a log file. ErrConfigRequired = errors.New("log config required") // - `ErrInvalidFileVersion` occurs when the version in the file header // is unrecognized. ErrInvalidFileVersion = errors.New("invalid file version") // - `ErrInvalidFileSignature` occurs when the signature in the file header // is unrecognized. ErrInvalidFileSignature = errors.New("invalid file signature") // - `ErrWriteLogRecord` occurs when a record fails to be written to the log ErrWriteLogRecord = errors.New("failed to write record") // - `ErrReadLogRecord` occurs when a record fails to be read from the log ErrReadLogRecord = errors.New("failed to read record") // ErrReadIndexRecord occurs when a record fails to be read from the index ErrReadIndexRecord = errors.New("failed to read index record") // - `ErrLogAlreadyOpen` occurs when an open log tries to be opened again ErrLogAlreadyOpen = errors.New("log already open") // - `ErrLogClosed` occurs when `Append` is called after the log has been // closed. ErrLogClosed = errors.New("log has been closed") // `ErrRecordTooLarge` occurs when writing a record which exceed the max // record size for the log. ErrRecordTooLarge = errors.New("record is too large") // ErrExceedsBufferSize occurs when the BufferedWriter is not large enough // to contain all the data being written to it. ErrExceedsBufferSize = errors.New("buffer too large") // ErrInvalidRecordSize ErrInvalidRecordSize = errors.New("invalid record size") // ErrInvalidSnapshot occurs when a Snapshot cannot be decoded. ErrInvalidSnapshot = errors.New("invalid snapshot") // ErrInvalidTTL occurs when the `config.TimeToLive` is less than 0 ErrInvalidTTL = errors.New("invalid ttl; must be >= 0") // ErrInvalidLogStrategy occurs when the `config.Strategy` is nil ErrInvalidLogStrategy = errors.New("invalid write strategy") // ErrRecordFactorySize ErrRecordFactorySize = errors.New("invalid record factory; max record size exceeded") )
Functions ¶
func WriteFileHeader ¶
Types ¶
type BasicFileHeader ¶
type BasicFileHeader struct {
// contains filtered or unexported fields
}
BasicFileHeader is the simplest implementation of the FileHeader interface.
func (BasicFileHeader) Expiration ¶
func (i BasicFileHeader) Expiration() int64
Expiration returns the duration at which log records expire
func (BasicFileHeader) Flags ¶
func (i BasicFileHeader) Flags() uint32
Flags returns the boolean flags for the file.
func (BasicFileHeader) Version ¶
func (i BasicFileHeader) Version() uint8
Version returns the file version.
type BasicIndexRecord ¶
type BasicIndexRecord struct {
// contains filtered or unexported fields
}
BasicIndexRecord implements the bare IndexRecord interface.
func (BasicIndexRecord) Index ¶
func (i BasicIndexRecord) Index() uint64
Index is a record's numerical id.
func (BasicIndexRecord) IsExpired ¶
func (i BasicIndexRecord) IsExpired(now, ttl int64) bool
IsExpired is a helper function for the index record which returns `true` if the current time is beyond the expiration time. The expiration time is calculated as written time + TTL.
func (BasicIndexRecord) Offset ¶
func (i BasicIndexRecord) Offset() int64
Offset is the distance the data record is from the start of the data file.
func (BasicIndexRecord) Time ¶
func (i BasicIndexRecord) Time() int64
Time returns when the record was written to the data file.
type BasicLogRecord ¶
type BasicLogRecord struct {
// contains filtered or unexported fields
}
BasicLogRecord represents an element in the log. Each record has a timestamp, an index, boolean flags, a length and the data.
func (BasicLogRecord) Data ¶
func (r BasicLogRecord) Data() []byte
Data returns the associated byte buffer.
func (BasicLogRecord) Flags ¶
func (r BasicLogRecord) Flags() uint32
Flags returns boolean fields associated with the record.
func (BasicLogRecord) IsExpired ¶
func (r BasicLogRecord) IsExpired(now, ttl int64) bool
IsExpired determines if the record is expired.
func (BasicLogRecord) Size ¶
func (r BasicLogRecord) Size() uint32
Size returns the length of the record's data.
func (BasicLogRecord) Time ¶
func (r BasicLogRecord) Time() int64
Time represents when the record was created.
type BasicSnapshot ¶
type BasicSnapshot struct {
// contains filtered or unexported fields
}
BasicSnapshot represents the simplest snapshot which fulfills the Snapshot interface. The timestamp is stored as nanoseconds since epoch. Both size and hash are stored as 64-bit integers.
func (BasicSnapshot) Hash ¶
func (b BasicSnapshot) Hash() uint64
Hash returns the XXH64 hash of the log file.
func (BasicSnapshot) MarshalBinary ¶
func (b BasicSnapshot) MarshalBinary() ([]byte, error)
MarshalBinary converts the snapshot into a byte array. The byte array is formatted like so:
``` 8-byte int64 time 8-byte uint64 size 8-byte uint64 hash
0 8 16 24 +--------+--------+--------+ | time | size | hash | +--------+--------+--------+ ```
func (BasicSnapshot) Size ¶
func (b BasicSnapshot) Size() int64
Size returns the number of records in the log at the time the snapshot was taken.
func (BasicSnapshot) Time ¶
func (b BasicSnapshot) Time() time.Time
Time converts the nanoseconds since epoch into a `time.Time` instance.
type Config ¶
type Config struct {
FileMode os.FileMode
MaxRecordSize int
Flags uint32
Version uint8
Truncate bool
TimeToLive int64
Strategy m3.WriteStrategy
}
Config stores several log settings. This is used to describe how the log should be opened.
type FileHeader ¶
FileHeader describes which version the file was written with. Flags represents boolean flags.
func NewFileHeader ¶
func NewFileHeader(version uint8, flags uint32, expiration int64) FileHeader
NewFileHeader creates a new FileHeader instance
func ReadFileHeader ¶
func ReadFileHeader(reader io.Reader) (FileHeader, error)
ReadFileHeader creates a FileHeader from an io.Reader. Presumably this reader would be a file.
type IndexFactory ¶
IndexFactory creates an index based on the filename, version and flags given
type IndexRecord ¶
type IndexRecord interface {
Time() int64
Index() uint64
Offset() int64
IsExpired(now, ttl int64) bool
}
IndexRecord describes each item in an index file.
func NewIndexRecord ¶
func NewIndexRecord(nanos, offset int64, index uint64) IndexRecord
NewIndexRecord creates a new index record.
type IndexRecordDecoder ¶
type IndexRecordDecoder func() (IndexRecord, error)
IndexRecordDecoder reads an `IndexRecord` from a bute array.
type IndexRecordEncoder ¶
type IndexRecordEncoder func(record IndexRecord) (int, error)
IndexRecordEncoder writes an `IndexRecord` into a byte array.
type IndexSlice ¶
type IndexSlice interface {
Get(index int) (IndexRecord, error)
Size() int
}
IndexSlice contains several buffered index records for fast access.
type LogCursor ¶
type LogCursor interface {
// Seek moves the Cursor to the given record index.
Seek(offset uint64) (LogRecord, error)
// Next moves the Cursor forward one record.
Next() (LogRecord, error)
// Close cursor and any associates file handles
Close() error
}
LogCursor allows for quite navigation through the log. All Cursor start at zero
and moves forward until EOF.
type LogIndex ¶
type LogIndex interface {
io.WriteCloser
Size() uint64
Header() FileHeader
}
LogIndex maintains a list of all the records in the log file. IndexRecords
type LogRecord ¶
type LogRecord interface {
// Size returns the size of the record data
Size() uint32
// Flags returns any boolean flags associated
Flags() uint32
// Time returns the record timestamp
Time() int64
// IsExpired returns whether the record has expired based on the log's ttl
IsExpired(now, ttl int64) bool
// Data returns record data
Data() []byte
}
LogRecord describes a single item in the log file. It consists of a time, an index id, a length and the data.
type LogRecordDecoder ¶
LogRecordDecoder reads a record from the given reader.
type LogRecordEncoder ¶
LogRecordEncoder writes a new record into the given writer.
type Snapshot ¶
Snapshot captures a specific state of the log. It consists of the time the snapshot was taken, the number of items in the log, and a XXH64 hash of all the log entries.
func NewSnapshot ¶
func UnmarshalShapshot ¶
UnmarshalSnapshot is a utility function which converts a byte array into a snapshot. If the byte array is too small, a `ErrInvalidSnapshot` is returned.
type State ¶
type State uint8
State is used to maintain the current status of the log.
const CLOSED State = 2
`CLOSED` represents a closed log file
const OPEN State = 1
`OPEN` signifies the log is currently open
const UNOPENED State = 0
type WriteAheadLog ¶
type WriteAheadLog interface {
io.WriteCloser
// Recover should be called when the log is opened to verify consistency
// of the log.
Recover() error
// Creates a new Cursor initialized at index 0
Cursor() (LogCursor, error)
// Snapshot records the current position of the log file.
Snapshot() (Snapshot, error)
// Metadata returns metadata of the log file.
Metadata() (Metadata, error)
}
WriteAheadLog implements an immutable write-ahead log file with indexes and snapshots.
Source Files