journal

Documentation

Overview

Package journal implements WAL-like append-only journals. A journal is split into segments; the last segment is the one being written to.

Intended use cases:

• Database WAL files.
• Log files of various kinds.
• Archival of historical database records.

Features:

• Suitable for a large number of very short records. Per-record overhead can be as low as 2 bytes.

• Suitable for very large records, too. (In the future, it will be possible to write records in chunks.)

• Fault-resistant.

• Self-healing. Verifies the checksums and truncates corrupted data when opening the journal.

• Performant.

• Automatically rotates the files when they reach a certain size.

TODO:

• Trigger rotation based on time (say, each day gets a new segment). Basically limit how old in-progress segments can be.

• Allow to rotate a file without writing a new record. (Otherwise rarely-used journals will never get archived.)

• Give work-in-progress file a prefixed name (W*).

• Auto-commit every N seconds, after K bytes, after M records.

• Option for millisecond timestamp precision?

• Reading API. (Search based on time and record ordinals.)

• Use mmap for reading.

File format

Segment files:

• file = segmentHeader item*
• segmentHeader = (see struct)
• item = record | commit
• record = (size << 1):uvarint timestampDelta:uvarint bytes*
• commit = checksum_with_bit_0_set:64

We always set bit 0 of commit checksums, and we use size*2 when encoding records; so bit 0 of the first byte of an item indicates whether it's a record or a commit.

Timestamps are 32-bit unix times and have 1 second precision. (Rationale is that the primary use of timestamps is to search logs by time, and that does not require a higher precision. For high-frequency logs, with 1-second precision, timestamp deltas will typically fit within 1 byte.)

Index

• Constants
• Variables
• type Journal
  • func New(dir string, o Options) *Journal
  • func (j *Journal) Commit() error
  • func (j *Journal) FinishWriting() error
  • func (j *Journal) Now() uint32
  • func (j *Journal) StartWriting()
  • func (j *Journal) String() string
  • func (j *Journal) WriteRecord(timestamp uint32, data []byte) error
• type Options

Constants

const DefaultMaxFileSize = 4 * 1024 * 1024

Variables

var (
	ErrIncompatible       = fmt.Errorf("incompatible journal")
	ErrUnsupportedVersion = fmt.Errorf("unsupported journal version")
)

Types

type Journal

type Journal struct {
	// contains filtered or unexported fields
}

func New

func New(dir string, o Options) *Journal

func (*Journal) Commit

func (j *Journal) Commit() error

func (*Journal) FinishWriting

func (j *Journal) FinishWriting() error

func (*Journal) Now

func (j *Journal) Now() uint32

func (*Journal) StartWriting

func (j *Journal) StartWriting()

func (*Journal) String

func (j *Journal) String() string

func (*Journal) WriteRecord

func (j *Journal) WriteRecord(timestamp uint32, data []byte) error

type Options

type Options struct {
	FileName         string // e.g. "mydb-*.bin"
	MaxFileSize      int64  // new segment after this size
	DebugName        string
	Now              func() time.Time
	JournalInvariant [32]byte
	SegmentInvariant [32]byte

	Context context.Context
	Logger  *slog.Logger
	OnLoad  func()
	Verbose bool
}