wal

README

GoLogMatrix

GoLogMatrix: A high-performance Write Ahead Log (WAL) implementation in Go, optimizing sequential data operations for durability, efficiency, and concurrency.

Benchmark Report

System Specifications

• Operating System: Darwin
• Architecture: amd64
• CPU: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz

Write Throughput

• Average Write Throughput: 362748.826031 entries/sec
• Benchmark Results:
  • BenchmarkWriteThroughput-12: 1 iteration, average time/op: 27606327162 ns/op

Read Throughput

Read Throughput

• Read Throughput Data:

  • 1. 73037739.958739 entries/sec
  • 1. 86811862.979975 entries/sec
  • 1. 81466642.664421 entries/sec
  • 1. 79318426.910318 entries/sec
  • 1. 87802423.264348 entries/sec
  • 1. 98858224.089311 entries/sec
  • 1. 87541712.969667 entries/sec
  • 1. 127440391.095189 entries/sec
  • 1. 95307441.840468 entries/sec
  • 1. 102740723.879085 entries/sec
  • 1. 97362184.825342 entries/sec
  • 1. 99515239.375649 entries/sec
  • 1. 104447312.242691 entries/sec
  • 1. 96782439.724453 entries/sec
  • 1. 104312695.747989 entries/sec
  • 1. 99664035.526162 entries/sec
  • 1. 125950175.999627 entries/sec
  • 1. 100666404.551482 entries/sec
  • 1. 103411143.773061 entries/sec
  • 1. 98106134.650179 entries/sec
  • 1. 92956667.710198 entries/sec
  • 1. 107951823.044508 entries/sec
  • 1. 94345075.074810 entries/sec
  • 1. 92782444.373909 entries/sec
  • 1. 120563161.135221 entries/sec
  • 1. 101296592.329960 entries/sec

• Average Read Throughput: 97348448.623486 entries/sec

• Benchmark Result: 10 iterations, average time/op: 100521993 ns/op

Concurrent Write Throughput

• Average Concurrent Write Throughput: 14582.800808 entries/sec
• Benchmark Results:
  • BenchmarkConcurrentWriteThroughPut-12: 1 iteration, average time/op: 68595948875 ns/op

Documentation

Index

• Variables
• func MarshalEntry(entry *WAL_Entry) []byte
• func UnmarshalEntry(data []byte, entry *WAL_Entry)
• func VerifyCRC(entity *WAL_Entry) bool
• type WAL
  • func OpenWAL(directory string, enableFsync bool, maxFileSize int, maxSegments int) (*WAL, error)
  • func (w *WAL) Close() error
  • func (w *WAL) CreateCheckPoint(data []byte) error
  • func (w *WAL) ReadAll(readFromCheckPoint bool) ([]*WAL_Entry, error)
  • func (w *WAL) ReadAllEntriesFromFile(file *os.File, readFromCheckPoint bool) ([]*WAL_Entry, uint64, error)
  • func (w *WAL) ReadAllFromOffset(offset int, readFromCheckPoint bool) ([]*WAL_Entry, error)
  • func (w *WAL) Repair() ([]*WAL_Entry, error)
  • func (w *WAL) Sync() error
  • func (w *WAL) WriteEntity(data []byte) error
• type WAL_Entry
  • func UnmarshalAndVerifyEntry(data []byte) (*WAL_Entry, error)
  • func (*WAL_Entry) Descriptor() ([]byte, []int)
  • func (x *WAL_Entry) GetCRC() uint32
  • func (x *WAL_Entry) GetData() []byte
  • func (x *WAL_Entry) GetIsCheckPoint() bool
  • func (x *WAL_Entry) GetLogSequenceNumber() uint64
  • func (*WAL_Entry) ProtoMessage()
  • func (x *WAL_Entry) ProtoReflect() protoreflect.Message
  • func (x *WAL_Entry) Reset()
  • func (x *WAL_Entry) String() string

Variables

var ErrCRCMismatch = errors.New("CRC mismatch: data may be corrupted")

var File_types_proto protoreflect.FileDescriptor

Functions

func MarshalEntry

func MarshalEntry(entry *WAL_Entry) []byte

MarshalEntry marshals the WAL_Entry into a byte slice.

func UnmarshalEntry

func UnmarshalEntry(data []byte, entry *WAL_Entry)

UnmarshalEntry unmarshal the byte slice into a WAL_Entry.

func VerifyCRC

func VerifyCRC(entity *WAL_Entry) bool

VerifyCRC verifies weather the given entity has the correct CRC.

Types

type WAL

type WAL struct {
	// contains filtered or unexported fields
}

WAL is a write-ahead log that is used to store the data before it is written to any storage It is used to ensure that the data is not lost in case of a crash. It is also used to recover the data in case of a crash using automatic recovery.

func OpenWAL

func OpenWAL(directory string, enableFsync bool, maxFileSize int, maxSegments int) (*WAL, error)

OpenWAL creates a new WAL instance if the directory does not exist. If the directory exists, the last log segment is opened and the last sequence number is read from the segment file. enableFsync enables fsync on the log segment file every time the log flushes. maxFileSize is the maximum size of a log segment file in bytes. maxSegments is the maximum number of log segment files to keep.

func (*WAL) Close

func (w *WAL) Close() error

Close closes the WAL, also calls sync to sync the data from the buffer to the disk

func (*WAL) CreateCheckPoint

func (w *WAL) CreateCheckPoint(data []byte) error

CreateCheckPoint create a check point in the WAL. A check point is a special entry in the WAL that is used to restore the state of the system to a point where the check point was created.

func (*WAL) ReadAll

func (w *WAL) ReadAll(readFromCheckPoint bool) ([]*WAL_Entry, error)

ReadAll reads all the entries from the WAL. if readFromCheckPoint is true, it reads from the check point. it will return all the entries from last checkpoint (if no checkpoint is found) if readFromCheckPoint is false, it reads from the beginning of the WAL.

func (*WAL) ReadAllEntriesFromFile

func (w *WAL) ReadAllEntriesFromFile(file *os.File, readFromCheckPoint bool) ([]*WAL_Entry, uint64, error)

ReadAllEntriesFromFile reads all the entries from the given file. returns all the entries from the file and the last checkpoint log sequence number.

func (*WAL) ReadAllFromOffset

func (w *WAL) ReadAllFromOffset(offset int, readFromCheckPoint bool) ([]*WAL_Entry, error)

ReadAllFromOffset start reading all log segment files from the given offset. (segment Index) and returns all the entries. If readFromCheckpoint is true, it will return all the entries from the last checkpoint (if no checkpoint is found, it will return an empty slice.)

func (*WAL) Repair

func (w *WAL) Repair() ([]*WAL_Entry, error)

func (*WAL) Sync

func (w *WAL) Sync() error

func (*WAL) WriteEntity

func (w *WAL) WriteEntity(data []byte) error

WriteEntity writes the data to WAL

type WAL_Entry

type WAL_Entry struct {
	LogSequenceNumber uint64 `protobuf:"varint,1,opt,name=logSequenceNumber,proto3" json:"logSequenceNumber,omitempty"`
	Data              []byte `protobuf:"bytes,2,opt,name=data,proto3" json:"data,omitempty"`
	CRC               uint32 `protobuf:"varint,3,opt,name=CRC,proto3" json:"CRC,omitempty"`
	// optional field for check pointing
	IsCheckPoint *bool `protobuf:"varint,4,opt,name=isCheckPoint,proto3,oneof" json:"isCheckPoint,omitempty"`
	// contains filtered or unexported fields
}

func UnmarshalAndVerifyEntry

func UnmarshalAndVerifyEntry(data []byte) (*WAL_Entry, error)

UnmarshalAndVerifyEntry unmarshals the data into a WAL_Entry and verifies the CRC.

func (*WAL_Entry) Descriptor

func (*WAL_Entry) Descriptor() ([]byte, []int)

Deprecated: Use WAL_Entry.ProtoReflect.Descriptor instead.

func (*WAL_Entry) GetCRC

func (x *WAL_Entry) GetCRC() uint32

func (*WAL_Entry) GetData

func (x *WAL_Entry) GetData() []byte

func (*WAL_Entry) GetIsCheckPoint

func (x *WAL_Entry) GetIsCheckPoint() bool

func (*WAL_Entry) GetLogSequenceNumber

func (x *WAL_Entry) GetLogSequenceNumber() uint64

func (*WAL_Entry) ProtoMessage

func (*WAL_Entry) ProtoMessage()

func (*WAL_Entry) ProtoReflect

func (x *WAL_Entry) ProtoReflect() protoreflect.Message

func (*WAL_Entry) Reset

func (x *WAL_Entry) Reset()

func (*WAL_Entry) String

func (x *WAL_Entry) String() string