toylog

README

Toy chunked log

This is a very basic implementation of a chunked log. It creates a new chunk once the old one reaches the size limit. Once the chunk count limit is reached, old chunks expire.

A ChunkedLogReader can seek within the chunks and change chunks automatically providing a regular ReadSeeker API.

That's all the lib does.

Documentation

Index

• Constants
• Variables
• type ChunkedLog
  • func (log *ChunkedLog) ChunkCount() int
  • func (log *ChunkedLog) Close() error
  • func (log *ChunkedLog) CurrentSize() int64
  • func (log *ChunkedLog) Drain(recs toyqueue.Records) (err error)
  • func (log *ChunkedLog) ExpiredSize() int64
  • func (log *ChunkedLog) Locate(pos int64, whence int) (fd int, off, fpos int64, err error)
  • func (log *ChunkedLog) Open(dir string) (err error)
  • func (log *ChunkedLog) Reader(pos int64, whence int) (*ChunkedLogReader, error)
  • func (log *ChunkedLog) Sync() error
  • func (log *ChunkedLog) TotalSize() int64
  • func (log *ChunkedLog) Write(rec []byte) (n int, err error)
• type ChunkedLogReader
  • func (r *ChunkedLogReader) Close() (err error)
  • func (r *ChunkedLogReader) Read(into []byte) (n int, err error)
  • func (r *ChunkedLogReader) Seek(offset int64, whence int) (int64, error)

Constants

const (
	ChunkSeekStart   = 3 // seek relative to the origin of the file
	ChunkSeekCurrent = 4 // seek relative to the current offset
	ChunkSeekEnd     = 5 // seek relative to the end
)

const Suffix = ".log.chunk"

Variables

var ErrAlreadyOpen = errors.New("the log is already open")

var ErrChunkMissing = errors.New("the requested chunk is missing")

var ErrNotOpen = errors.New("the log is not open")

var ErrOmission = errors.New("the log has missing chunks")

var ErrOutOfRange = errors.New("the requested offset is out of range")

var ErrOverlap = errors.New("the log has overlapping chunks")

var ErrSeekModeUnsupported = errors.New("seek mode not supported")

Types

type ChunkedLog

type ChunkedLog struct {
	// Chunk size never exceeds this number (can not make a bigger write)
	MaxChunkSize int64
	// Old chunks above this number are dropped
	MaxChunks int
	// whether all writes are `fsynced` *before* `Write()` returns
	Synced bool
	// header maker for new chunks
	Header toyqueue.Feeder
	// contains filtered or unexported fields
}

Concurrent access guarantees: one writer, many readers.

func (*ChunkedLog) ChunkCount

func (log *ChunkedLog) ChunkCount() int

func (*ChunkedLog) Close

func (log *ChunkedLog) Close() error

func (*ChunkedLog) CurrentSize

func (log *ChunkedLog) CurrentSize() int64

func (*ChunkedLog) Drain

func (log *ChunkedLog) Drain(recs toyqueue.Records) (err error)

We expect a dense stream of tiny ops here, so we bundle writes here. Note: a slice goes into one chunk of the log, no torn writes.

func (*ChunkedLog) ExpiredSize

func (log *ChunkedLog) ExpiredSize() int64

func (*ChunkedLog) Locate

func (log *ChunkedLog) Locate(pos int64, whence int) (fd int, off, fpos int64, err error)

The returned fd has to be closed!

func (*ChunkedLog) Open

func (log *ChunkedLog) Open(dir string) (err error)

func (*ChunkedLog) Reader

func (log *ChunkedLog) Reader(pos int64, whence int) (*ChunkedLogReader, error)

func (*ChunkedLog) Sync

func (log *ChunkedLog) Sync() error

func (*ChunkedLog) TotalSize

func (log *ChunkedLog) TotalSize() int64

func (*ChunkedLog) Write

func (log *ChunkedLog) Write(rec []byte) (n int, err error)

type ChunkedLogReader

type ChunkedLogReader struct {
	// contains filtered or unexported fields
}

func (*ChunkedLogReader) Close

func (r *ChunkedLogReader) Close() (err error)

func (*ChunkedLogReader) Read

func (r *ChunkedLogReader) Read(into []byte) (n int, err error)

func (*ChunkedLogReader) Seek

func (r *ChunkedLogReader) Seek(offset int64, whence int) (int64, error)