margaret

README

Margaret

Margaret is go-ssb's append-only log* provider, and greatly inspired by flumedb. Compatible with Go 1.13+.

the project name is inspired by Twin Peaks's character Margaret aka the log lady

Margaret has the following facilities:

• an append-only log interface .Append(interface{}), .Get(int64)
• queries .Query(...QuerySpec) for retrieving ranges based on sequence numbers e.g. .Gt(int64), or limiting the amount of data returned .Limit(int64)
• a variety of index mechanisms, both for categorizing log entries into buckets and for creating virtual logs (aka sublogs)

Margaret is one of a few key components that make the go implementation of ssb tick, for example:

• ssb/sbot uses margaret for storing each peer's data

Log storage

Margaret outputs data according to the offset2 format, which is inspired by (but significantly differs from) flumelog-offset.

In brief: margaret stores the data of all logs in the three following files:

• data stores the actual data (with a length-prefix before each entry)
• ofst indexes the starting locations for each data entry in data
• jrnl an integrity checking mechanism for all three files; a checksum of sorts, more details

More details

There are a few concepts that might be tough to digest for newcomers on first approach:

• multilogs, a kind of tree-based index, where each leaf is a margaret.Log
  • in other words: it creates virtual sublogs that map to entries in an offset log (see log storage above)
• margaret/indexes similar to leveldb indexes (arbitrary key-value stores)
• sublogs (and rxLog/receiveLog/offsetLog and its equivalence to offset.log)
• queries
• zeroing out, or replacing, written data

For more on these concepts, visit the dev.scutttlebutt.nz portal for in-depth explanations.

* margaret is technically an append-based log, as there is support for both zeroing out and replacing items in the log after they have been written. Given the relative ubiquity of append-only logs & their uses, it's easier to just say append-only log.

Documentation

Index

• Constants
• Variables
• func IsErrNulled(err error) bool
• func IsOutOfBounds(err error) bool
• type Alterer
• type Codec
• type Decoder
• type Encoder
• type Log
• type Query
• type QuerySpec
  • func ErrorQuerySpec(err error) QuerySpec
  • func Gt(s int64) QuerySpec
  • func Gte(s int64) QuerySpec
  • func Limit(n int) QuerySpec
  • func Live(live bool) QuerySpec
  • func Lt(s int64) QuerySpec
  • func Lte(s int64) QuerySpec
  • func MergeQuerySpec(spec ...QuerySpec) QuerySpec
  • func Reverse(yes bool) QuerySpec
  • func SeqWrap(wrap bool) QuerySpec
• type SeqWrapper
  • func WrapWithSeq(v interface{}, seq int64) SeqWrapper
• type Seqer

Constants

const (
	// SeqEmpty is the current sequence number of an empty log
	SeqEmpty int64 = -1

	// SeqErrored is returned if an operation (like Append) fails
	SeqErrored int64 = -2

	SeqSublogDeleted int64 = -255
)

Variables

var ErrNulled = errors.New("margaret: Entry Nulled")

var OOB oob

OOB is an out of bounds error

Functions

func IsErrNulled

func IsErrNulled(err error) bool

func IsOutOfBounds

func IsOutOfBounds(err error) bool

IsOutOfBounds returns whether a particular error is an out-of-bounds error

Types

type Alterer

type Alterer interface {
	Null(int64) error

	Replace(int64, []byte) error
}

type Codec

type Codec interface {
	// Marshal encodes a single value and returns the serialized byte slice.
	Marshal(value interface{}) ([]byte, error)

	// Unmarshal decodes and returns the value stored in data.
	Unmarshal(data []byte) (interface{}, error)

	NewDecoder(io.Reader) Decoder
	NewEncoder(io.Writer) Encoder
}

Codec marshals and unmarshals values and creates encoders and decoders

type Decoder

type Decoder interface {
	Decode() (interface{}, error)
}

Decoder decodes values

type Encoder

type Encoder interface {
	Encode(v interface{}) error
}

Encoder encodes values

type Log

type Log interface {
	// Seq returns the current sequence number, which is also the number of entries in the log
	Seqer

	// Changes returns an observable that holds the current sequence number
	Changes() luigi.Observable

	// Get returns the entry with sequence number seq
	Get(seq int64) (interface{}, error)

	// Query returns a stream that is constrained by the passed query specification
	Query(...QuerySpec) (luigi.Source, error)

	// Append appends a new entry to the log
	Append(interface{}) (int64, error)
}

Log stores entries sequentially, which can be queried individually using Get or as streams using Query.

type Query

type Query interface {
	// Gt makes the source return only items with sequence numbers > seq.
	Gt(seq int64) error
	// Gte makes the source return only items with sequence numbers >= seq.
	Gte(seq int64) error
	// Lt makes the source return only items with sequence numbers < seq.
	Lt(seq int64) error
	// Lte makes the source return only items with sequence numbers <= seq.
	Lte(seq int64) error
	// Limit makes the source return only up to n items.
	Limit(n int) error

	// Reverse makes the source return the lastest values first
	Reverse(yes bool) error

	// Live makes the source block at the end of the log and wait for new values
	// that are being appended.
	Live(bool) error

	// SeqWrap makes the source return values that contain both the item and its
	// sequence number, instead of the item alone.
	SeqWrap(bool) error
}

Query is the interface implemented by the concrete log implementations that collects the constraints of the query.

type QuerySpec

type QuerySpec func(Query) error

QuerySpec is a constraint on the query.

func ErrorQuerySpec

func ErrorQuerySpec(err error) QuerySpec

ErrorQuerySpec makes the log.Query call return the passed error.

func Gt

func Gt(s int64) QuerySpec

Gt makes the source return only items with sequence numbers > seq.

func Gte

func Gte(s int64) QuerySpec

Gte makes the source return only items with sequence numbers >= seq.

func Limit

func Limit(n int) QuerySpec

Limit makes the source return only up to n items.

func Live

func Live(live bool) QuerySpec

Live makes the source block at the end of the log and wait for new values that are being appended.

func Lt

func Lt(s int64) QuerySpec

Lt makes the source return only items with sequence numbers < seq.

func Lte

func Lte(s int64) QuerySpec

Lte makes the source return only items with sequence numbers <= seq.

func MergeQuerySpec

func MergeQuerySpec(spec ...QuerySpec) QuerySpec

MergeQuerySpec collects several contraints and merges them into one.

func Reverse

func Reverse(yes bool) QuerySpec

func SeqWrap

func SeqWrap(wrap bool) QuerySpec

SeqWrap makes the source return values that contain both the item and its sequence number, instead of the item alone.

type SeqWrapper

type SeqWrapper interface {
	Seqer

	// Value returns the item itself.
	Value() interface{}
}

SeqWrapper wraps a value to attach a sequence number to it.

func WrapWithSeq

func WrapWithSeq(v interface{}, seq int64) SeqWrapper

WrapWithSeq wraps the value v to attach a sequence number to it.

type Seqer

type Seqer interface {
	Seq() int64
}

Seqer returns the current sequence of a log