catchup

Documentation

Overview

Package catchup implements log-oriented data processing on top of SQLite. Rows from an append-only table are processed incrementally.

Source table MUST be append-only, with the following exceptions.

1. Rows CAN be deleted, but consumers will NOT be notified, and those rows may or may not have been seen. Deleting rows with id <= minimum of last seen of all consumers SHOULD be done to prevent unlimited database growth.

2. Rows CAN be updated, but consumers will NOT be notified; hence only columns not directly used by consumers are good candidates.

Source table MUST NOT be inserted into concurrently (this can cause ids to show up out of order).

Source tables that see any deletes (including pruning of oldest entries) MUST use AUTOINCREMENT.

Destination table and catchup's internal bookkeeping MUST be protected by the same SQLite savepoint (which roughly means they must be in the same database, and changed through the same database connection).

Index

• type Catchup
  • func New(conf *Config) *Catchup
  • func (c *Catchup) Run(ctx context.Context, fn Func) (err error)
• type Config
• type Func

Types

type Catchup

type Catchup struct {
	// contains filtered or unexported fields
}

func New

func New(conf *Config) *Catchup

func (*Catchup) Run

func (c *Catchup) Run(ctx context.Context, fn Func) (err error)

type Config

type Config struct {
	DB   *database.DB
	Log  *zap.Logger
	Name string
	// MaxSQL is the SQL query to fetch the maximum ID in a source
	// table. The result must have column named max.
	MaxSQL string
	// NextSQL is the SQL query to fetch rows from the source table.
	// The result must have column named id, and should use bind
	// parameters @last and @max to limit the rows.
	NextSQL string
}

type Func

type Func func(conn *sqlite.Conn, stmt *sqlite.Stmt) error

Func is a function that does the actual work. The current source row is accessible via stmt, and all database updates should be done through conn.

The function may be called multiple times for the same input. All side effects must happen either through conn, or be idempotent.