swo

README

Switchover (SWO)

Switchover (SWO) is a feature that allows a live system to switch from one database to another safely and with little to no user impact.

Development

To start the dev instance in switchover mode, run make start-swo

Theory of Operation

Switchover mode is initiated by starting GoAlert with an additional DB URL --db-url-next. The database referenced by --db-url is referred to as the "old" DB and the --db-url-next is the "new" DB.

All new application DB connections first acquire a shared advisory lock, then check the use_next_db pointer. If the pointer is set, all new connections will be made to the "new" DB (without the checking overhead), and the connection to the "old" DB will be terminated.

The switch is performed by first replicating a complete snapshot of the "old" DB to the "new" DB. After the initial sync, subsequent synchronization is an incremental "diff" of snapshots -- more info on how this works is available in the swosync package.

After repeated logical sync operations (to keep the next-sync time low), a stop-the-world lock (i.e., an exclusive lock that conflicts with the shared advisory locks) is acquired, followed by the final logical sync. During the same transaction, the use_next_db pointer is set. After the lock is released, the connector will send all new queries to the "new" DB.

Documentation

Index

• type Config
• type ConnInfo
  • func ParseConnInfo(s string) (*ConnInfo, error)
  • func (c ConnInfo) String() string
• type ConnType
  • func (t ConnType) IsNext() bool
  • func (t ConnType) IsValid() bool
• type Connector
  • func NewConnector(dbcOld, dbcNew driver.Connector) *Connector
  • func (drv *Connector) Connect(ctx context.Context) (driver.Conn, error)
  • func (drv *Connector) Driver() driver.Driver
• type Executor
  • func NewExecutor(mgr *Manager) *Executor
  • func (e *Executor) Cancel()
  • func (e *Executor) Exec(ctx context.Context) error
  • func (e *Executor) Sync(ctx context.Context) error
• type Manager
  • func NewManager(cfg Config) (*Manager, error)
  • func (m *Manager) ConnInfo(ctx context.Context) (counts []swoinfo.ConnCount, err error)
  • func (m *Manager) DB() *sql.DB
  • func (m *Manager) Reset(ctx context.Context) error
  • func (m *Manager) SetPauseResumer(app lifecycle.PauseResumer)
  • func (m *Manager) StartExecute(ctx context.Context) error
  • func (m *Manager) Status() Status
• type Node
• type Status
• type WithFunc
  • func NewWithFunc[V any](withFn func(ctx context.Context, useFn func(V)) error) *WithFunc[V]
  • func (w *WithFunc[V]) Begin(ctx context.Context) (v V, err error)
  • func (w *WithFunc[V]) Cancel()

Types

type Config

type Config struct {
	OldDBURL, NewDBURL string
	CanExec            bool
	Logger             *log.Logger
}

Config configures the current node for SWO.

type ConnInfo

type ConnInfo struct {
	Version string
	Type    ConnType
	ID      uuid.UUID
}

ConnInfo contains information about a connection to the DB for SWO.

This is stored as the `application_name` for a connection in Postgres in the format of "GoAlert <version> SWO:<type>:<id>" where id is a base64 encoded UUID that should match what ends up in a `switchover_log` hello message.

func ParseConnInfo

func ParseConnInfo(s string) (*ConnInfo, error)

ParseConnInfo parses a connection string into a ConnInfo.

func (ConnInfo) String

func (c ConnInfo) String() string

String returns a string representation of the ConnInfo.

type ConnType

type ConnType byte

ConnType indicates a type of SWO connection.

const (
	// ConnTypeMainMgr is the connection pool to the main/old DB used to coordinate the switchover.
	ConnTypeMainMgr ConnType = iota + 'A'

	// ConnTypeMainApp is the connection pool used by the GoAlert application to the main/old DB.
	//
	// Connections here are protected with a shared advisory lock.
	ConnTypeMainApp

	// ConnTypeNextMgr is the connection pool to the next/new DB used for applying changes during the switchover.
	ConnTypeNextMgr

	// ConnTypeNextApp is the connection pool used by the GoAlert application to the next/new DB, after the switchover is completed.
	ConnTypeNextApp
)

func (ConnType) IsNext

func (t ConnType) IsNext() bool

IsNext returns true if the connection is for the next DB.

func (ConnType) IsValid

func (t ConnType) IsValid() bool

IsValid returns true if the ConnType is valid.

type Connector

type Connector struct {
	// contains filtered or unexported fields
}

Connector is a driver.Connector that will use the old database until the switchover_state table indicates that the new database should be used.

Until the switchover is complete, the old database will be protected with a shared advisory lock (4369).

func NewConnector

func NewConnector(dbcOld, dbcNew driver.Connector) *Connector

NewConnector creates a new Connector pointing to the old and new DBs, respectively.

func (*Connector) Connect

func (drv *Connector) Connect(ctx context.Context) (driver.Conn, error)

Connect returns a new connection to the database.

A shared advisory lock is acquired on the connection to the old DB, and then switchover_state is checked.

When `current_state` is `use_next_db`, the connection is closed and a connection to the new DB is returned instead. Future connections then skip the check and are returned directly from the new DB.

func (*Connector) Driver

func (drv *Connector) Driver() driver.Driver

Driver is a stub method for driver.Connector that returns nil.

type Executor

type Executor struct {
	// contains filtered or unexported fields
}

Executor is responsible for executing the switchover process.

func NewExecutor

func NewExecutor(mgr *Manager) *Executor

NewExecutor initializes a new Executor for the given Manager.

func (*Executor) Cancel

func (e *Executor) Cancel()

Cancel cancels the switchover process.

func (*Executor) Exec

func (e *Executor) Exec(ctx context.Context) error

Exec executes the switchover process, blocking until it is complete.

func (*Executor) Sync

func (e *Executor) Sync(ctx context.Context) error

Sync begins the switchover process by resetting and starting the logical replication process.

type Manager

type Manager struct {
	Config

	MainDBInfo *swoinfo.DB
	NextDBInfo *swoinfo.DB
	// contains filtered or unexported fields
}

A Manager is responsible for managing the switchover process.

func NewManager

func NewManager(cfg Config) (*Manager, error)

NewManager will create a new Manager with the given configuration.

func (*Manager) ConnInfo

func (m *Manager) ConnInfo(ctx context.Context) (counts []swoinfo.ConnCount, err error)

ConnInfo returns information about all current DB connections.

func (*Manager) DB

func (m *Manager) DB() *sql.DB

DB returns a sql.DB that will always return safe connections to be used during the switchover.

All application code/queries should use this DB.

func (*Manager) Reset

func (m *Manager) Reset(ctx context.Context) error

Reset will disable the changelog and reset the cluster state.

func (*Manager) SetPauseResumer

func (m *Manager) SetPauseResumer(app lifecycle.PauseResumer)

SetPauseResumer allows setting the pause/resume functionality for the manager.

Pause is called during the switchover process to minimize the number of long-lived DB connections so that the final sync can be performed quickly.

After a switchover, or if it is aborted, Resume will be called.

func (*Manager) StartExecute

func (m *Manager) StartExecute(ctx context.Context) error

StartExecute will trigger the switchover to begin.

func (*Manager) Status

func (m *Manager) Status() Status

Status will return the current switchover status.

type Node

type Node struct {
	ID uuid.UUID

	// OldValid indicates that the old database config is valid.
	OldValid bool

	// NewValid indicates that the new database config is valid.
	NewValid bool

	// CanExec indicates the node is NOT in API-only mode and is capable of executing tasks.
	CanExec bool

	Status string
}

Node contains information on a GoAlert instance in SWO mode.

type Status

type Status struct {
	swogrp.Status

	MainDBID      uuid.UUID
	NextDBID      uuid.UUID
	MainDBVersion string
	NextDBVersion string
}

Status represents the current status of the switchover process.

type WithFunc

type WithFunc[V any] struct {
	// contains filtered or unexported fields
}

WithFunc flattens a with-type func providing it's value with Begin() and ending with Cancel().

func NewWithFunc

func NewWithFunc[V any](withFn func(ctx context.Context, useFn func(V)) error) *WithFunc[V]

NewWithFunc creates a new WithFunc.

withFn must not return an error after useFn is called.

func (*WithFunc[V]) Begin

func (w *WithFunc[V]) Begin(ctx context.Context) (v V, err error)

Begin will return a new instance of V, Cancel should be called when it's no longer needed.

If err is nil, Cancel must be called before calling Begin again.

func (*WithFunc[V]) Cancel

func (w *WithFunc[V]) Cancel()

Cancel will cancel context passed to withFn and wait for it to finish.