gooseplus

README

gooseplus

Goose DB migrations, on steroids.

Purpose

gooseplus extends the great DB migration tool goose to support a few advanced use cases:

1. Leaves a failed deployment in its initial state: upon failure, rollbacks migrations back to when the deployment started
2. Support environment-specific migrations, so we can add migrations for tests, etc.
3. A global locking mechanism to run migrations once, even in a parallel deployment
4. More options: structured zap logger, fined-grained timeouts ...

gooseplus is primarily intended to be used as a library, and does not come with a CLI command.

Usage

    db, _ := sql.Open("postgres", "test")
    migrator := New(db)

    err := migrator.Migrate(context.Background())
    ...

Feel free to look at the various examples.

Features

• Everything goose/v3 does out of the box.
• Rollback to the state at the start of the call to Migrate() after a failure.
• Environment-specific migration folders
• Global lock table

Concepts

Defaults

I've tried to define sensible defaults as follows:

• default DB driver: postgres (like goose)
• default base path for migrations: sql
• default FS: os.DirFS(".")
• default timeout on the whole migration process: 5m
• default timeout on any single migration: 1m

Environment-specific folders

Migrations are stored in a base directory as a linear sequence of SQL scripts or go migration programs.

In this directory, the base folder contains migrations that apply to all environments.

Additional folders may be defined to run migrations for specific environments (i.e. specific deployment contexts).

This comes in handy in situations where we want data initialization scripts (not just schema changes) to run under different environments.

Example:

sql/base/
sql/base/20231103204811_populate_example.sql
sql/base/20231102204811_create_example.sql

sql/production/
sql/production/20231103204911_populate_prod.sql

sql/test/
sql/test/20231103204911_populate_test.sql

You can change the base folder by setting the new list of folders: SetEnvironments([]string{"default", "production", "test").

If you don't want to manage sub-folders at all, you can disable it with the option SetEnvironments(nil). In this case, no base folder will be used.

Attention point: if you use go migrations these folders become go packages, and folder names should not be reserved names with a special meaning for golang. Hence default, xxx_test are names to be avoided for package names.

Embedded file system

goose/v3 supports embedded file systems at build time.

You can use it with gooseplus like so:

	//go:embed sql/*/*.sql
	var embedMigrations embed.FS

	db, _ := sql.Open("postgres", "test")

	migrator := gooseplus.New(
		db,
		gooseplus.WithFS(embedMigrations),
	)

Global lock

This is enabled with option WithGlobalLock(true). It doesn't work with non-locking environments, such as sqlite3. It should work with postgres and mysql.

An additional technical table, goose_db_version_lock is created to hold the outcome of the currently running migration. Note that if you change the name of the version table, the lock table is created as {goose version table name}_lock.

Whenever a process or go routine starts the migration process, a single record in this table is created and locked until the migration completes.

• any other competing migration process would wait until the lock is released
• if the migration fails, it rolls back to the starting point before deployment. Other instances will resume from there and try again.

Example: let's suppose that a deployment starts 3 instances of a service that starts by applying DB migrations.

The first deployment acquires the lock, then runs the migrations. Other instances also attempt to run migrations, and wait on the lock. When the first deployment is done, the lock is released. The other deployments, one by one, acquire a lock, verify that the current version is up to date and release the lock.

If the migration process of the first deployment failed at some point, the other ones attempt to run the sequence again.

Attention point: since migrations wait each other, make sure the global timeout can support this waiting.

Logging

gooseplus injects a structured zap logger from go.uber.org/zap

Caveats

• Concurrent usage is not supported: goose/v3 relies on a lot of globals. Migrations should normally run once.
• Minimal locking has ben added so you can run your tests with -race

Documentation

Overview

Package gooseplus provides a configurable DB Migrator.

It extends github.com/pressly/goose/v3 with the following features: * rollbacks to the initial state of a deployment whenever a migration fails in a sequence of several migrations * supports multiple-environments, so that it is possible to define environment-specific migrations * supports options: such as structured logging with zap.

Index

• Variables
• type Migrator
  • func New(db *sql.DB, opts ...Option) *Migrator
  • func (m Migrator) Migrate(parentCtx context.Context) error
• type Option
  • func SetEnvironments(envs []string) Option
  • func WithBasePath(base string) Option
  • func WithDialect(dialect string) Option
  • func WithEnvironments(envs ...string) Option
  • func WithFS(fsys fs.FS) Option
  • func WithGlobalLock(enabled bool) Option
  • func WithLogger(zlg *zap.Logger) Option
  • func WithMigrationTimeout(timeout time.Duration) Option
  • func WithTimeout(timeout time.Duration) Option
  • func WithVersionTable(table string) Option

Examples

• Migrator

Variables

var (
	ErrMigrationTable  = errors.New("could not ensure goose migration table")
	ErrMergeMigrations = errors.New("error merging migrations")
	ErrRollForward     = errors.New("error rolling forward migrations. Recovered error: the db has been rollbacked to its initial state")
	ErrRollBack        = errors.New("error rolling back migrations")
)

Errors returned by the Migrator

Types

type Migrator

type Migrator struct {
	DB *sql.DB
	// contains filtered or unexported fields
}

Migrator knows how to apply changes (migrations) to a versioned database schema.

By default, the migrator will run migrations from the "base" environment folder.

If extra environments are added using options, the migrator will merge the migrations with the other folders corresponding to these environments.

Example

package main

import (
	"context"
	"database/sql"
	"embed"
	"log"
	"os"
	"path/filepath"

	"github.com/fredbi/gooseplus"
	"go.uber.org/zap"

	// registers go migrations for unit tests
	_ "github.com/fredbi/gooseplus/test_sql/unittest"
	_ "github.com/fredbi/gooseplus/test_sql/unittest3"

	// init driver
	_ "github.com/mattn/go-sqlite3"
)

//go:embed test_sql/*/*.sql
//go:embed test_sql/*/*.go
var embedMigrations embed.FS

func main() {
	const (
		dir            = "exampledata"
		driver         = "sqlite3"
		migrationsRoot = "test_sql"
	)

	if err := os.MkdirAll(dir, 0700); err != nil {
		log.Println(err)

		return
	}

	defer func() {
		_ = os.RemoveAll(dir)
	}()

	tempDB, err := os.MkdirTemp(dir, "db")
	if err != nil {
		log.Println(err)

		return
	}

	db, err := sql.Open("sqlite3", filepath.Join(tempDB, "example.db"))
	if err != nil {
		log.Println(err)

		return
	}

	zlg := zap.NewExample()

	migrator := gooseplus.New(db,
		gooseplus.WithDialect(driver),
		gooseplus.WithFS(embedMigrations),
		gooseplus.WithBasePath(migrationsRoot),
		gooseplus.WithLogger(zlg),
	)

	if err := migrator.Migrate(context.Background()); err != nil {
		log.Println(err)

		return
	}
}

func New

func New(db *sql.DB, opts ...Option) *Migrator

New migrator with options.

func (Migrator) Migrate

func (m Migrator) Migrate(parentCtx context.Context) error

Migrate applies a sequence database migrations, provided either as SQL scripts or go migration functions.

Whenever a migration fails, Migrate applies rollbacks back to the initial state before returning an error.

type Option

type Option func(*options)

Option for the Migrator.

Default settings are:

dialect: "postgres",
base:    "sql",
envs: []string{"base"},
timeout: 5 * time.Minute,
migrationTimeout: 1 * time.Minute,
logger:  zap.NewExample(),
fsys:    os.DirFS("."),

func SetEnvironments

func SetEnvironments(envs []string) Option

SetEnvironments overrides environment-specific folders to merge with the migrations.

Setting to nil or to an empty slice will disable folders: migrations will be searched for in the base path only.

func WithBasePath

func WithBasePath(base string) Option

WithBasePath provides the root directory where migrations are located on the FS.

func WithDialect

func WithDialect(dialect string) Option

WithDialect indicates the database SQL dialect.

For details see https://pkg.go.dev/github.com/pressly/goose/v3#Dialect

func WithEnvironments

func WithEnvironments(envs ...string) Option

WithEnvironments appends environment-specific folders to merge with the migrations.

The default setting is a single folder "base".

func WithFS

func WithFS(fsys fs.FS) Option

WithFS provides the file system where migrations are located.

The base is os.Dir(".").

func WithGlobalLock

func WithGlobalLock(enabled bool) Option

WithGlobalLock prevent several migrations to run in parallel by applying a global lock.

The default is false, meaning that deploying several instances in parallel won't work well.

func WithLogger

func WithLogger(zlg *zap.Logger) Option

WithLogger provides a structured zap logger to the migrator.

func WithMigrationTimeout

func WithMigrationTimeout(timeout time.Duration) Option

WithMigrationTimeout specifies a timeout to apply for each individual migration.

The zero value disables the timeout.

Default is 1m.

func WithTimeout

func WithTimeout(timeout time.Duration) Option

WithTimeout specifies a timeout to apply to the whole migration process.

NOTE: if Migrate(ctx) is called with a context that already contains a deadline, that deadline will override this option.

The zero value disables the timeout.

Default is 5m.

func WithVersionTable

func WithVersionTable(table string) Option

WithVersionTable tells goose to use an non-default version table.

The default is "". Setting an empty table equates to using the default.