golembic

package module
v0.0.0-...-4d8b559 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 14, 2024 License: Apache-2.0 Imports: 9 Imported by: 1

README

golembic

SQL Schema Management in Go, inspired by sqlalchemy/alembic

GoDoc

Usage

If a *golembic.Migrations sequence has been formed, then a binary can be created as follows:

func run() error {
	cmd, err := command.MakeRootCommand(examples.AllMigrations)
	if err != nil {
		return err
	}

	return cmd.Execute()
}

func main() {
	err := run()
	if err != nil {
		fmt.Fprintf(os.Stderr, "%v\n", err)
		os.Exit(1)
	}
}

NOTE: For usage in Go code (vs. as a binary), see examples/postgres-script/main.go.

The root command of this binary has a subcommand for each provider

$ go build -o golembic ./examples/cmd/main.go
$ ./golembic --help
Manage database migrations for Go codebases

Usage:
  golembic [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  help        Help about any command
  mysql       Manage database migrations for a MySQL database
  postgres    Manage database migrations for a PostgreSQL database

Flags:
      --dev                     Flag indicating that the migrations should be run in development mode
  -h, --help                    help for golembic
      --metadata-table string   The name of the table that stores migration metadata (default "golembic_migrations")
      --sql-directory string    Path to a directory containing ".sql" migration files

Use "golembic [command] --help" for more information about a command.

and the given subcommands have the same set of actions they can perform (as subcommands)

$ ./golembic postgres --help
Manage database migrations for a PostgreSQL database.

Use the PGPASSWORD environment variable to set the password for the database connection.

Usage:
  golembic postgres [command]

Available Commands:
  describe    Describe the registered sequence of migrations
  up          Run all migrations that have not yet been applied
  up-one      Run the first migration that has not yet been applied
  up-to       Run all the migrations up to a fixed revision that have not yet been applied
  verify      Verify the stored migration metadata against the registered sequence
  version     Display the revision of the most recent migration to be applied

Flags:
      --connect-timeout duration     The timeout to use when waiting on a new connection to PostgreSQL, must be exactly convertible to seconds
      --dbname string                The database name to use when connecting to PostgreSQL (default "postgres")
      --driver-name string           The name of SQL driver to be used when creating a new database connection pool (default "postgres")
  -h, --help                         help for postgres
      --host string                  The host to use when connecting to PostgreSQL (default "localhost")
      --idle-connections int         The maximum number of idle connections (in a connection pool) to PostgreSQL (default 16)
      --lock-timeout duration        The lock timeout to use when connecting to PostgreSQL, must be exactly convertible to milliseconds (default 4s)
      --max-connections int          The maximum number of connections (in a connection pool) to PostgreSQL (default 32)
      --max-lifetime duration        The maximum time a connection (from a connection pool) to PostgreSQL can remain open
      --port string                  The port to use when connecting to PostgreSQL (default "5432")
      --schema string                The schema to use when connecting to PostgreSQL
      --ssl-mode string              The SSL mode to use when connecting to PostgreSQL
      --statement-timeout duration   The statement timeout to use when connecting to PostgreSQL, must be exactly convertible to milliseconds (default 5s)
      --username string              The username to use when connecting to PostgreSQL

Global Flags:
      --dev                     Flag indicating that the migrations should be run in development mode
      --metadata-table string   The name of the table that stores migration metadata (default "golembic_migrations")
      --sql-directory string    Path to a directory containing ".sql" migration files

Use "golembic postgres [command] --help" for more information about a command.

Some of the "leaf" commands have their own flags as well, but this is uncommon:

$ ./golembic postgres up-to --help
Run all the migrations up to a fixed revision that have not yet been applied

Usage:
  golembic postgres up-to [flags]

Flags:
  -h, --help              help for up-to
      --revision string   The revision to run migrations up to
      --verify-history    If set, verify that all of the migration history matches the registered migrations

Global Flags:
      --connect-timeout duration     The timeout to use when waiting on a new connection to PostgreSQL, must be exactly convertible to seconds
      --dbname string                The database name to use when connecting to PostgreSQL (default "postgres")
      --dev                          Flag indicating that the migrations should be run in development mode
      --driver-name string           The name of SQL driver to be used when creating a new database connection pool (default "postgres")
      --host string                  The host to use when connecting to PostgreSQL (default "localhost")
      --idle-connections int         The maximum number of idle connections (in a connection pool) to PostgreSQL (default 16)
      --lock-timeout duration        The lock timeout to use when connecting to PostgreSQL, must be exactly convertible to milliseconds (default 4s)
      --max-connections int          The maximum number of connections (in a connection pool) to PostgreSQL (default 32)
      --max-lifetime duration        The maximum time a connection (from a connection pool) to PostgreSQL can remain open
      --metadata-table string        The name of the table that stores migration metadata (default "golembic_migrations")
      --port string                  The port to use when connecting to PostgreSQL (default "5432")
      --schema string                The schema to use when connecting to PostgreSQL
      --sql-directory string         Path to a directory containing ".sql" migration files
      --ssl-mode string              The SSL mode to use when connecting to PostgreSQL
      --statement-timeout duration   The statement timeout to use when connecting to PostgreSQL, must be exactly convertible to milliseconds (default 5s)
      --username string              The username to use when connecting to PostgreSQL

Examples

up

NOTE: If GOLEMBIC_CMD is not provided to the Makefile, the default is up.

$ make restart-postgres
...
$ make run-postgres-cmd GOLEMBIC_CMD=up
Applying c9b52448285b: Create users table
Applying f1be62155239: Seed data in users table
Applying dce8812d7b6f: Add city column to users table
Applying 0430566018cc: Rename the root user [MILESTONE]
Applying 0501ccd1d98c: Add index on user emails (concurrently)
Applying e2d4eecb1841: Create books table
Applying 432f690fcbda: Create movies table

After creation, the next run does nothing

$ make run-postgres-cmd GOLEMBIC_CMD=up
No migrations to run; latest revision: 432f690fcbda

If we manually delete one, the last migration will get run

$ make psql
...
golembic=> DELETE FROM golembic_migrations WHERE revision = '432f690fcbda';
DELETE 1
golembic=> DROP TABLE movies;
DROP TABLE
golembic=> \q
$
$
$ make run-postgres-cmd GOLEMBIC_CMD=up
Applying 432f690fcbda: Create movies table
up-one
$ make restart-postgres
...
$ make run-postgres-cmd GOLEMBIC_CMD=up-one
Applying c9b52448285b: Create users table
$
$ make run-postgres-cmd GOLEMBIC_CMD=up-one
Applying f1be62155239: Seed data in users table
$
$ make run-postgres-cmd GOLEMBIC_CMD=up-one
Applying dce8812d7b6f: Add city column to users table
$
$ make run-postgres-cmd GOLEMBIC_CMD=up-one
Applying 0430566018cc: Rename the root user [MILESTONE]
$
$ make run-postgres-cmd GOLEMBIC_CMD=up-one
Applying 0501ccd1d98c: Add index on user emails (concurrently)
$
$ make run-postgres-cmd GOLEMBIC_CMD=up-one
Applying e2d4eecb1841: Create books table
$
$ make run-postgres-cmd GOLEMBIC_CMD=up-one
Applying 432f690fcbda: Create movies table
$
$ make run-postgres-cmd GOLEMBIC_CMD=up-one
No migrations to run; latest revision: 432f690fcbda
up-to
$ make restart-postgres
...
$ make run-postgres-cmd GOLEMBIC_CMD=up-to GOLEMBIC_ARGS="--revision c9b52448285b"
Applying c9b52448285b: Create users table
$ make run-postgres-cmd GOLEMBIC_CMD=up-to GOLEMBIC_ARGS="--revision 0501ccd1d98c"
If a migration sequence contains a milestone, it must be the last migration; revision 0430566018cc (3 / 4 migrations)
exit status 1
make: *** [run-postgres-cmd] Error 1
$
$ make run-postgres-cmd GOLEMBIC_CMD=up-to GOLEMBIC_ARGS="--revision 0430566018cc"
Applying f1be62155239: Seed data in users table
Applying dce8812d7b6f: Add city column to users table
Applying 0430566018cc: Rename the root user [MILESTONE]
$
$ make run-postgres-cmd GOLEMBIC_CMD=up-to GOLEMBIC_ARGS="--revision dce8812d7b6f"
No migrations to run; latest revision: 0430566018cc [MILESTONE]
$
$ make run-postgres-cmd GOLEMBIC_CMD=up-to GOLEMBIC_ARGS="--revision 432f690fcbda"
Applying 0501ccd1d98c: Add index on user emails (concurrently)
Applying e2d4eecb1841: Create books table
Applying 432f690fcbda: Create movies table
$
$ make run-postgres-cmd GOLEMBIC_CMD=up-to GOLEMBIC_ARGS="--revision 432f690fcbda"
No migrations to run; latest revision: 432f690fcbda
version
$ make restart-postgres
...
$ make run-postgres-cmd GOLEMBIC_CMD=version
No migrations have been run

Then run all of the migrations and check the version

$ make run-postgres-cmd GOLEMBIC_CMD=up
...
$ make run-postgres-cmd GOLEMBIC_CMD=version
432f690fcbda: Create movies table (applied 2024-06-14 02:30:35.121731 +0000 UTC)
verify
$ make restart-postgres
...
$ make run-postgres-cmd GOLEMBIC_CMD=verify
0 | c9b52448285b | Create users table (not yet applied)
1 | f1be62155239 | Seed data in users table (not yet applied)
2 | dce8812d7b6f | Add city column to users table (not yet applied)
3 | 0430566018cc | Rename the root user [MILESTONE] (not yet applied)
4 | 0501ccd1d98c | Add index on user emails (concurrently) (not yet applied)
5 | e2d4eecb1841 | Create books table (not yet applied)
6 | 432f690fcbda | Create movies table (not yet applied)
$
$
$ make run-postgres-cmd GOLEMBIC_CMD=up-one
Applying c9b52448285b: Create users table
$ make run-postgres-cmd GOLEMBIC_CMD=verify
0 | c9b52448285b | Create users table (applied 2024-06-14 02:32:01.255611 +0000 UTC)
1 | f1be62155239 | Seed data in users table (not yet applied)
2 | dce8812d7b6f | Add city column to users table (not yet applied)
3 | 0430566018cc | Rename the root user [MILESTONE] (not yet applied)
4 | 0501ccd1d98c | Add index on user emails (concurrently) (not yet applied)
5 | e2d4eecb1841 | Create books table (not yet applied)
6 | 432f690fcbda | Create movies table (not yet applied)
$
$
$ make run-postgres-cmd GOLEMBIC_CMD=up-one
Applying f1be62155239: Seed data in users table
$ make run-postgres-cmd GOLEMBIC_CMD=verify
0 | c9b52448285b | Create users table (applied 2024-06-14 02:32:01.255611 +0000 UTC)
1 | f1be62155239 | Seed data in users table (applied 2024-06-14 02:32:11.797511 +0000 UTC)
2 | dce8812d7b6f | Add city column to users table (not yet applied)
3 | 0430566018cc | Rename the root user [MILESTONE] (not yet applied)
4 | 0501ccd1d98c | Add index on user emails (concurrently) (not yet applied)
5 | e2d4eecb1841 | Create books table (not yet applied)
6 | 432f690fcbda | Create movies table (not yet applied)
$
$
$ make run-postgres-cmd GOLEMBIC_CMD=up GOLEMBIC_ARGS="--dev"
Ignoring error in development mode
  If a migration sequence contains a milestone, it must be the last migration; revision 0430566018cc (2 / 5 migrations)
Applying dce8812d7b6f: Add city column to users table
Applying 0430566018cc: Rename the root user [MILESTONE]
Applying 0501ccd1d98c: Add index on user emails (concurrently)
Applying e2d4eecb1841: Create books table
Applying 432f690fcbda: Create movies table
$ make run-postgres-cmd GOLEMBIC_CMD=verify
0 | c9b52448285b | Create users table (applied 2024-06-14 02:32:01.255611 +0000 UTC)
1 | f1be62155239 | Seed data in users table (applied 2024-06-14 02:32:11.797511 +0000 UTC)
2 | dce8812d7b6f | Add city column to users table (applied 2024-06-14 02:32:22.438631 +0000 UTC)
3 | 0430566018cc | Rename the root user [MILESTONE] (applied 2024-06-14 02:32:22.440512 +0000 UTC)
4 | 0501ccd1d98c | Add index on user emails (concurrently) (applied 2024-06-14 02:32:22.442373 +0000 UTC)
5 | e2d4eecb1841 | Create books table (applied 2024-06-14 02:32:22.447998 +0000 UTC)
6 | 432f690fcbda | Create movies table (applied 2024-06-14 02:32:22.450277 +0000 UTC)

We can artificially introduce a "new" migration and see failure to verify

$ make psql
...
golembic=> INSERT INTO golembic_migrations (serial_id, revision, previous) VALUES (7, 'not-in-sequence', '432f690fcbda');
INSERT 0 1
golembic=> \q
$
$ make run-postgres-cmd GOLEMBIC_CMD=verify
Migration stored in SQL doesn't match sequence; sequence has 7 migrations but 8 are stored in the table
exit status 1
make: *** [run-postgres-cmd] Error 1

Similarly, if we can introduce an unknown entry "in sequence"

$ make psql
...
golembic=> DELETE FROM golembic_migrations WHERE revision IN ('not-in-sequence', '432f690fcbda');
DELETE 2
golembic=> INSERT INTO golembic_migrations (serial_id, revision, previous) VALUES (6, 'not-in-sequence', 'e2d4eecb1841');
INSERT 0 1
golembic=> \q
$
$ make run-postgres-cmd GOLEMBIC_CMD=verify
Migration stored in SQL doesn't match sequence; stored migration 6: "not-in-sequence:e2d4eecb1841" does not match migration "432f690fcbda:e2d4eecb1841" in sequence
exit status 1
make: *** [run-postgres-cmd] Error 1

Luckily more painful cases such as one migration being deleted "in the middle" are protected by the constraints on the table:

$ make psql
...
golembic=> DELETE FROM golembic_migrations WHERE revision = '0430566018cc';
ERROR:  update or delete on table "golembic_migrations" violates foreign key constraint "fk_golembic_migrations_previous" on table "golembic_migrations"
DETAIL:  Key (revision)=(0430566018cc) is still referenced from table "golembic_migrations".
golembic=> \q
describe
$ make run-postgres-cmd GOLEMBIC_CMD=describe
0 | c9b52448285b | Create users table
1 | f1be62155239 | Seed data in users table
2 | dce8812d7b6f | Add city column to users table
3 | 0430566018cc | Rename the root user [MILESTONE]
4 | 0501ccd1d98c | Add index on user emails (concurrently)
5 | e2d4eecb1841 | Create books table
6 | 432f690fcbda | Create movies table

Development

$ make
Makefile for `golembic` project

Usage:
   make dev-deps                Install (or upgrade) development time dependencies
   make vet                     Run `go vet` over source tree
   make shellcheck              Run `shellcheck` on all shell files in `./_bin/`
PostgreSQL-specific Targets:
   make start-postgres          Starts a PostgreSQL database running in a Docker container and set up users
   make stop-postgres           Stops the PostgreSQL database running in a Docker container
   make restart-postgres        Stops the PostgreSQL database (if running) and starts a fresh Docker container
   make require-postgres        Determine if PostgreSQL database is running; fail if not
   make psql                    Connects to currently running PostgreSQL DB via `psql`
   make psql-superuser          Connects to currently running PostgreSQL DB via `psql` as superuser
   make run-postgres-cmd        Run `./examples/cmd/main.go` with `postgres` subcommand
   make run-postgres-example    Run `./examples/postgres-script/main.go`
MySQL-specific Targets:
   make start-mysql             Starts a MySQL database running in a Docker container and set up users
   make stop-mysql              Stops the MySQL database running in a Docker container
   make restart-mysql           Stops the MySQL database (if running) and starts a fresh Docker container
   make require-mysql           Determine if MySQL database is running; fail if not
   make mysql                   Connects to currently running MySQL DB via `mysql`
   make mysql-superuser         Connects to currently running MySQL DB via `mysql` as superuser
   make run-mysql-cmd           Run `./examples/cmd/main.go` with `mysql` subcommand
   make run-mysql-example       Run `./examples/mysql-script/main.go`
SQLite-specific Targets:
   make run-sqlite3-example     Run `./examples/sqlite3-script/main.go`

Resources and Inspiration

  • alembic tutorial
  • goose package
  • Blog post: Move fast and migrate things: how we automated migrations in Postgres (in particular, the notes about lock timeouts)
  • Blog post: Update your Database Schema Without Downtime
  • Blog post: Multiple heads in alembic migrations - what to do
  • StackOverflow answer about setting a lock timeout and statement timeout in Postgres 
    BEGIN;
SET LOCAL lock_timeout TO '4s';
SET LOCAL statement_timeout TO '5s';
SELECT * FROM users;
COMMIT;
  • Blog post: When Postgres blocks: 7 tips for dealing with locks

Multiple Revision Heads

Documentation

Overview

Package golembic is intended to provide tooling for managing SQL migrations in Go.

The underlying principles of golembic are as follows:

- Migrations should follow a straight line (from some root); this line should be verified to avoid merge conflicts causing "branching"

- The "current" state of the world will be tracked in a `migrations` table containing all migrations that have been applied.

- A series of migrations should be easy to use both in a script or as part of a larger piece of Go code

- Avoid all import time side effects caused either by importing a package that uses `init()` or by requiring migrations files to use `init()`

- Down migrations are not supported. The idea being that the risk of data loss from a down migration is not worth it and that writing down migrations can be more challenging than writing up migrations.

The design allows for running "arbitrary" code inside migrations so that even non-SQL tasks can be tracked as a "run-once" migration.

The name is a portmanteau of Go (the programming language) and `alembic`, the Python migrations package. The name `alembic` itself is motivated by the foundational ORM SQLAlchemy (an alembic is a distilling apparatus used by alchemists).

Index

Constants

View Source 
const (
	// DefaultMetadataTable is the default name for the table used to store
	// metadata about migrations.
	DefaultMetadataTable = "golembic_migrations"
)

Variables

View Source 
var (
	// ErrDurationConversion is the error returned when a duration cannot be
	// converted to multiple of some base (e.g. milliseconds or seconds)
	// without round off.
	ErrDurationConversion = errors.New("Cannot convert duration")
	// ErrNotRoot is the error returned when attempting to start a sequence of
	// migration with a non-root migration.
	ErrNotRoot = errors.New("Root migration cannot have a previous migration set")
	// ErrMissingRevision is the error returned when attempting to register a migration
	// with no revision.
	ErrMissingRevision = errors.New("A migration must have a revision")
	// ErrNoPrevious is the error returned when attempting to register a migration
	// with no previous.
	ErrNoPrevious = errors.New("Cannot register a migration with no previous migration")
	// ErrPreviousNotRegistered is the error returned when attempting to register
	// a migration with a previous that is not yet registered.
	ErrPreviousNotRegistered = errors.New("Cannot register a migration until previous migration is registered")
	// ErrAlreadyRegistered is the error returned when a migration has already been
	// registered.
	ErrAlreadyRegistered = errors.New("Migration has already been registered")
	// ErrNilInterface is the error returned when a value satisfying an interface
	// is nil in a context where it is not allowed.
	ErrNilInterface = errors.New("Value satisfying interface was nil")
	// ErrMigrationNotRegistered is the error returned when no migration has been
	// registered for a given revision.
	ErrMigrationNotRegistered = errors.New("No migration registered for revision")
	// ErrMigrationMismatch is the error returned when the migration stored in
	// SQL does not match the registered migration.
	ErrMigrationMismatch = errors.New("Migration stored in SQL doesn't match sequence")
	// ErrCannotInvokeUp is the error returned when a migration cannot invoke the
	// up function (e.g. if it is `nil`).
	ErrCannotInvokeUp = errors.New("Cannot invoke up function for a migration")
	// ErrCannotPassMilestone is the error returned when a migration sequence
	// contains a milestone migration that is **NOT** the last step.
	ErrCannotPassMilestone = errors.New("If a migration sequence contains a milestone, it must be the last migration")
)

Functions

func CreateMigrationsTable 

func CreateMigrationsTable(ctx context.Context, manager *Manager) (err error)

CreateMigrationsTable invokes SQL statements required to create the metadata table used to track migrations. If the table already exists (as detected by `provider.TableExistsSQL()`), this function will not attempt to create a table or any constraints.

func QuoteIdentifier 

func QuoteIdentifier(name string) string

QuoteIdentifier quotes an identifier, such as a table name, for usage in a query.

This implementation is vendored in here to avoid the side effects of importing `github.com/lib/pq`.

See: - https://github.com/lib/pq/blob/v1.8.0/conn.go#L1564-L1581 - https://www.sqlite.org/lang_keywords.html - https://github.com/ronsavage/SQL/blob/a67e7eaefae89ed761fa4dcbc5431ec9a235a6c8/sql-99.bnf#L412

func QuoteLiteral 

func QuoteLiteral(literal string) string

QuoteLiteral quotes a literal, such as `2023-01-05 15:00:00Z`, for usage in a query.

This implementation is vendored in here to avoid the side effects of importing `github.com/lib/pq`.

See: - https://github.com/lib/pq/blob/v1.8.0/conn.go#L1583-L1614 - https://www.sqlite.org/lang_keywords.html - https://github.com/ronsavage/SQL/blob/a67e7eaefae89ed761fa4dcbc5431ec9a235a6c8/sql-99.bnf#L758-L761 - https://github.com/ronsavage/SQL/blob/a67e7eaefae89ed761fa4dcbc5431ec9a235a6c8/sql-99.bnf#L290

func ToRoundDuration 

func ToRoundDuration(d, base time.Duration) (int64, error)

ToRoundDuration converts a duration to an **exact** multiple of some base duration or errors if round off is required.

Types

type ApplyConfig 

type ApplyConfig struct {
	VerifyHistory bool
	Revision      string
}

ApplyConfig provides configurable fields for "up" commands that will apply migrations.

func NewApplyConfig 

func NewApplyConfig(opts ...ApplyOption) (*ApplyConfig, error)

NewApplyConfig creates a new `ApplyConfig` and applies options.

type ApplyOption 

type ApplyOption = func(*ApplyConfig) error

ApplyOption describes options used to create an apply configuration.

func OptApplyRevision 

func OptApplyRevision(revision string) ApplyOption

OptApplyRevision sets `Revision` on an `ApplyConfig`.

func OptApplyVerifyHistory 

func OptApplyVerifyHistory(verify bool) ApplyOption

OptApplyVerifyHistory sets `VerifyHistory` on an `ApplyConfig`.

type Column 

type Column interface {
	sql.Scanner
	driver.Valuer
}

Column represents an abstract SQL column value; it requires a `Scan()` interface for reading and a `Value()` interface for writing.

type CreateTableOption 

type CreateTableOption = func(*CreateTableParameters)

CreateTableOption describes options used for create table parameters.

func OptCreateTableConstraints 

func OptCreateTableConstraints(constraints string) CreateTableOption

OptCreateTableConstraints sets the `Constraints` field in create table options.

func OptCreateTableCreatedAt 

func OptCreateTableCreatedAt(createdAt string) CreateTableOption

OptCreateTableCreatedAt sets the `CreatedAt` field in create table options.

func OptCreateTablePrevious 

func OptCreateTablePrevious(previous string) CreateTableOption

OptCreateTablePrevious sets the `Previous` field in create table options.

func OptCreateTableRevision 

func OptCreateTableRevision(revision string) CreateTableOption

OptCreateTableRevision sets the `Revision` field in create table options.

func OptCreateTableSerialID 

func OptCreateTableSerialID(serialID string) CreateTableOption

OptCreateTableSerialID sets the `SerialID` field in create table options.

func OptCreateTableSkip 

func OptCreateTableSkip(skip bool) CreateTableOption

OptCreateTableSkip sets the `SkipConstraintStatements` field in create table options.

type CreateTableParameters 

type CreateTableParameters struct {
	SerialID                 string
	Revision                 string
	Previous                 string
	CreatedAt                string
	Constraints              string
	SkipConstraintStatements bool
}

CreateTableParameters specifies a set of parameters that are intended to be used in a `CREATE TABLE` statement. This allows providers to specify custom column types or add custom checks / constraints that are engine specific.

func NewCreateTableParameters 

func NewCreateTableParameters(opts ...CreateTableOption) CreateTableParameters

NewCreateTableParameters populates a `CreateTableParameters` with a basic set of defaults and allows optional overrides for all fields.

type EngineProvider 

type EngineProvider interface {
	// QueryParameter produces a placeholder like `$1` or `?` for a numbered
	// parameter in a SQL query.
	QueryParameter(index int) string
	// NewCreateTableParameters produces column types and constraints for the
	// `CREATE TABLE` statement used to create the migrations table.
	NewCreateTableParameters() CreateTableParameters
	// TimestampColumn produces a value that can be used for reading / writing
	// the `created_at` column.
	TimestampColumn() TimestampColumn
	// QuoteIdentifier quotes an identifier, such as a table name, for usage
	// in a query.
	QuoteIdentifier(name string) string
	// QuoteLiteral quotes a literal, such as `2023-01-05 15:00:00Z`, for usage
	// in a query.
	QuoteLiteral(literal string) string
	// Open creates a database connection for the engine provider.
	Open() (*sql.DB, error)
	// TableExistsSQL returns a SQL query that can be used to determine if a
	// table exists. It is expected to use a clause such as `WHERE tablename = $1`
	// or `WHERE tablename = ?` to filter results.
	TableExistsSQL() string
}

EngineProvider describes the interface required for a database engine.

type Manager 

type Manager struct {
	// MetadataTable is the name of the table that stores migration metadata.
	// The expected default value (`DefaultMetadataTable`) is
	// "golembic_migrations".
	MetadataTable string
	// ConnectionPool is a cache-able pool of connections to the database.
	ConnectionPool *sql.DB
	// Provider delegates all actions to an abstract SQL database engine, with
	// the expectation that the provider also encodes connection information.
	Provider EngineProvider
	// Sequence is the collection of registered migrations to be applied,
	// verified, described, etc. by this manager.
	Sequence *Migrations
	// DevelopmentMode is a flag indicating that this manager is currently
	// being run in development mode, so things like extra validation should
	// intentionally be disabled. This is intended for use in testing and
	// development, where an entire database is spun up locally (e.g. in Docker)
	// and migrations will be applied from scratch (including milestones that
	// may not come at the end).
	DevelopmentMode bool
	// Log is used for printing output
	Log PrintfReceiver
}

Manager orchestrates database operations done via `Up` / `UpConn` as well as supporting operations such as creating a table for migration metadata and writing rows into that metadata table during a migration.

func NewManager 

func NewManager(opts ...ManagerOption) (*Manager, error)

NewManager creates a new manager for orchestrating migrations.

func (*Manager) ApplyMigration 

func (m *Manager) ApplyMigration(ctx context.Context, migration Migration) (err error)

ApplyMigration creates a transaction that runs the "Up" migration.

func (*Manager) CloseConnectionPool 

func (m *Manager) CloseConnectionPool() error

CloseConnectionPool closes the connection pool and removes it, if one is set / cached on the current manager.

func (*Manager) Describe 

func (m *Manager) Describe(_ context.Context) error

Describe displays all of the registered migrations (with descriptions).

func (*Manager) EnsureConnectionPool 

func (m *Manager) EnsureConnectionPool(ctx context.Context) (*sql.DB, error)

EnsureConnectionPool returns a cached database connection pool (if already set) or invokes `NewConnection()` to create a new one.

func (*Manager) EnsureMigrationsTable 

func (m *Manager) EnsureMigrationsTable(ctx context.Context) error

EnsureMigrationsTable checks that the migrations metadata table exists and creates it if not.

func (*Manager) GetVersion 

func (m *Manager) GetVersion(ctx context.Context, opts ...ApplyOption) (*Migration, error)

GetVersion returns the migration that corresponds to the version that was most recently applied.

func (*Manager) InsertMigration 

func (m *Manager) InsertMigration(ctx context.Context, tx *sql.Tx, migration Migration) error

InsertMigration inserts a migration into the migrations metadata table.

func (*Manager) IsApplied 

func (m *Manager) IsApplied(ctx context.Context, tx *sql.Tx, migration Migration) (bool, error)

IsApplied checks if a migration has already been applied.

NOTE: This assumes, but does not check, that the migrations metadata table exists.

func (*Manager) Latest 

func (m *Manager) Latest(ctx context.Context) (revision string, createdAt time.Time, err error)

Latest determines the revision and timestamp of the most recently applied migration.

NOTE: This assumes, but does not check, that the migrations metadata table exists.

func (*Manager) NewConnectionPool 

func (m *Manager) NewConnectionPool(ctx context.Context) (*sql.DB, error)

NewConnectionPool creates a new database connection pool and validates that it can ping the DB.

func (*Manager) NewTx 

func (m *Manager) NewTx(ctx context.Context) (*sql.Tx, error)

NewTx creates a new transaction after ensuring there is an existing connection.

func (*Manager) Up 

func (m *Manager) Up(ctx context.Context, opts ...ApplyOption) error

Up applies all migrations that have not yet been applied.

func (*Manager) UpOne 

func (m *Manager) UpOne(ctx context.Context, opts ...ApplyOption) error

UpOne applies the **next** migration that has yet been applied, if any.

func (*Manager) UpTo 

func (m *Manager) UpTo(ctx context.Context, opts ...ApplyOption) error

UpTo applies all migrations that have yet to be applied up to (and including) a revision, if any. This expects the `ApplyConfig` revision to be set in `opts`.

func (*Manager) Verify 

func (m *Manager) Verify(ctx context.Context) (err error)

Verify checks that the rows in the migrations metadata table match the sequence.

func (*Manager) Version 

func (m *Manager) Version(ctx context.Context, opts ...ApplyOption) error

Version displays the revision of the most recent migration to be applied

type ManagerOption 

type ManagerOption = func(*Manager) error

ManagerOption describes options used to create a new manager.

func OptDevelopmentMode 

func OptDevelopmentMode(mode bool) ManagerOption

OptDevelopmentMode sets the development mode flag on a manager.

func OptManagerConnectionPool 

func OptManagerConnectionPool(pool *sql.DB) ManagerOption

OptManagerConnectionPool sets the connection pool on a manager.

func OptManagerLog 

func OptManagerLog(log PrintfReceiver) ManagerOption

OptManagerLog sets the logger interface on a manager. If `log` is `nil` the option will return an error.

func OptManagerMetadataTable 

func OptManagerMetadataTable(table string) ManagerOption

OptManagerMetadataTable sets the metadata table name on a manager.

func OptManagerProvider 

func OptManagerProvider(provider EngineProvider) ManagerOption

OptManagerProvider sets the provider on a manager.

func OptManagerSequence 

func OptManagerSequence(migrations *Migrations) ManagerOption

OptManagerSequence sets the migrations sequence on a manager.

type Migration 

type Migration struct {
	// Previous is the revision identifier for the migration immediately
	// preceding this one. If absent, this indicates that this migration is
	// the "base" or "root" migration.
	Previous string
	// Revision is an opaque name that uniquely identifies a migration. It
	// is required for a migration to be valid.
	Revision string
	// Description is a long form description of why the migration is being
	// performed. It is intended to be used in "describe" scenarios where
	// a long form "history" of changes is presented.
	Description string
	// Milestone is a flag indicating if the current migration is a milestone.
	// A milestone is a special migration that **must** be the last migration
	// in a sequence whenever applied. This is intended to be used in situations
	// where a change must be "staged" in two (or more parts) and one part
	// must run and "stabilize" before the next migration runs. For example, in
	// a rolling update deploy strategy some changes may not be compatible with
	// "old" and "new" versions of the code that may run simultaneously, so a
	// milestone marks the last point where old / new versions of application
	// code should be expected to be able to interact with the current schema.
	Milestone bool
	// Up is the function to be executed when a migration is being applied. Either
	// this field or `UpConn` are required (not both) and this field should be
	// the default choice in most cases. This function will be run in a transaction
	// that also writes a row to the migrations metadata table to signify that
	// this migration was applied.
	Up UpMigration
	// UpConn is the non-transactional form of `Up`. This should be used in
	// rare situations where a migration cannot run inside a transaction, e.g.
	// a `CREATE UNIQUE INDEX CONCURRENTLY` statement.
	UpConn UpMigrationConn
	// contains filtered or unexported fields
}

Migration represents an individual migration to be applied; typically as a set of SQL statements.

func NewMigration 

func NewMigration(opts ...MigrationOption) (*Migration, error)

NewMigration creates a new migration from a variadic slice of options.

func (Migration) Compact 

func (m Migration) Compact() string

Compact gives a "limited" representation of the migration

func (Migration) ExtendedDescription 

func (m Migration) ExtendedDescription() string

ExtendedDescription is an extended form of `m.Description` that also incorporates other information like whether `m` is a milestone.

func (Migration) InvokeUp 

func (m Migration) InvokeUp(ctx context.Context, pool *sql.DB, tx *sql.Tx) error

InvokeUp dispatches to `Up` or `UpConn`, depending on which is set. If both or neither is set, that is considered an error. If `UpConn` needs to be invoked, this lazily creates a new connection from a pool. It's crucial that the pool sets the relevant timeouts when creating a new connection to make sure migrations don't cause disruptions in application performance due to accidentally holding locks for an extended period.

func (Migration) Like 

func (m Migration) Like(other Migration) bool

Like is "almost" an equality check, it compares the `Previous` and `Revision`.

type MigrationOption 

type MigrationOption = func(*Migration) error

MigrationOption describes options used to create a new migration.

func OptAlwaysError 

func OptAlwaysError(err error) MigrationOption

OptAlwaysError returns an option that always returns an error.

func OptDescription 

func OptDescription(description string) MigrationOption

OptDescription sets the description on a migration.

func OptMilestone 

func OptMilestone(milestone bool) MigrationOption

OptMilestone sets the milestone flag on a migration.

func OptPrevious 

func OptPrevious(previous string) MigrationOption

OptPrevious sets the previous on a migration.

func OptRevision 

func OptRevision(revision string) MigrationOption

OptRevision sets the revision on a migration.

func OptUp 

func OptUp(up UpMigration) MigrationOption

OptUp sets the `up` function on a migration.

func OptUpConn 

func OptUpConn(up UpMigrationConn) MigrationOption

OptUpConn sets the non-transactional `up` function on a migration.

func OptUpConnFromFile 

func OptUpConnFromFile(filename string) MigrationOption

OptUpConnFromFile returns an option that sets the non-transctional `up` function to execute a SQL statement that is stored in a file.

func OptUpConnFromSQL 

func OptUpConnFromSQL(statement string) MigrationOption

OptUpConnFromSQL returns an option that sets the non-transctional `up` function to execute a SQL statement.

func OptUpFromFile 

func OptUpFromFile(filename string) MigrationOption

OptUpFromFile returns an option that sets the `up` function to execute a SQL statement that is stored in a file.

func OptUpFromSQL 

func OptUpFromSQL(statement string) MigrationOption

OptUpFromSQL returns an option that sets the `up` function to execute a SQL statement.

type Migrations 

type Migrations struct {
	// contains filtered or unexported fields
}

Migrations represents a sequence of migrations to be applied.

func NewSequence 

func NewSequence(root Migration) (*Migrations, error)

NewSequence creates a new sequence of migrations rooted in a single base / root migration.

func (*Migrations) All 

func (m *Migrations) All() []Migration

All produces the migrations in the sequence, in order.

NOTE: This does not verify or enforce the invariant that there must be exactly one migration without a previous migration. This invariant is enforced by the exported methods such as `Register()` and `RegisterMany()` and the constructor `NewSequence()`.

func (*Migrations) Between 

func (m *Migrations) Between(since, until string) (int, []Migration, error)

Between returns the migrations that occur between two revisions.

This can be seen as a combination of `Since()` and `Until()`.

func (*Migrations) Describe 

func (m *Migrations) Describe(log PrintfReceiver)

Describe displays all of the registered migrations (with descriptions).

func (*Migrations) Get 

func (m *Migrations) Get(revision string) *Migration

Get retrieves a revision from the sequence, if present. If not, returns `nil`.

func (*Migrations) Register 

func (m *Migrations) Register(migration Migration) error

Register adds a new migration to an existing sequence of migrations, if possible. The new migration must have a previous migration and have a valid revision that is not already registered.

func (*Migrations) RegisterMany 

func (m *Migrations) RegisterMany(ms ...Migration) error

RegisterMany attempts to register multiple migrations (in order) with an existing sequence.

func (*Migrations) RegisterManyOpt 

func (m *Migrations) RegisterManyOpt(manyOpts ...[]MigrationOption) error

RegisterManyOpt attempts to register multiple migrations (in order) with an existing sequence. It differs from `RegisterMany()` in that the construction of `Migration` objects is handled directly here by taking a slice of option slices.

func (*Migrations) Revisions 

func (m *Migrations) Revisions() []string

Revisions produces the revisions in the sequence, in order.

This utilizes `All()` and just extracts the revisions.

func (*Migrations) Root 

func (m *Migrations) Root() Migration

Root does a linear scan of every migration in the sequence and returns the root migration. In the "general" case such a scan would be expensive, but the number of migrations should always be a small number.

NOTE: This does not verify or enforce the invariant that there must be exactly one migration without a previous migration. This invariant is enforced by the exported methods such as `Register()` and `RegisterMany()` and the constructor `NewSequence()`.

func (*Migrations) Since 

func (m *Migrations) Since(revision string) (int, []Migration, error)

Since returns the migrations that occur **after** `revision`.

This utilizes `All()` and returns all migrations after the one that matches `revision`. If none match, an error will be returned. If `revision` is the **last** migration, the migrations returned will be an empty slice.

func (*Migrations) Until 

func (m *Migrations) Until(revision string) (int, []Migration, error)

Until returns the migrations that occur **before** `revision`.

This utilizes `All()` and returns all migrations up to and including the one that matches `revision`. If none match, an error will be returned.

type PrintfReceiver 

type PrintfReceiver interface {
	Printf(format string, a ...interface{}) (n int, err error)
}

PrintfReceiver is a generic interface for logging and printing. In cases where a trailing newline is desired (e.g. STDOUT), the type implemented `PrintfReceiver` must add the newline explicitly.

type TimeColumnPointer 

type TimeColumnPointer struct {
	Stored time.Time
}

TimeColumnPointer provides the default implementation of `TimestampColumn`.

func (*TimeColumnPointer) Pointer 

func (tcp *TimeColumnPointer) Pointer() interface{}

Pointer returns a pointer to the stored timestamp value.

func (TimeColumnPointer) Timestamp 

func (tcp TimeColumnPointer) Timestamp() time.Time

Timestamp returns the stored timestamp value.

type TimestampColumn 

type TimestampColumn interface {
	Pointer() interface{}
	Timestamp() time.Time
}

TimestampColumn represents an abstract SQL column that stores a timestamp.

type UpMigration 

type UpMigration = func(context.Context, *sql.Tx) error

UpMigration defines a function interface to be used for up / forward migrations. The SQL transaction will be started **before** `UpMigration` is invoked and will be committed **after** the `UpMigration` exits without error. In addition to the contents of `UpMigration`, a row will be written to the migrations metadata table as part of the transaction.

The expectation is that the migration runs SQL statements within the transaction. If a migration cannot run inside a transaction, e.g. a `CREATE UNIQUE INDEX CONCURRENTLY` statement, then the `UpMigration` interface should be used.

type UpMigrationConn 

type UpMigrationConn = func(context.Context, *sql.Conn) error

UpMigrationConn defines a function interface to be used for up / forward migrations. This is the non-transactional form of `UpMigration` and should only be used in rare situations.

Source Files

View all Source files

Directories

Path Synopsis
Package command provides helpers to create a binary / command from a migration sequence.
 Package command provides helpers to create a binary / command from a migration sequence.
cmd
mysql-script
postgres-script
sqlite3-script
Package mysql provides MySQL helpers for golembic.
 Package mysql provides MySQL helpers for golembic.
Package postgres provides PostgreSQL helpers for golembic.
 Package postgres provides PostgreSQL helpers for golembic.
Package sqlite3 provides SQLite helpers for golembic.
 Package sqlite3 provides SQLite helpers for golembic.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.