Documentation ¶
Overview ¶
SQL Schema migration tool for Go.
Key features:
- Usable as a CLI tool or as a library
- Supports SQLite, PostgreSQL, MySQL, MSSQL and Oracle databases (through gorp)
- Can embed migrations into your application
- Migrations are defined with SQL for full flexibility
- Atomic migrations
- Up/down migrations to allow rollback
- Supports multiple database types in one project
Installation ¶
To install the library and command line program, use the following:
go get -v github.com/kva3umoda/sql-migrate/...
Command-line tool ¶
The main command is called sql-migrate.
$ sql-migrate --help usage: sql-migrate [--version] [--help] <command> [<args>] Available commands are: down Undo a database migration new Create a new migration redo Reapply the last migration status Show migration status up Migrates the database to the most recent version available
Each command requires a configuration file (which defaults to dbconfig.yml, but can be specified with the -config flag). This config file should specify one or more environments:
development: dialect: sqlite3 datasource: test.db dir: migrations/sqlite3 production: dialect: postgres datasource: dbname=myapp sslmode=disable dir: migrations/postgres table: migrations
The `table` setting is optional and will default to `gorp_migrations`.
The environment that will be used can be specified with the -env flag (defaults to development).
Use the --help flag in combination with any of the commands to get an overview of its usage:
$ sql-migrate up --help Usage: sql-migrate up [options] ... Migrates the database to the most recent version available. Options: -config=config.yml Configuration file to use. -env="development" Environment. -limit=0 Limit the number of migrations (0 = unlimited). -dryrun Don't apply migrations, just print them.
The up command applies all available migrations. By contrast, down will only apply one migration by default. This behavior can be changed for both by using the -limit parameter.
The redo command will unapply the last migration and reapply it. This is useful during development, when you're writing migrations.
Use the status command to see the state of the applied migrations:
$ sql-migrate status +---------------+-----------------------------------------+ | MIGRATION | APPLIED | +---------------+-----------------------------------------+ | 1_initial.sql | 2014-09-13 08:19:06.788354925 +0000 UTC | | 2_record.sql | no | +---------------+-----------------------------------------+
MySQL Caveat ¶
If you are using MySQL, you must append ?parseTime=true to the datasource configuration. For example:
production: dialect: mysql datasource: root@/dbname?parseTime=true dir: migrations/mysql table: migrations
See https://github.com/go-sql-driver/mysql#parsetime for more information.
Library ¶
Import sql-migrate into your application:
import "github.com/kva3umoda/sql-migrate"
Set up a source of migrations, this can be from memory, from a set of files or from bindata (more on that later):
// Hardcoded strings in memory: migrations := &migrate.MemoryMigrationSource{ Migrations: []*migrate.Migration{ &migrate.Migration{ Id: "123", Up: []string{"CREATE TABLE people (id int)"}, Down: []string{"DROP TABLE people"}, }, }, } // OR: Read migrations from a folder: migrations := &migrate.FileMigrationSource{ Dir: "db/migrations", } // OR: Use migrations from bindata: migrations := &migrate.AssetMigrationSource{ Asset: Asset, AssetDir: AssetDir, Dir: "migrations", }
Then use the Exec function to upgrade your database:
db, err := sql.Open("sqlite3", filename) if err != nil { // Handle errors! } n, err := migrate.Exec(db, "sqlite3", migrations, migrate.Up) if err != nil { // Handle errors! } fmt.Printf("Applied %d migrations!\n", n)
Note that n can be greater than 0 even if there is an error: any migration that succeeded will remain applied even if a later one fails.
The full set of capabilities can be found in the API docs below.
Writing migrations ¶
Migrations are defined in SQL files, which contain a set of SQL statements. Special comments are used to distinguish up and down migrations.
-- +migrate Up -- SQL in section 'Up' is executed when this migration is applied CREATE TABLE people (id int); -- +migrate Down -- SQL section 'Down' is executed when this migration is rolled back DROP TABLE people;
You can put multiple statements in each block, as long as you end them with a semicolon (;).
If you have complex statements which contain semicolons, use StatementBegin and StatementEnd to indicate boundaries:
-- +migrate Up CREATE TABLE people (id int); -- +migrate StatementBegin CREATE OR REPLACE FUNCTION do_something() returns void AS $$ DECLARE create_query text; BEGIN -- Do something here END; $$ language plpgsql; -- +migrate StatementEnd -- +migrate Down DROP FUNCTION do_something(); DROP TABLE people;
The order in which migrations are applied is defined through the filename: sql-migrate will sort migrations based on their name. It's recommended to use an increasing version number or a timestamp as the first part of the filename.
Normally each migration is run within a transaction in order to guarantee that it is fully atomic. However some SQL commands (for example creating an index concurrently in PostgreSQL) cannot be executed inside a transaction. In order to execute such a command in a migration, the migration can be run using the notransaction option:
-- +migrate Up notransaction CREATE UNIQUE INDEX people_unique_id_idx CONCURRENTLY ON people (id); -- +migrate Down DROP INDEX people_unique_id_idx;
Embedding migrations with packr ¶
If you like your Go applications self-contained (that is: a single binary): use packr (https://github.com/gobuffalo/packr) to embed the migration files.
Just write your migration files as usual, as a set of SQL files in a folder.
Use the PackrMigrationSource in your application to find the migrations:
migrations := &migrate.PackrMigrationSource{ Box: packr.NewBox("./migrations"), }
If you already have a box and would like to use a subdirectory:
migrations := &migrate.PackrMigrationSource{ Box: myBox, Dir: "./migrations", }
Embedding migrations with bindata ¶
As an alternative, but slightly less maintained, you can use bindata (https://github.com/shuLhan/go-bindata) to embed the migration files.
Just write your migration files as usual, as a set of SQL files in a folder.
Then use bindata to generate a .go file with the migrations embedded:
go-bindata -pkg myapp -o bindata.go db/migrations/
The resulting bindata.go file will contain your migrations. Remember to regenerate your bindata.go file whenever you add/modify a migration (go generate will help here, once it arrives).
Use the AssetMigrationSource in your application to find the migrations:
migrations := &migrate.AssetMigrationSource{ Asset: Asset, AssetDir: AssetDir, Dir: "db/migrations", }
Both Asset and AssetDir are functions provided by bindata.
Then proceed as usual.
Extending ¶
Adding a new migration source means implementing MigrationSource.
type MigrationSource interface { FindMigrations() ([]*Migration, error) }
The resulting slice of migrations will be executed in the given order, so it should usually be sorted by the Id field.
Index ¶
- func DefaultLogger() *defaultLogger
- func Exec(db *sql.DB, dialect dialect.Dialect, m MigrationSource, dir MigrationDirection) (int, error)
- func ExecContext(ctx context.Context, db *sql.DB, dialect dialect.Dialect, m MigrationSource, ...) (int, error)
- func ExecMax(db *sql.DB, dialect dialect.Dialect, m MigrationSource, dir MigrationDirection, ...) (int, error)
- func ExecMaxContext(ctx context.Context, db *sql.DB, dialect dialect.Dialect, m MigrationSource, ...) (int, error)
- func ExecVersion(db *sql.DB, dialect dialect.Dialect, m MigrationSource, dir MigrationDirection, ...) (int, error)
- func ExecVersionContext(ctx context.Context, db *sql.DB, dialect dialect.Dialect, m MigrationSource, ...) (int, error)
- func GetDialect(name DialectName) (dialect.Dialect, error)
- func PlanMigration(db *sql.DB, dialect dialect.Dialect, m MigrationSource, dir MigrationDirection, ...) ([]*PlannedMigration, *MigrationRepository, error)
- func PlanMigrationToVersion(db *sql.DB, dialect dialect.Dialect, m MigrationSource, dir MigrationDirection, ...) ([]*PlannedMigration, *MigrationRepository, error)
- func SetCreateSchema(enable bool)
- func SetCreateTable(enable bool)
- func SetIgnoreUnknown(v bool)
- func SetLogger(logger Logger)
- func SetSchema(name string)
- func SetTable(name string)
- func SkipMax(db *sql.DB, dialect dialect.Dialect, m MigrationSource, dir MigrationDirection, ...) (int, error)
- type AssetDirFunc
- type AssetFunc
- type AssetMigrationSource
- type DialectName
- type FileSystemMigrationSource
- type Logger
- type MemoryMigrationSource
- type Migration
- type MigrationDirection
- type MigrationExecutor
- func (ex *MigrationExecutor) Exec(db *sql.DB, dialect dialect.Dialect, source MigrationSource, ...) (int, error)
- func (ex *MigrationExecutor) ExecContext(ctx context.Context, db *sql.DB, dialect dialect.Dialect, m MigrationSource, ...) (int, error)
- func (ex *MigrationExecutor) ExecMax(db *sql.DB, dialect dialect.Dialect, m MigrationSource, dir MigrationDirection, ...) (int, error)
- func (ex *MigrationExecutor) ExecMaxContext(ctx context.Context, db *sql.DB, dialect dialect.Dialect, ...) (int, error)
- func (ex *MigrationExecutor) ExecVersion(db *sql.DB, dialect dialect.Dialect, source MigrationSource, ...) (int, error)
- func (ex *MigrationExecutor) ExecVersionContext(ctx context.Context, db *sql.DB, dialect dialect.Dialect, ...) (int, error)
- func (ex *MigrationExecutor) GetMigrationRecords(ctx context.Context, db *sql.DB, dialect dialect.Dialect) ([]MigrationRecord, error)
- func (ex *MigrationExecutor) PlanMigration(ctx context.Context, db *sql.DB, dialect dialect.Dialect, ...) ([]*PlannedMigration, *MigrationRepository, error)
- func (ex *MigrationExecutor) PlanMigrationToVersion(ctx context.Context, db *sql.DB, dialect dialect.Dialect, ...) ([]*PlannedMigration, *MigrationRepository, error)
- func (ex *MigrationExecutor) SkipMax(ctx context.Context, db *sql.DB, dialect dialect.Dialect, m MigrationSource, ...) (int, error)
- type MigrationRecord
- type MigrationRepository
- func (r *MigrationRepository) BeginTx(ctx context.Context) (*sql.Tx, context.Context, error)
- func (r *MigrationRepository) CreateSchema(ctx context.Context) error
- func (r *MigrationRepository) CreateTable(ctx context.Context) error
- func (r *MigrationRepository) DeleteMigration(ctx context.Context, id string) error
- func (r *MigrationRepository) ExecContext(ctx context.Context, query string, args ...any) (sql.Result, error)
- func (r *MigrationRepository) ListMigration(ctx context.Context) ([]MigrationRecord, error)
- func (r *MigrationRepository) QueryContext(ctx context.Context, query string, args ...any) (*sql.Rows, error)
- func (r *MigrationRepository) SaveMigration(ctx context.Context, record MigrationRecord) error
- type MigrationSource
- type PlanError
- type PlannedMigration
- type SqlExecutor
- type TxError
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func DefaultLogger ¶
func DefaultLogger() *defaultLogger
func Exec ¶
func Exec(db *sql.DB, dialect dialect.Dialect, m MigrationSource, dir MigrationDirection) (int, error)
Exec Execute a set of migrations Returns the number of applied migrations.
func ExecContext ¶
func ExecContext(ctx context.Context, db *sql.DB, dialect dialect.Dialect, m MigrationSource, dir MigrationDirection) (int, error)
ExecContext Execute a set of migrations with an input context. Returns the number of applied migrations.
func ExecMax ¶
func ExecMax(db *sql.DB, dialect dialect.Dialect, m MigrationSource, dir MigrationDirection, max int) (int, error)
ExecMax Execute a set of migrations Will apply at most `max` migrations. Pass 0 for no limit (or use Exec). Returns the number of applied migrations.
func ExecMaxContext ¶
func ExecMaxContext(ctx context.Context, db *sql.DB, dialect dialect.Dialect, m MigrationSource, dir MigrationDirection, max int) (int, error)
ExecMaxContext Execute a set of migrations with an input context. Will apply at most `max` migrations. Pass 0 for no limit (or use Exec). Returns the number of applied migrations.
func ExecVersion ¶
func ExecVersion(db *sql.DB, dialect dialect.Dialect, m MigrationSource, dir MigrationDirection, version int64) (int, error)
ExecVersion Execute a set of migrations Will apply at the target `version` of migration. Cannot be a negative value. Returns the number of applied migrations.
func ExecVersionContext ¶
func ExecVersionContext(ctx context.Context, db *sql.DB, dialect dialect.Dialect, m MigrationSource, dir MigrationDirection, version int64) (int, error)
ExecVersionContext Execute a set of migrations with an input context. Will apply at the target `version` of migration. Cannot be a negative value. Returns the number of applied migrations.
func GetDialect ¶
func GetDialect(name DialectName) (dialect.Dialect, error)
func PlanMigration ¶
func PlanMigration(db *sql.DB, dialect dialect.Dialect, m MigrationSource, dir MigrationDirection, max int) ([]*PlannedMigration, *MigrationRepository, error)
PlanMigration Plan a migration.
func PlanMigrationToVersion ¶
func PlanMigrationToVersion(db *sql.DB, dialect dialect.Dialect, m MigrationSource, dir MigrationDirection, version int64) ([]*PlannedMigration, *MigrationRepository, error)
PlanMigrationToVersion Plan a migration to version.
func SetCreateSchema ¶
func SetCreateSchema(enable bool)
SetCreateSchema sets the boolean to enable the creation of the migration schema
func SetCreateTable ¶
func SetCreateTable(enable bool)
SetCreateTable sets the boolean to enable the creation of the migration table
func SetIgnoreUnknown ¶
func SetIgnoreUnknown(v bool)
SetIgnoreUnknown sets the flag that skips database check to see if there is a migration in the database that is not in migration source.
This should be used sparingly as it is removing a safety check.
func SetSchema ¶
func SetSchema(name string)
SetSchema sets the name of a schema that the migration table be referenced.
Types ¶
type AssetDirFunc ¶
type AssetMigrationSource ¶
type AssetMigrationSource struct { // Asset should return content of file in path if exists Asset AssetFunc // AssetDir should return list of files in the path AssetDir AssetDirFunc // Dir Path in the bindata to use. Dir string }
AssetMigrationSource Migrations from a bindata asset set.
func NewAssetMigrationSource ¶
func NewAssetMigrationSource(asset AssetFunc, assetDir AssetDirFunc, dir string) *AssetMigrationSource
func (*AssetMigrationSource) FindMigrations ¶
func (a *AssetMigrationSource) FindMigrations() ([]*Migration, error)
type DialectName ¶
type DialectName string
const ( SQLite3 DialectName = "sqlite3" Postgres DialectName = "postgres" MySQL DialectName = "mysql" MSSQL DialectName = "mssql" OCI8 DialectName = "oci8" GoDrOr DialectName = "godror" Snowflake DialectName = "snowflake" ClickHouse DialectName = "clickhouse" )
type FileSystemMigrationSource ¶
type FileSystemMigrationSource struct {
// contains filtered or unexported fields
}
func NewEmbedFileSystemMigrationSource ¶
func NewEmbedFileSystemMigrationSource(fs embed.FS, root string) *FileSystemMigrationSource
NewEmbedFileSystemMigrationSource A set of migrations loaded from an go1.16 embed.FS
func NewFileMigrationSource ¶
func NewFileMigrationSource(dir string) *FileSystemMigrationSource
NewFileSource A set of migrations loaded from a directory.
func NewHttpFileSystemMigrationSource ¶
func NewHttpFileSystemMigrationSource(fs http.FileSystem) *FileSystemMigrationSource
NewHttpFileSystemMigrationSource A set of migrations loaded from an http.FileServer
func (*FileSystemMigrationSource) FindMigrations ¶
func (fs *FileSystemMigrationSource) FindMigrations() ([]*Migration, error)
type Logger ¶
type Logger interface { Tracef(format string, v ...any) Infof(format string, v ...any) Errorf(format string, v ...any) }
Logger is the type that gorp uses to log SQL statements. See DbMap.TraceOn.
type MemoryMigrationSource ¶
type MemoryMigrationSource struct {
Migrations []*Migration
}
MemoryMigrationSource A hardcoded set of migrations, in-memory.
func NewMemoryMigrationSource ¶
func NewMemoryMigrationSource(migrations []*Migration) *MemoryMigrationSource
NewMemoryMigrationSource A hardcoded set of migrations, in-memory.
func (*MemoryMigrationSource) FindMigrations ¶
func (m *MemoryMigrationSource) FindMigrations() ([]*Migration, error)
type Migration ¶
type Migration struct { Id string Up []string Down []string DisableTransactionUp bool DisableTransactionDown bool }
func (*Migration) NumberPrefixMatches ¶
func (*Migration) VersionInt ¶
type MigrationDirection ¶
type MigrationDirection int
const ( Up MigrationDirection = iota + 1 Down )
type MigrationExecutor ¶
type MigrationExecutor struct { // TableName name of the table used to store migration info. TableName string // SchemaName schema that the migration table be referenced. SchemaName string // IgnoreUnknown skips the check to see if there is a migration // ran in the database that is not in MigrationSource. // // This should be used sparingly as it is removing a safety check. IgnoreUnknown bool // CreateTable disable the creation of the migration table CreateTable bool // CreateSchema disable the creation of the migration schema CreateSchema bool Logger Logger }
MigrationExecutor provides database parameters for a migration execution
func NewMigrationExecutor ¶
func NewMigrationExecutor() *MigrationExecutor
func (*MigrationExecutor) Exec ¶
func (ex *MigrationExecutor) Exec( db *sql.DB, dialect dialect.Dialect, source MigrationSource, dir MigrationDirection, ) (int, error)
Exec Returns the number of applied migrations.
func (*MigrationExecutor) ExecContext ¶
func (ex *MigrationExecutor) ExecContext( ctx context.Context, db *sql.DB, dialect dialect.Dialect, m MigrationSource, dir MigrationDirection, ) (int, error)
ExecContext Returns the number of applied migrations.
func (*MigrationExecutor) ExecMax ¶
func (ex *MigrationExecutor) ExecMax( db *sql.DB, dialect dialect.Dialect, m MigrationSource, dir MigrationDirection, max int, ) (int, error)
ExecMax Returns the number of applied migrations.
func (*MigrationExecutor) ExecMaxContext ¶
func (ex *MigrationExecutor) ExecMaxContext( ctx context.Context, db *sql.DB, dialect dialect.Dialect, source MigrationSource, dir MigrationDirection, max int, ) (int, error)
ExecMaxContext Returns the number of applied migrations, but applies with an input context.
func (*MigrationExecutor) ExecVersion ¶
func (ex *MigrationExecutor) ExecVersion( db *sql.DB, dialect dialect.Dialect, source MigrationSource, dir MigrationDirection, version int64, ) (int, error)
ExecVersion Returns the number of applied migrations.
func (*MigrationExecutor) ExecVersionContext ¶
func (ex *MigrationExecutor) ExecVersionContext( ctx context.Context, db *sql.DB, dialect dialect.Dialect, source MigrationSource, dir MigrationDirection, version int64, ) (int, error)
func (*MigrationExecutor) GetMigrationRecords ¶
func (ex *MigrationExecutor) GetMigrationRecords(ctx context.Context, db *sql.DB, dialect dialect.Dialect) ([]MigrationRecord, error)
func (*MigrationExecutor) PlanMigration ¶
func (ex *MigrationExecutor) PlanMigration( ctx context.Context, db *sql.DB, dialect dialect.Dialect, source MigrationSource, dir MigrationDirection, max int, ) ([]*PlannedMigration, *MigrationRepository, error)
PlanMigration Plan a migration.
func (*MigrationExecutor) PlanMigrationToVersion ¶
func (ex *MigrationExecutor) PlanMigrationToVersion( ctx context.Context, db *sql.DB, dialect dialect.Dialect, source MigrationSource, dir MigrationDirection, version int64, ) ([]*PlannedMigration, *MigrationRepository, error)
PlanMigrationToVersion Plan a migration to version.
func (*MigrationExecutor) SkipMax ¶
func (ex *MigrationExecutor) SkipMax(ctx context.Context, db *sql.DB, dialect dialect.Dialect, m MigrationSource, dir MigrationDirection, max int) (int, error)
SkipMax Skip a set of migrations Will skip at most `max` migrations. Pass 0 for no limit. Returns the number of skipped migrations.
type MigrationRecord ¶
func GetMigrationRecords ¶
type MigrationRepository ¶
type MigrationRepository struct {
// contains filtered or unexported fields
}
func NewMigrationRepository ¶
func (*MigrationRepository) CreateSchema ¶
func (r *MigrationRepository) CreateSchema(ctx context.Context) error
func (*MigrationRepository) CreateTable ¶
func (r *MigrationRepository) CreateTable(ctx context.Context) error
func (*MigrationRepository) DeleteMigration ¶
func (r *MigrationRepository) DeleteMigration(ctx context.Context, id string) error
func (*MigrationRepository) ExecContext ¶
func (r *MigrationRepository) ExecContext(ctx context.Context, query string, args ...any) (sql.Result, error)
Exec runs an arbitrary SQL statement. args represent the bind parameters. This is equivalent to running: Exec() using database/sql
func (*MigrationRepository) ListMigration ¶
func (r *MigrationRepository) ListMigration(ctx context.Context) ([]MigrationRecord, error)
func (*MigrationRepository) QueryContext ¶
func (*MigrationRepository) SaveMigration ¶
func (r *MigrationRepository) SaveMigration(ctx context.Context, record MigrationRecord) error
type MigrationSource ¶
type PlanError ¶
PlanError happens where no migration plan could be created between the sets of already applied migrations and the currently found. For example, when the database contains a migration which is not among the migrations list found for an operation.