migration

package module
v0.20.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 22, 2020 License: Apache-2.0 Imports: 14 Imported by: 21

README

Migration

GoDoc Tests Status Test Coverage

Simple and pragmatic migrations for Go applications.

Features

  • Super simple driver interface to allow easy implementation for more database/migration drivers.
  • Embeddable migration files.
  • Support for up/down migrations.
  • Atomic migrations (where possible, depending on database support).
  • Support for using Go code as migrations

Drivers

  • Apache Phoenix
  • Golang (runs generic go functions)
  • MySQL
  • PostgreSQL
  • SQLite

Quickstart

// Create migration source
dir := pkger.Include("/migrations") // Remember to include forward slash at the beginning of the directory's name

packrSource := &migration.PkgerMigrationSource{
    Dir: dir,
}

// Create driver
driver, err := mysql.New("root:@tcp(localhost)/mydatabase?multiStatements=true")

// Run all up migrations
applied, err := Migrate(driver, packrSource, migration.Up, 0)

// Remove the last 2 migrations
applied, err := Migrate(driver, packrSource, migration.Down, 2)

Writing migrations

Migrations are extremely simple to write:

  • Separate your up and down migrations into different files. For example, 1_init.up.sql and 1_init.down.sql.
  • Prefix your migration with a number or timestamp for versioning: 1_init.up.sql or 1475813115_init.up.sql.
  • The file-extension can be anything you want, but must be present. For example, 1_init.up.sql is valid, but 1_init.up is not,
  • Note: Underscores (_) must be used to separate the number and description in the filename.

Let's say we want to write our first migration to initialize the database.

In that case, we would have a file called 1_init.up.sql containing SQL statements for the up migration:

CREATE TABLE test_data (
  id BIGINT NOT NULL PRIMARY KEY,
)

We also create a 1_init.down.sql file containing SQL statements for the down migration:

DROP TABLE IF EXISTS test_data

By default, migrations are run within a transaction. If you do not want a migration to run within a transaction, start the migration file with -- +migration NoTransaction:

-- +migration NoTransaction

CREATE TABLE test_data1 (
  id BIGINT NOT NULL PRIMARY KEY,
)

CREATE TABLE test_data2 (
  id BIGINT NOT NULL PRIMARY KEY,
)

If you would like to create stored procedures, triggers or complex statements that contain semicolns, use BeginStatement and EndStatement to delineate them:

CREATE TABLE test_data1 (
  id BIGINT NOT NULL PRIMARY KEY,
)

CREATE TABLE test_data2 (
  id BIGINT NOT NULL PRIMARY KEY,
)

-- +migration BeginStatement
CREATE TRIGGER`test_trigger_1`BEFORE UPDATE ON`test_data1`FOR EACH ROW BEGIN
		INSERT INTO test_data2
		SET id = OLD.id;
END
-- +migration EndStatement

Embedding migration files

Using pkger

This is the recommended method for embedding migration files as the author of packr has announced he will be focusing his efforts on pkger.

Assuming your migration files are in Assuming your migration files are in migrations/, initialize pkger and a PkgerMigrationSource:

dir := pkger.Include("/test-migrations") // Remember to include forward slash at the beginning of the directory's name

pkgerSource := &PkgerMigrationSource{
    Dir: dir,
}

During development, pkger will read the migration files from disk. When building for production, run pkger to generate a Go file containing your migrations. For more information, see the pkger documenation.

Using packr

Assuming your migration files are in migrations/, initialize a PackrMigrationSource:

packrSource := &migration.PackrMigrationSource{
	Box: packr.New("migrations", "migrations"),
}

If your migrations are contained in a subdirectory inside your packr box, you can point to it using the Dir property:

packrSource := &migration.PackrMigrationSource{
	Box: packr.New("migrations", "."),
	Dir: "migrations",
}

During development, packr will read the migration files from disk. When building for production, run packr to generate a Go file containing your migrations, or use packr build to build for your binary. For more information, see the packr documenation.

Using go-bindata

Note: We recommend using packr as it allows you to use migrations from disk during development

In the simplest case, assuming your migration files are in migrations/, just run:

go-bindata -o bindata.go -pkg myapp migrations/

Then, use GoBindataMigrationSource to find the migrations:

goBindataSource := &migration.GoBindataMigrationSource{
    Asset:    Asset,
    AssetDir: AssetDir,
    Dir:      "test-migrations",
}

The Asset and AssetDir functions are generated by go-bindata.

Using Go for migrations

Sometimes, we might be working with a database or have a situation where the query language is not expressive enough to perform the required migrations. For example, we might have to get some data out of the database, perform some transformations and then write it back. For these type of situations, you can use Go for migrations.

When using Go for migrations, create a golang.Source using golang.NewSource(). Then, simply add migrations to the source using the AddMigration() method. You will need to pass in the name of the migration without the extension and direction, e.g. 1_init. For the second parameter, pass in the direction (migration.Up or migration.Down) and for the third parameter, pass in a function or method with this signature: func() error for running the migration.

Finally, you need to define 2 functions:

  • A function for writing or deleting an applied migration matching this signature: func(id string, direction migration.Direction) error
  • A function for getting a list of applied migrations matching this signature: func() ([]string, error)

These are required for initializing the driver:

driver, err := golang.New(source, updateVersion, applied)

Here's a quick example:

source := migration.NewGolangMigrationSource()

source.AddMigration("1_init", migration.Up, func() error {
    // Run up migration here
})

source.AddMigration("1_init", migration.Down, func() error {
    // Run down migration here
})

// Define functions
applied := func() ([]string, error) {
    // Return list of applied migrations
}

updateVersion := func(id string, direction migration.Direction) error {
    // Write or delete applied migration in storage
}

// Create driver
driver, err := golang.New(source, updateVersion, applied)

// Run migrations
count, err = migration.Migrate(driver, source, migration.Up, 0)

TODO (Pull requests welcomed!)

  • Command line program to run migrations
  • More drivers

Why yet another migration library?

We wanted a migration library with the following features:

  • Open to extension for all sorts of databases, not just database/sql drivers or an ORM.
  • Easily embeddable in a Go application.
  • Support for embedding migration files directly into the app.

We narrowed our focus down to 2 contenders: sql-migrate and migrate

sql-migrate leans heavily on the gorp ORM library to perform migrations. Unfortunately, this means that we were restricted to databases supported by gorp. It is easily embeddable in a Go app and supports embedding migration files directly into the Go binary. If database support was a bit more flexible, we would have gone with it.

migrate is highly extensible, and adding support for another database is extremely trivial. However, due to it using the scheme in the dsn to determine which database driver to use, it prevented us from easily implementing an Apache Phoenix driver, which uses the scheme to determine if we should connect over http or https. Due to the way the project is structured, it was also almost impossible to add support for embeddable migration files without major changes.

Contributing

We automatically run some linters using golangci-lint to check code quality before merging it. This is executed using a Makefile target.

You should run and ensure all the checks pass locally before submitting a pull request. The version of golangci-lint to be used is pinned in go.mod.

To execute the linters:

  1. Install make.
  2. Install golangci-lint by executing go install github.com/golangci/golangci-lint/cmd/golangci-lint.
  3. Execute make sanity-check.

License

This library is licensed under the Apache 2 License.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Migrate 

func Migrate(driver Driver, migrations Source, direction Direction, max int) (int, error)

Migrate runs a migration using a given driver and MigrationSource. The direction defines whether the migration is up or down, and max is the maximum number of migrations to apply. If max is set to 0, then there is no limit on the number of migrations to apply.

func SetLogger added in v0.14.1 

func SetLogger(l *log.Logger)

SetLogger will set the logger to be used during migrations

Types

type Direction added in v0.2.0 

type Direction int

Direction type up/down 

const (
	Up Direction = iota
	Down
)

Constants for direction

func (Direction) String added in v0.14.1 

func (d Direction) String() string

String returns a string representation of the direction

type Driver 

type Driver interface {
	// Close is the last function to be called.
	// Close any open connection here.
	Close() error

	// Migrate is the heart of the driver.
	// It will receive a PlannedMigration which the driver should apply
	// to its backend or whatever.
	Migrate(migration *PlannedMigration) error

	// Version returns all applied migration versions
	Versions() ([]string, error)
}

Driver is the interface type that needs to implemented by all drivers.

type GoBindataMigrationSource added in v0.11.0 

type GoBindataMigrationSource struct {
	// Asset should return content of file in path if exists
	Asset func(path string) ([]byte, error)

	// AssetDir should return list of files in the path
	AssetDir func(path string) ([]string, error)

	// Path in the bindata to use.
	Dir string
}

GoBindataMigrationSource is a MigrationSource that uses migration files embedded in a Go application using go-bindata.

func (GoBindataMigrationSource) GetMigrationFile added in v0.11.0 

func (a GoBindataMigrationSource) GetMigrationFile(name string) (io.Reader, error)

GetMigrationFile gets a gobindata migration file

func (GoBindataMigrationSource) ListMigrationFiles added in v0.11.0 

func (a GoBindataMigrationSource) ListMigrationFiles() ([]string, error)

ListMigrationFiles returns a list of gobindata migration files

type GolangMigrationSource added in v0.15.0 

type GolangMigrationSource struct {
	sync.Mutex
	// contains filtered or unexported fields
}

GolangMigrationSource implements migration.Source

func NewGolangMigrationSource added in v0.15.0 

func NewGolangMigrationSource() *GolangMigrationSource

NewGolangMigrationSource creates a source for storing Go functions as migrations.

func (*GolangMigrationSource) AddMigration added in v0.15.0 

func (s *GolangMigrationSource) AddMigration(file string, direction Direction, migration func() error)

AddMigration adds a new migration to the source. The file parameter follows the same conventions as you would use for a physical file for other types of migrations, however you should omit the file extension. Example: 1_init.up and 1_init.down

func (*GolangMigrationSource) GetMigration added in v0.15.0 

func (s *GolangMigrationSource) GetMigration(file string) func() error

GetMigration gets a golang migration

func (*GolangMigrationSource) GetMigrationFile added in v0.15.0 

func (s *GolangMigrationSource) GetMigrationFile(file string) (io.Reader, error)

GetMigrationFile retrieves a migration given the filename.

func (*GolangMigrationSource) ListMigrationFiles added in v0.15.0 

func (s *GolangMigrationSource) ListMigrationFiles() ([]string, error)

ListMigrationFiles lists the available migrations in the source

type MemoryMigrationSource 

type MemoryMigrationSource struct {
	Files map[string]string
}

MemoryMigrationSource is a MigrationSource that uses migration sources in memory. It is mainly used for testing.

func (MemoryMigrationSource) GetMigrationFile 

func (m MemoryMigrationSource) GetMigrationFile(name string) (io.Reader, error)

GetMigrationFile gets a memory migration file

func (MemoryMigrationSource) ListMigrationFiles 

func (m MemoryMigrationSource) ListMigrationFiles() ([]string, error)

ListMigrationFiles returns a list of memory migration files

type Migration 

type Migration struct {
	ID   string
	Up   *parser.ParsedMigration
	Down *parser.ParsedMigration
}

Migration represents a migration, containing statements for migrating up and down.

func (Migration) Less 

func (m Migration) Less(other *Migration) bool

Less compares two migrations to determine how they should be ordered.

func (Migration) NumberPrefixMatches 

func (m Migration) NumberPrefixMatches() []string

NumberPrefixMatches returns a list of string matches

func (Migration) VersionInt 

func (m Migration) VersionInt() int64

VersionInt converts the migration version to an 64-bit integer.

type PackrBox added in v0.11.0 

type PackrBox interface {
	List() []string
	Find(name string) ([]byte, error)
}

PackrBox avoids pulling in the packr library for everyone, mimics the bits of packr.Box that we need.

type PackrMigrationSource added in v0.11.0 

type PackrMigrationSource struct {
	Box PackrBox

	// The path in the packr box to use
	Dir string
}

PackrMigrationSource holds the box and dir info

func (PackrMigrationSource) GetMigrationFile added in v0.11.0 

func (p PackrMigrationSource) GetMigrationFile(name string) (io.Reader, error)

GetMigrationFile gets a packr migration file

func (PackrMigrationSource) ListMigrationFiles added in v0.11.0 

func (p PackrMigrationSource) ListMigrationFiles() ([]string, error)

ListMigrationFiles returns a list of packr migration files

type PkgerMigrationSource added in v0.19.0 

type PkgerMigrationSource struct {
	// The path in to use
	Dir string
}

PkgerMigrationSource holds the underlying pkger and dir info

func (PkgerMigrationSource) GetMigrationFile added in v0.19.0 

func (p PkgerMigrationSource) GetMigrationFile(name string) (io.Reader, error)

GetMigrationFile gets a pkger migration file

func (PkgerMigrationSource) ListMigrationFiles added in v0.19.0 

func (p PkgerMigrationSource) ListMigrationFiles() ([]string, error)

ListMigrationFiles returns a list of pkger migration files

type PlannedMigration 

type PlannedMigration struct {
	*Migration
	Direction Direction
}

PlannedMigration is a migration with a direction defined. This allows the driver to work out how to apply the migration.

type Source added in v0.2.0 

type Source interface {
	ListMigrationFiles() ([]string, error)
	GetMigrationFile(file string) (io.Reader, error)
}

Source is an interface that defines how a source can find and read migration files.

Source Files

View all Source files

Directories

Path Synopsis
driver
golang
mysql
phoenix
postgres
sqlite

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.