migrations

package module
v2.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 21, 2025 License: Apache-2.0 Imports: 6 Imported by: 1

README

migrations

Migrations is an abstraction for a migration system and it can migrate anything.

How to use

This example can be found in the exapmles/postgres/main.go file.

package main

import (
	"context"
	"database/sql"
	"embed"

	_ "github.com/lib/pq"
	"go.uber.org/zap"

	"github.com/jamillosantos/migrations/v2"
	"github.com/jamillosantos/migrations/v2/reporters"
	migrationsql "github.com/jamillosantos/migrations/v2/sql"
)

//go:embed migrations/*.sql
var migrationsFolder embed.FS

func main() {
	logger, err := zap.NewDevelopmentConfig().Build()
	if err != nil {
		panic(err)
	}
	defer func() {
		_ = logger.Sync()
    }()

	db, err := sql.Open("postgres", "postgres://postgres:postgres@localhost:5432/postgres?sslmode=disable")
	if err != nil {
		panic(err)
	}

	ctx := context.Background()

	s, err := migrationsql.SourceFromFS(func() migrationsql.DBExecer {
		return db
	}, migrationsFolder, "migrations")
	if err != nil {
		panic(err)
	}

	t, err := migrationsql.NewTarget(db)
	if err != nil {
		panic(err)
	}

	reporters.NewZapReporter(logger)

	_, err = migrations.Migrate(ctx, s, t, migrations.WithRunnerOptions(migrations.WithReporter(reporters.NewZapReporter(logger))))
	if err != nil {
		panic(err)
	}
}

Generating new migration files

go run github.com/jamillosantos/migrations/v2/cli/migrations create -destination=migrations

This will create an emtpy migration file in the migrations with the correct timestamp and a description:

migrations/20210101000000_my_migration.sql`

How it works

The migrations package is a simple abstraction for a migration system. It is able to migrate anything that migrations would apply.

It has a simple design split into 3 main components:

Source

A Source is the media that persists the migrations themselves. It is an entity that will load the migrations from SQL files, or from a S3 bucket (or anything else you need!).

We have implemented support for the fs.FS interface, so you can use the embed.FS to load migrations from the binary.

Also, fnc was created to allow you to load migrations from a function AND they can be used together. Please, check the examples/fncmigrations folder.

Target

A Target is what the migrations persisted will be stored. If you are dealing with relational databases, like postgres, you would use the _migrations table by default. However, you could create a JSON file that would store the executed migrations.

Runner

The runner is the entity that will run the migrations. It will use the Source and Target to execute the migrations to retrive information about the available migrations (from the Source) and what migrations are already applied ( from the Target).

Extending

The migrations package is designed to be extended and you will, probably, only work with Sources and/or Targets.

1. Source

In some migration systems (like golang-migrate/migrate) are stored as .sql files and are stored into a directory as <timestamp>_<description>.(up|down).sql. In other systems, the migration is a func that need to do some complex work and should connect many components before the actual database migration.

So, in the first example, the Source is a directory containing a bunch of .sql files with specific names. In the second example, the Source are function that should be organized chronologically.

Hence, Source is the media that persists the migrations themselves. In practice, it is just an interface{} with a bunch of methods that will list all available migrations (check the code).

TODO: Link the Source interface.

2. Target

A Target are what the migrations are transforming. If you are dealing with relational databases, like postgres, you would use a TargetSQL implementation (we provide one, check the our examples folder|TODO).

In practice, a Target is just an interface{} with a bunch of methods that will list executed migrations, mark and unmark migrations as executed (check the code).

TODO: Link the Target interface.

3. Executer

An Executer integrations Source and Target and is responsible for step actions, like Do and Undo. Each call will step forward or backward one migration at a time.

4. Runner

Runners are, also, concrete. They capture the developer intentions and call the Executer.

Let's say that you want to migrate your system. By that, you mean to run all pending migrations. So the runner will use the Executer.Do calling it multiple times to get all migrations executed.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrMigrationNotFound          = errors.New("migration not found")
	ErrNoCurrentMigration         = errors.New("no current migration")
	ErrCurrentMigrationNotFound   = errors.New("current migration not found in the list")
	ErrCurrentMigrationMoreRecent = errors.New("current migration is more recent than target migration")
	ErrNoMigrationsAvailable      = errors.New("no migrations available")
	ErrMigrationNotUndoable       = errors.New("migration cannot be undone")

	// ErrStepOutOfIndex is returned when a `StepResolver` cannot resolve a
	// migration due to the resolved index be outside of the migration list.
	ErrStepOutOfIndex = errors.New("step out of bounds")

	// ErrMigrationNotListed is returned when a migration is not found in the
	// `Source` list.
	ErrMigrationNotListed = errors.New("migration not in the source list")

	// ErrStaleMigrationDetected is returned when a migration with an ID eariler of the current applied migration is
	// detected.
	ErrStaleMigrationDetected = errors.New("stale migration detected")

	// ErrInvalidAction is returned when, while executing, the `Action.Action`
	// has an invalid value.
	ErrInvalidAction = errors.New("undefined action")

	ErrDirtyMigration = errors.New("migration was started but not completed and now it is in a dirty state")
)
View Source 
var (
	// DefaultMigrationIDFormat is the default format for the migrations ID.
	DefaultMigrationIDFormat = "20060102150405"
)
View Source 
var ErrMigrationAlreadyExists = errors.New("migration already exists")

Functions

func NewQueryError 

func NewQueryError(err error, query string) error

func WrapMigration 

func WrapMigration(err error, migration Migration) *migrationError

WrapMigration creates a `MigrationError` based on an existing error.

func WrapMigrationID 

func WrapMigrationID(err error, migrationID string) *migrationIDError

WrapMigrationID creates a `MigrationCodeError` based on an existing error.

Types

type Action 

type Action struct {
	Action    ActionType
	Migration Migration
}

type ActionPLanner 

type ActionPLanner func(source Source, target Target) Planner

func StepPlanner 

func StepPlanner(step int) ActionPLanner

StepPlanner build an ActionPlanner that will step the migrations by the given number. If the given number is positive, then the step will be forward, otherwise, it will be backward.

type ActionType 

type ActionType string
const (
	ActionTypeDo   ActionType = "do"
	ActionTypeUndo ActionType = "undo"
)

type AfterExecuteInfo 

type AfterExecuteInfo struct {
	Plan  Plan
	Stats *ExecutionResponse
	Err   error
}

type AfterExecuteMigrationInfo 

type AfterExecuteMigrationInfo struct {
	ActionType ActionType
	Migration  Migration
	Err        error
}

type BaseMigration 

type BaseMigration struct {
	// contains filtered or unexported fields
}

func NewMigration 

func NewMigration(id, description string, do, undo migrationFunc) *BaseMigration

func (*BaseMigration) CanUndo 

func (migration *BaseMigration) CanUndo() bool

CanUndo is a flag that mark this flag as undoable.

func (*BaseMigration) Description 

func (migration *BaseMigration) Description() string

Description is the humanized description for the migration.

func (*BaseMigration) Do 

func (migration *BaseMigration) Do(ctx context.Context) error

Do will execute the migration.

func (*BaseMigration) ID 

func (migration *BaseMigration) ID() string

ID identifies the migration. Through the ID, all the sorting is done.

func (*BaseMigration) Next 

func (migration *BaseMigration) Next() Migration

Next will link this migration with the next. This link should be created by the source while it is being loaded.

func (*BaseMigration) Previous 

func (migration *BaseMigration) Previous() Migration

Previous will link this migration with the previous. This link should be created by the Source while it is being loaded.

func (*BaseMigration) SetNext 

func (migration *BaseMigration) SetNext(value Migration) Migration

SetNext will set the next migration

func (*BaseMigration) SetPrevious 

func (migration *BaseMigration) SetPrevious(value Migration) Migration

SetPrevious will set the previous migration

func (*BaseMigration) String 

func (migration *BaseMigration) String() string

String will return a representation of the migration into a string format for user identification.

func (*BaseMigration) Undo 

func (migration *BaseMigration) Undo(ctx context.Context) error

Undo will undo the migration.

type BeforeExecuteInfo 

type BeforeExecuteInfo struct {
	Plan Plan
}

type BeforeExecuteMigrationInfo 

type BeforeExecuteMigrationInfo struct {
	ActionType ActionType
	Migration  Migration
}

type ExecuteRequest 

type ExecuteRequest struct {
	Plan Plan
}

type ExecutionResponse 

type ExecutionResponse struct {
	Successful []*Action
	Errored    []*Action
}

func Migrate 

func Migrate(ctx context.Context, source Source, target Target, opts ...MigrateOption) (ExecutionResponse, error)

type MigrateOption 

type MigrateOption func(*migrateOptions)

func WithPlanner 

func WithPlanner(planner ActionPLanner) MigrateOption

WithPlanner sets the planner to be used by the migration process. Default planner is MigratePlanner.

func WithRunner 

func WithRunner(runner *Runner) MigrateOption

WithRunner sets the reporter to be used by the migration process.

func WithRunnerOptions 

func WithRunnerOptions(opts ...RunnerOption) MigrateOption

WithRunnerOptions sets the options to be used by the runner created. If a `WithRunner` is used, this is ignored.

type Migration 

type Migration interface {
	// ID identifies the migration. Through the ID, all the sorting is done.
	ID() string

	// String will return a representation of the migration into a string format
	// for user identification.
	String() string

	// Description is the humanized description for the migration.
	Description() string

	// Next will link this migration with the next. This link should be created
	// by the source while it is being loaded.
	Next() Migration

	// SetNext will set the next migration
	SetNext(Migration) Migration

	// Previous will link this migration with the previous. This link should be
	// created by the Source while it is being loaded.
	Previous() Migration

	// SetPrevious will set the previous migration
	SetPrevious(Migration) Migration

	// Do will execute the migration.
	Do(ctx context.Context) error

	// CanUndo is a flag that mark this flag as undoable.
	CanUndo() bool

	// Undo will undo the migration.
	Undo(ctx context.Context) error
}

Migration is the abstraction that defines the migration contract.

Migrations, by default, cannot be undone. But, if the final migration implement the `MigrationUndoable` interface, the system will let it be undone.

type MigrationError 

type MigrationError interface {
	Migration() Migration
	Unwrap() error
}

MigrationError wraps an error with a migration property.

type MigrationIDError 

type MigrationIDError interface {
	MigrationID() string
}

MigrationIDError wraps an error with a migration ID.

type MigrationsError 

type MigrationsError interface {
	Migrations() []Migration
}

MigrationsError wraps an error with a list of migrations.

func WrapMigrations 

func WrapMigrations(err error, migrations ...Migration) MigrationsError

type Plan 

type Plan []*Action

type Planner 

type Planner interface {
	Plan(ctx context.Context) (Plan, error)
}

func DoPlanner 

func DoPlanner(source Source, target Target) Planner

func MigratePlanner 

func MigratePlanner(source Source, target Target) Planner

MigratePlanner is an ActionPlanner that returns a Planner that plans actions to take the current version of the database to the latest.

func ResetPlanner 

func ResetPlanner(source Source, target Target) Planner

ResetPlanner is an ActionPlanner that returns a Planner that plans actions to rewind all migrations and then migrate again to the latest version.

func RewindPlanner 

func RewindPlanner(source Source, target Target) Planner

RewindPlanner is a planner that will undo all migrations starting from the current revision.

func UndoPlanner 

func UndoPlanner(source Source, target Target) Planner

type ProgressReporter 

type ProgressReporter interface {
	SetStep(current int)
	SetSteps(steps []string)
	SetTotal(total int)
	SetProgress(progress int)
}

type QueryError 

type QueryError interface {
	Query() string
}

type Repository 

type Repository struct {
	// contains filtered or unexported fields
}

Repository stores the migrations in memory and provide methods for acessing them.

func (*Repository) Add 

func (r *Repository) Add(migration Migration) error

func (*Repository) ByID 

func (r *Repository) ByID(id string) (Migration, error)

func (*Repository) List 

func (r *Repository) List(_ context.Context) ([]Migration, error)

type Runner 

type Runner struct {
	// contains filtered or unexported fields
}

Runner will receive the `Plan` from the `Planner` and execute it.

func NewRunner 

func NewRunner(source Source, target Target, options ...RunnerOption) *Runner

func (*Runner) Execute 

func (runner *Runner) Execute(ctx context.Context, req *ExecuteRequest) (ExecutionResponse, error)

Execute performs a plan, running all actions migration by migration.

Before running, Execute will check for Undo actions that cannot be performed into undoable migrations. If that happens, an `ErrMigrationNotUndoable` will be returned and nothing will be executed.

For each migration executed, the system will move the cursor to that point. So that, if any error happens during the migration execution (do or undo), the execution will be stopped and the error will be returned. All performed actions WILL NOT be rolled back.

type RunnerOption 

type RunnerOption func(*runnerOptions)

func WithReporter 

func WithReporter(reporter RunnerReporter) RunnerOption

type RunnerReporter 

type RunnerReporter interface {
	BeforeExecute(ctx context.Context, info *BeforeExecuteInfo)
	BeforeExecuteMigration(ctx context.Context, info *BeforeExecuteMigrationInfo)
	AfterExecuteMigration(ctx context.Context, info *AfterExecuteMigrationInfo)
	AfterExecute(ctx context.Context, info *AfterExecuteInfo)
}

type Source 

type Source interface {
	// Add will add a Migration to the source list.
	//
	// If the migration id cannot be found, this function should return ErrMigrationAlreadyExists.
	Add(ctx context.Context, migration Migration) error

	// Load will return the list of available migrations, sorted by ID (the older first, the newest last).
	//
	// If there is no migrations available, an ErrNoMigrationsAvailable should
	// be returned.
	Load(ctx context.Context) (Repository, error)
}

Source is responsible to list all migrations available to run.

Migrations can be stored into many medias, from Go source code files, plain SQL files, go:embed. So, this interface is responsible for abstracting how this system accepts any media to list the

func NewMemorySource added in v2.1.0 

func NewMemorySource() Source

NewMemorySource creates a source in which the migrations are stored only in memory. This is useful for `github.com/jamillosantos/migrations/v2/fnc` migrations.

type Target 

type Target interface {
	// Current returns the reference to the most recent migration applied to the system.
	//
	// If there is no migration run, the system will return an ErrNoCurrentMigration error.
	Current(ctx context.Context) (string, error)

	// Create ensures the creation of the list of migrations is done successfully. As an example, if this was an SQL
	// database implementation, this method would create the `_migrations` table.
	Create(ctx context.Context) error

	// Destroy removes the list of the applied migrations. As an example, if this was an SQL database implementation,
	// this would drop the `_migrations` table.
	Destroy(ctx context.Context) error

	// Done list all the migrations that were successfully applied.
	Done(ctx context.Context) ([]string, error)

	// Add adds a migration to the list of successful migrations.
	Add(ctx context.Context, id string) error

	// Remove removes a migration from the list of successful migrations.
	Remove(ctx context.Context, id string) error

	// FinishMigration will mark the migration as finished. This is only used when the migration is being added.
	FinishMigration(ctx context.Context, id string) error

	// StartMigration will mark the migration as dirty. This is only used when the migration is being removed.
	StartMigration(ctx context.Context, id string) error

	// Lock will try locking the migration system in such way no other instance of the process can run the migrations.
	Lock(ctx context.Context) (Unlocker, error)
}

Target is responsible for managing the state of the migration system. This interface abstracts the operations needed to list what migrations were successfully executed, what is the current migration, etc.

type Unlocker 

type Unlocker interface {
	Unlock(ctx context.Context) error
}

Unlocker abstracts an implementation for unlocking the migration system.

Source Files

View all Source files

Directories

Path Synopsis
cli
internal/cmd
migrations
examples
fncmigrations
fncmigrations/migrations
postgres
sql
drivers

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.