fig

README

GoFig

GoFig is a tool for managing Firestore database migrations. This project aims to achieve the following:

1. The end user will have visibility into all pending migration changes and effects in relation to the active state of their database. The user can then decide to push or cancel the migration.
2. When a migration is pushed to the database, all the changes presented to the end user are implemented and no other changes/effects are implemented by the migrator.
3. All changes made by the migrator in a migration job can be completely reversed by staging and pushing the generated _rollback migration file.

Install

go get github.com/aaronhough/GoFig

Initialize Migrator

The migrator is initialized with a few configuration parameters. If your storage path starts with [firestore]/, migrations will be stored on your database (suffix path must be a collection). The Name value will be the file/doc name so it should only contain alphanumeric digits and underscores/dashes A-Z,a-z,0-9,_,-.

import fig "github.com/aaronhough/GoFig"

config := fig.Config{
    KeyPath: "~/project/.keys/my-firestore-admin-key.json",
    StoragePath: "~/project/storage",
    Name: "my-migration",
}

fg, err := fig.New(config)

defer fg.Close()

Stage a New Migration

Stage each change into the migrator using the Stage utility.

// Sample data.
data := map[string]any{
    "foo": "bar",
    "fiz": false,
    "buz": map[string][]int{
        "a": { 2, 2, 3 },
    },
}

// Stage options are Add, Update, Set, and Delete.
fg.Stage().Add("fig/fog", data)
fg.Stage().Update("foo/bar", map[string]string{ "hello": "world" })

Save a Migration To Storage

Save a staged migration to storage then load and run it at a later time.

// A new file is saved to `StoragePath` folder.
fg.StoreMigration()

Load an Existing Migration

The migrator will search for an existing migration file/doc in the StoragePath location that matches the configured Name.

// If a file matching the migration `Name` exists, it will be loaded.
fg.LoadFromStorage()

Run a Migration

The interactive migration shell will present the changes and prompt for confirmation. On confirmation, the changes will be executed against the database and a rollback migration file will be saved to the StoragePath location.

// Launch the interactive shell.
fg.ManageStagedMigration()

Rollback

Locate the _rollback file/doc generated by the target migration job. Ensure the migration config matches the name of the rollback file. Load and run the migration.

// Migration file expected at ~/project/storage/my-migration_rollback.json
config := fig.Config{
    KeyPath: "~/project/.keys/my-admin-key.json",
    StoragePath: "~/project/storage",
    Name: "my-migration_rollback",
}

fg, err := fig.New(config)
defer fg.Close()

fg.LoadFromStorage()
fg.ManageStagedMigration()

Complex types

Note some examples of supported complex types. Document references, date-times, and deletions are represented in order:

data := map[string]any{
    "ref": fg.RefField("fig/fog"),
    "time": time.Now(),
    "prev": fg.DeleteField(),
}

Expansions

Since the migrator can load any migration file, feel free to use your own languages/scripts to build migrations then load them by path/name with GoFig. You will need to serialize complex types as follows:

• Timestamp: "<time>2023-05-13T13:44:40.522Z<time>"
• Document reference: "<ref>fig/fog<ref>"
• Delete: "<delete>!delete<delete>"

The actual migration file simply needs to host an array of serialized changeUnits. Each change contains a docPath, a patch, and a numeric command. Commands are 0, 1, 2, 3, 4 which represent MigratorUnknown, MigratorUpdate, MigratorSet, MigratorAdd, and MigratorDelete respectively.

To Do

• Abbreviate large diffs in terminal

Documentation

Index

• Constants
• type Change
  • func NewChange(docPath string, before map[string]any, patch map[string]any, command Command, ...) *Change
  • func (c *Change) Present() ([]string, string)
  • func (c *Change) SolveChange() error
• type Command
• type Config
• type Diff
• type Fig
  • func New(config Config) (*Fig, error)
  • func (c *Fig) Close()
  • func (c *Fig) DeleteField() any
  • func (c *Fig) LoadFromStorage() error
  • func (c *Fig) ManageStagedMigration()
  • func (c *Fig) RefField(docPath string) any
  • func (c *Fig) SaveToStorage() error
  • func (c *Fig) Stage() FigStager
• type FigMigrator
• type FigStager
• type GoFig
• type Migration
• type Migrator
  • func NewMigrator(storagePath string, database figFirestore, name string) *Migrator
  • func (m *Migrator) LoadMigration() error
  • func (m *Migrator) PrepMigration() error
  • func (m *Migrator) PresentMigration()
  • func (m *Migrator) RunMigration()
  • func (m *Migrator) SetDeleteFlag(flag string)
  • func (m *Migrator) Stage() FigStager
  • func (m *Migrator) Store(target any, tag string) error
  • func (m *Migrator) StoreMigration() error
• type Stager
  • func (s Stager) Add(colPath string, data map[string]any) error
  • func (s Stager) Delete(docPath string) error
  • func (s Stager) Set(docPath string, data map[string]any) error
  • func (s Stager) Unknown(docPath string, data map[string]any) error
  • func (s Stager) Update(docPath string, data map[string]any) error
• type WorkUnit

Constants

const (
	Serialize transformMode = iota
	DeSerialize
	Exclude
)

Types

type Change

type Change struct {
	// contains filtered or unexported fields
}

Change represents one change on one document. A change must contain enough data points to be solved. For example, given a before and a patch, we can solve for the after value.

func NewChange

func NewChange(docPath string,
	before map[string]any,
	patch map[string]any,
	command Command,
	database figFirestore) *Change

NewChange is a Change factory.

func (*Change) Present

func (c *Change) Present() ([]string, string)

Present returns a pretty representation of the change for printing to stdout.

func (*Change) SolveChange

func (c *Change) SolveChange() error

SolveChange will attempt to solve for all needed Change values given the current state.

type Command

type Command int

Command is an enum of supported Change command types for example 'Update'.

const (
	MigratorUnknown Command = iota
	MigratorUpdate
	MigratorSet
	MigratorAdd
	MigratorDelete
)

type Config

type Config struct {
	KeyPath     string
	StoragePath string
	Name        string
}

Config is the expected structure for Fig config

type Diff

type Diff struct {
	Diff string `json:"diff" firestore:"diff,omitempty"`
}

Diff represents how we want to store our diffs

type Fig

type Fig struct {
	// contains filtered or unexported fields
}

Fig is meant to orchestrate Migrator functionality.

func New

func New(config Config) (*Fig, error)

New is a Fig factory. Defer *Fig.Close() after initialization.

func (*Fig) Close

func (c *Fig) Close()

Close should be deferred on initialization to handle any database session cleanup.

func (*Fig) DeleteField

func (c *Fig) DeleteField() any

DeleteField is a shortcut to the controlled database DeleteField.

func (*Fig) LoadFromStorage

func (c *Fig) LoadFromStorage() error

LoadFromStorage attempts to load a pre staged migration from a file if it exists in the storagePath folder

func (*Fig) ManageStagedMigration

func (c *Fig) ManageStagedMigration()

ManageStagedMigration launches the interactive CLI script.

func (*Fig) RefField

func (c *Fig) RefField(docPath string) any

RefField is a shortcut to the controlled database RefField.

func (*Fig) SaveToStorage

func (c *Fig) SaveToStorage() error

SaveToStorage attempts to save a migration staged in runtime memory to a file in the storagePath folder

func (*Fig) Stage

func (c *Fig) Stage() FigStager

Stage exposes the Migrator Stager as an API to the end user.

type FigMigrator

type FigMigrator interface {
	SetDeleteFlag(flag string)
	PrepMigration() error
	PresentMigration()
	RunMigration()
	LoadMigration() error
	StoreMigration() error

	Stage() FigStager
	// contains filtered or unexported methods
}

FigMigrator described what a GoFig Migrator implementeation should do.

type FigStager

type FigStager interface {
	Update(docPath string, data map[string]any) error
	Set(docPath string, data map[string]any) error
	Add(colPath string, data map[string]any) error
	Delete(docPath string) error
	Unknown(docPath string, data map[string]any) error
}

FigStager represents the staging API.

type GoFig

type GoFig interface {
	Close()
	Stage() FigStager
	LoadFromStorage() error
	SaveToStorage() error
	ManageStagedMigration()
	DeleteField() any
	RefField(docPath string) any
}

GoFig represents the main Fig API.

type Migration

type Migration struct {
	DatabaseName string     `json:"databaseName" firestore:"databaseName,omitempty"`
	Timestamp    time.Time  `json:"timestamp" firestore:"timestamp,omitempty"`
	ChangeUnits  []WorkUnit `json:"changeUnits" firestore:"changeUnits,omitempty"`
	Executed     bool       `json:"executed" firestore:"executed,omitempty"`
}

Migration represents all the instructions needed by the migrator to orchestrate a job. All migration jobs including rollbacks take this form.

type Migrator

type Migrator struct {
	// contains filtered or unexported fields
}

Migrator is the API for performing migration tasks within a job it implements FigMigrator.

func NewMigrator

func NewMigrator(storagePath string, database figFirestore, name string) *Migrator

NewMigrator is a Migrator factory.

func (*Migrator) LoadMigration

func (m *Migrator) LoadMigration() error

LoadMigration will look for an existing migration file matching this Migrator's name. If a file exists, the state of the migrator will be replaced by the contents of the file. This is the preferred workflow for loading a rollback.

func (*Migrator) PrepMigration

func (m *Migrator) PrepMigration() error

PrepMigration is run after all changes are staged. This function validates and solves all of the changes. No changes are pushed to the database.

func (*Migrator) PresentMigration

func (m *Migrator) PresentMigration()

PresentMigration prints all the staged changes to stdout for review.

func (*Migrator) RunMigration

func (m *Migrator) RunMigration()

RunMigration executes all of the staged changes against the database.

func (*Migrator) SetDeleteFlag

func (m *Migrator) SetDeleteFlag(flag string)

SetDeleteFlag updates the Migrator delete flag with a new string value. When this value is on a Change field, that field is deleted from the database document when the change is pushed.

func (*Migrator) Stage

func (m *Migrator) Stage() FigStager

Stage is a Stager factory

func (*Migrator) Store

func (m *Migrator) Store(target any, tag string) error

func (*Migrator) StoreMigration

func (m *Migrator) StoreMigration() error

StoreMigration converts the Migrator state to a Migration file and stores it to disc.

type Stager

type Stager struct {
	// contains filtered or unexported fields
}

Stager is an abstraction on top of Migrator which is used as an API to stage new Change units on the Migrator.

func (Stager) Add

func (s Stager) Add(colPath string, data map[string]any) error

Add stages a new Add change on the Migrator.

func (Stager) Delete

func (s Stager) Delete(docPath string) error

Delete stages a new Delete change on the Migrator.

func (Stager) Set

func (s Stager) Set(docPath string, data map[string]any) error

Set stages a new Set change on the Migrator.

func (Stager) Unknown

func (s Stager) Unknown(docPath string, data map[string]any) error

Unknown stages a new change on the Migrator of an Unknown command type.

func (Stager) Update

func (s Stager) Update(docPath string, data map[string]any) error

Update stages a new Update change on the Migrator.

type WorkUnit

type WorkUnit struct {
	DocPath string         `json:"docPath" firestore:"docpath,omitempty"`
	Patch   map[string]any `json:"patch,omitempty" patch:"executed,omitempty"`
	Command Command        `json:"command,omitempty" firestore:"command,omitempty"`
}

WorkUnit is one change mapped to one document within a migration. You cannot have multiple work units pointing to the same document in a migration.