db

README

Migrations

Database migrations are handled by this package. All migrations should be created in separate files, following a starndard naming convetion

The migrations.go file defines an array of migrate functions that are called by the gormigrate helper. Each migration function should perform a specific migration.

Creating a new migration

Create a migration ID based on the time using the YYYYMMDDHHMM format. Example: August 21 2018 at 2:54pm would be 201808211454.

Your migration's name should be used in the file name and in the function name and should adequately represent the actions your migration is taking. If your migration is doing too much to fit in a name, you should consider creating multiple migrations.

Create a separate file in pkg/db/ following the naming schema in place: <migration_id>_<migration_name>.go. In the file, you'll create a function that returns a gormmigrate.Migration object with gormigrate.Migrate and gormigrate.Rollback functions defined.

Add the function you created in the separate file to the migrations list in pkg/db/migrations.go.

If necessary, write a test to verify the migration. See test/integration/migrations_test.go for examples.

Migration Rules

Migration IDs

Each migration has an ID that defines the order in which the migration is run.

IDs are numerical timestamps that must sort ascending. Use YYYYMMDDHHMM w/ 24 hour time for format. Example: August 21 2018 at 2:54pm would be 201808211454.

Migration IDs must be descending. If you create a migration, submit an MR, and another MR is merged before yours is able to be merged, you must update the ID to represent a date later than any previous migration.

Models in Migrations

Represent modesl inline with migrations to represent the evolution of the object over time.

For example, it is necessary to add a boolean field "hidden" to the "Account" model. This is how you would represent the account model in that migration:

type Account struct {
  Model
  Username string
  FirstName string
  LastName string
  Hidden boolean
}

err := tx.AutoMigrate(&Account{}).Error
if err != nil {
...

DO NOT IMPORT THE API PKG. When a migration imports the api pkg and uses models defined in it, the migration may work the first time it is run. The models in pkg/api are bound to change as the project grows. Eventually, the models could change so that the migration breaks, causing any new deployments to fail on your old shitty migration.

Record Deletions

If it is necessary to delete a record in a migration, be aware of a couple caveats around deleting wth gorm:

1. You must pass a record with an ID to gorm.Delete(&record), otherwise ALL RECORDS WILL BE DELETED

2. Gorm soft deletes by default. This means it sets the deleted_at field to a non-null value and any subsequent gorm calls will ignore it. If you are deleting a record that needs to be permanently deleted (like permissions), use gorm.Unscoped().Delete.

See the gorm documentation around deletions for more information

Migration tests

In most cases, it shouldn't be necessary to create a test for a migration. However, if the migration is manipulating records and poses a significant risk of completely borking up important data, a test should be written.

Tests are difficult to write for migrations and are likely to fail one day long after the migration has already run in production. After a migration is run in production, it is safe to delete the test from the integration test suite.

The test helper has a couple helpful functions for testing migrations. You can use h.CleanDB() to completely wipe the database clean, then h.MigrateTo(<migration_id>) to migrate to a specific migration ID. You should then be able to create whatever records in the database you need to test against and finally run h.MigrateDB() to run your created migration and all subsequent migrations.

Documentation

Index

• Variables
• func ArgsToOrderBy(orderByArgs []string, disallowedFields map[string]string) (orderBy []string, err *errors.ServiceError)
• func FieldNameWalk(n tsl.Node, disallowedFields map[string]string) (newNode tsl.Node, err *errors.ServiceError)
• func GetTableName(g2 *gorm.DB) string
• func MarkForRollback(ctx context.Context, err error)
• func Migrate(g2 *gorm.DB) error
• func MigrateTo(sessionFactory SessionFactory, migrationID string)
• func NewContext(ctx context.Context, connection SessionFactory) (context.Context, error)
• func RegisterAdvisoryLockMetrics()
• func ResetAdvisoryLockMetricsCollectors()
• func Resolve(ctx context.Context)
• func TransactionMiddleware(next http.Handler, connection SessionFactory) http.Handler
• func UnregisterAdvisoryLockMetrics()
• func UpdateAdvisoryLockCountMetric(lockType LockType, status string)
• func UpdateAdvisoryLockDurationMetric(lockType LockType, status string, startTime time.Time)
• type AdvisoryLock
• type AdvisoryLockFactory
  • func NewAdvisoryLockFactory(connection SessionFactory) *AdvisoryLockFactory
  • func (f *AdvisoryLockFactory) NewAdvisoryLock(ctx context.Context, id string, lockType LockType) (string, error)
  • func (f *AdvisoryLockFactory) NewNonBlockingLock(ctx context.Context, id string, lockType LockType) (string, bool, error)
  • func (f *AdvisoryLockFactory) Unlock(ctx context.Context, uuid string)
• type LockFactory
• type LockType
• type MetricsCollector
• type SessionFactory

Variables

var MetricsLabels = []string{
	metricsTypeLabel,
	metricsStatusLabel,
}

MetricsLabels - Array of labels added to metrics:

var MetricsNames = []string{
	countMetric,
	durationMetric,
}

MetricsNames - Array of Names of the metrics:

Functions

func ArgsToOrderBy

func ArgsToOrderBy(
	orderByArgs []string,
	disallowedFields map[string]string) (orderBy []string, err *errors.ServiceError)

ArgsToOrderBy returns cleaned orderBy list.

func FieldNameWalk

func FieldNameWalk(
	n tsl.Node,
	disallowedFields map[string]string) (newNode tsl.Node, err *errors.ServiceError)

FieldNameWalk walks on the filter tree and check/replace the search fields names: a. the the field name is valid. b. replace the field name with the SQL column name.

func GetTableName

func GetTableName(g2 *gorm.DB) string

func MarkForRollback

func MarkForRollback(ctx context.Context, err error)

MarkForRollback flags the transaction stored in the context for rollback and logs whatever error caused the rollback

func Migrate

func Migrate(g2 *gorm.DB) error

func MigrateTo

func MigrateTo(sessionFactory SessionFactory, migrationID string)

MigrateTo a specific migration will not seed the database, seeds are up to date with the latest schema based on the most recent migration This should be for testing purposes mainly

func NewContext

func NewContext(ctx context.Context, connection SessionFactory) (context.Context, error)

NewContext returns a new context with transaction stored in it. Upon error, the original context is still returned along with an error

func RegisterAdvisoryLockMetrics

func RegisterAdvisoryLockMetrics()

Register the metrics:

func ResetAdvisoryLockMetricsCollectors

func ResetAdvisoryLockMetricsCollectors()

ResetMetricCollectors resets all collectors

func Resolve

func Resolve(ctx context.Context)

Resolve resolves the current transaction according to the rollback flag.

func TransactionMiddleware

func TransactionMiddleware(next http.Handler, connection SessionFactory) http.Handler

TransactionMiddleware creates a new HTTP middleware that begins a database transaction and stores it in the request context.

func UnregisterAdvisoryLockMetrics

func UnregisterAdvisoryLockMetrics()

Unregister the metrics:

func UpdateAdvisoryLockCountMetric

func UpdateAdvisoryLockCountMetric(lockType LockType, status string)

func UpdateAdvisoryLockDurationMetric

func UpdateAdvisoryLockDurationMetric(lockType LockType, status string, startTime time.Time)

Types

type AdvisoryLock

type AdvisoryLock struct {
	// contains filtered or unexported fields
}

AdvisoryLock represents a postgres advisory lock

begin                                       # start a Tx
select pg_advisory_xact_lock(id, lockType)  # obtain the lock (blocking)
end                                         # end the Tx and release the lock

UUID is a way to own the lock. Only the very first service call that owns the lock will have the correct UUID. This is necessary to allow functions to call other service functions as part of the same lock (id, lockType).

type AdvisoryLockFactory

type AdvisoryLockFactory struct {
	// contains filtered or unexported fields
}

func NewAdvisoryLockFactory

func NewAdvisoryLockFactory(connection SessionFactory) *AdvisoryLockFactory

NewAdvisoryLockFactory returns a new factory with AdvisoryLock stored in it.

func (*AdvisoryLockFactory) NewAdvisoryLock

func (f *AdvisoryLockFactory) NewAdvisoryLock(ctx context.Context, id string, lockType LockType) (string, error)

func (*AdvisoryLockFactory) NewNonBlockingLock

func (f *AdvisoryLockFactory) NewNonBlockingLock(ctx context.Context, id string, lockType LockType) (string, bool, error)

func (*AdvisoryLockFactory) Unlock

func (f *AdvisoryLockFactory) Unlock(ctx context.Context, uuid string)

Unlock searches current locks and unlocks the one matching its owner id.

type LockFactory

type LockFactory interface {
	// NewAdvisoryLock constructs a new AdvisoryLock that is a blocking PostgreSQL advisory lock
	// defined by (id, lockType) and returns a UUID as this AdvisoryLock owner id.
	NewAdvisoryLock(ctx context.Context, id string, lockType LockType) (string, error)
	// NewNonBlockingLock constructs a new nonblocking AdvisoryLock defined by (id, lockType),
	// returns a UUID and a boolean on whether the lock is acquired.
	NewNonBlockingLock(ctx context.Context, id string, lockType LockType) (string, bool, error)
	// Unlock unlocks one AdvisoryLock by its owner id.
	Unlock(ctx context.Context, uuid string)
}

LockFactory provides the blocking/unblocking locks based on PostgreSQL advisory lock.

type LockType

type LockType string

const (
	Migrations LockType = "migrations"
	Dinosaurs  LockType = "dinosaurs"
	Events     LockType = "events"
)

type MetricsCollector

type MetricsCollector interface {
}

type SessionFactory

type SessionFactory interface {
	Init(*config.DatabaseConfig)
	DirectDB() *sql.DB
	New(ctx context.Context) *gorm.DB
	CheckConnection() error
	Close() error
	ResetDB()
	NewListener(ctx context.Context, channel string, callback func(id string))
}