mig

README

mig

Package mig provides tools to version, record and migrate a project schema, and also provides rules to migrate the project data.

Project dom nodes are assigned a semantic Version, that are automatically determined based on the node's content and its last known version. The version starts at v0.0.0 for new nodes and increments either the minor or the patch version if the old and new definition differ.

The distinction between minor and patch version is made by considering the impact of the changes. As a rule of thumb we want the minor version to increment if the change requires a db schema migration. User should be able to explicitly bump major and minor versions.

A project Manifest contains the version information for the project and all of its nodes. Each version includes two sha256 hashes for the node contents, seperated by minor and patch version relevant data.

All datasets like backups or databases should store project versions. Programs involved with data migration have the full project history to calculate any new versions, other programs only need the project manifest.

The schema history and manifest are managed by the daql command and are written to files. Changes need to be explicitly recorded into the project history and manifest. Data migration rules should also be recorded for each version as part of the history. Simple migration rules can be expressed as xelf expressions and interpreted by the daql command. Complex migration rules call any command, usually a simple go script that migrates one or more model changes for a dataset. The daql command should be able to generate simple rules and migration templates.

Documentation

Index

• Variables
• func RawSchema() string
• func WriteDataset(path string, d Dataset) error
• func WriteStream(it Stream, w io.Writer) error
• func WriteZip(z *zip.Writer, d Dataset) error
• type Dataset
  • func ReadDataset(path string) (Dataset, error)
  • func ReadZip(r *zip.Reader) (Dataset, error)
• type History
  • func ReadHistory(reg *lit.Regs, path string) (_ History, err error)
• type IODataset
• type Manifest
  • func ReadManifest(r io.Reader) (mf Manifest, err error)
  • func (mf Manifest) Diff(old Manifest) map[string]byte
  • func (mf Manifest) First() (v *Version)
  • func (mf Manifest) Get(name string) (Version, bool)
  • func (mf Manifest) Len() int
  • func (mf Manifest) Less(i, j int) bool
  • func (mf Manifest) Set(v Version) Manifest
  • func (mf Manifest) Sort() Manifest
  • func (mf Manifest) Swap(i, j int)
  • func (mf Manifest) Update(pr *dom.Project) (Manifest, error)
  • func (mf Manifest) WriteTo(w io.Writer) (nn int64, err error)
• type Migration
• type Record
  • func ReadRecord(reg *lit.Regs, path string) (res Record, err error)
  • func (r *Record) Version() Version
• type Stream
  • func NewFileStream(f io.ReadCloser, name string, gzipped bool) (Stream, error)
  • func NewLitStream(l lit.Idxr) Stream
  • func OpenFileStream(path string) (Stream, error)
• type Vers
  • func ParseVers(str string) (v Vers, err error)
  • func (v Vers) String() string
• type Version
  • func ReadVersion(r io.Reader) (v Version, err error)
  • func (v Version) WriteTo(w io.Writer) (int64, error)
• type Versioner
  • func NewVersioner(mf Manifest) Versioner
  • func (mv Versioner) Manifest(pr *dom.Project) (Manifest, error)
  • func (mv Versioner) Version(n dom.Node) (res Version, err error)

Variables

var ErrNoChanges = fmt.Errorf("no changes")

var ErrNoHistory = fmt.Errorf("no history")

var NoVers = Vers{}

Functions

func RawSchema

func RawSchema() string

func WriteDataset

func WriteDataset(path string, d Dataset) error

WriteDataset writes a dataset to path or returns an error. If the path ends in '.zip' a zip file is written, otherwise the dataset is written as individual gzipped file to the directory at path.

func WriteStream

func WriteStream(it Stream, w io.Writer) error

WriteStream writes stream to writer w or returns an error.

func WriteZip

func WriteZip(z *zip.Writer, d Dataset) error

WriteZip writes a dataset to the given zip file or returns an error. It is the caller's responsibility to close the zip writer.

Types

type Dataset

type Dataset interface {
	// Vers returns the project version of this dataset.
	Vers() *Version
	// Keys returns all stream keys that are part of this data set.
	Keys() []string
	// Stream returns the stream with key or an error.
	Stream(key string) (Stream, error)
	// Close closes the dataset.
	Close() error
}

Dataset is an abstraction over project data, be it a database, a test fixture or backup archive.

func ReadDataset

func ReadDataset(path string) (Dataset, error)

ReadDataset returns a dataset with the manifest and data streams found at path or an error.

Path must either point to directory or a zip file containing individual files for the project version and data steams. The version file must be named 'version.json' and the individual data streams use the qualified model name with the '.json' extension for the format and optional a '.gz' extension, for gzipped files. The returned data streams are first read when iterated. We only allow the json format, because the files are usually machine written and read, and to make working with backups easier in other language.

func ReadZip

func ReadZip(r *zip.Reader) (Dataset, error)

ReadZip returns a dataset read from the given zip reader as described in ReadDataset or an error. It is the caller's responsibility to close a zip read closer or any underlying reader.

type History

type History interface {
	Path() string
	Curr() Record
	Last() Manifest
	Versions() []Version
	Manifest(vers string) (Manifest, error)
	Record(vers string) (Record, error)
	Commit(slug string) error
}

History provides project records.

func ReadHistory

func ReadHistory(reg *lit.Regs, path string) (_ History, err error)

ReadHistory returns the prepared project history based on a project path or an error.

The history folder path defaults to '$project/hist', but can changed in the project definition. The folder contains a link to, or copy of the last recorded manifest and record folders each containing a project definition and manifest file, encoded as optionally gzipped JSON streams. The record folders can also contain migration rules and scripts, to migration data to that version. The history folder acts as staging area for migrations for unrecorded project changes. Record folders can have any name starting with a 'v', the daql tool uses the padded version number and an optional record note that only acts as memory aid. The actual record version should always be read from the included manifest file.

type IODataset

type IODataset interface {
	Dataset
	Open(key string) (io.ReadCloser, error)
}

IODataset is a dataset that is based on file streams and provides access to those files.

type Manifest

type Manifest []Version

Manifest is set of versions sorted by name. A Manifest usually contain exactly one project version as first element, due to the project name qualifier prefix.

func ReadManifest

func ReadManifest(r io.Reader) (mf Manifest, err error)

ReadManifest returns a manifest read from r or an error. Manifests are read as JSON stream of version objects.

func (Manifest) Diff

func (mf Manifest) Diff(old Manifest) map[string]byte

Diff returns a map of all changed version names to a byte indicating the kind of change. The byte is the '+' for addition, '-' for deletion or '*' for modification.

func (Manifest) First

func (mf Manifest) First() (v *Version)

First returns the first version of the manifest or nil. The first version is the project version, by the nature of the sort order, unless the manifest is unnaturally contains none or many project versions.

func (Manifest) Get

func (mf Manifest) Get(name string) (Version, bool)

Get returns the version for the qualified name or false if no version was found.

func (Manifest) Len

func (mf Manifest) Len() int

func (Manifest) Less

func (mf Manifest) Less(i, j int) bool

func (Manifest) Set

func (mf Manifest) Set(v Version) Manifest

Set inserts a version into the manifest and returns the result.

func (Manifest) Sort

func (mf Manifest) Sort() Manifest

func (Manifest) Swap

func (mf Manifest) Swap(i, j int)

func (Manifest) Update

func (mf Manifest) Update(pr *dom.Project) (Manifest, error)

Update sets the node versions in project and returns the updated manifest or an error.

func (Manifest) WriteTo

func (mf Manifest) WriteTo(w io.Writer) (nn int64, err error)

WriteTo writes the manifest to w and returns the written bytes or an error. Manifests are written as JSON stream of version objects.

type Migration

type Migration struct {
	ID   int64     `json:"id"`
	Vers string    `json:"vers"`
	Date time.Time `json:"date"`
	Note string    `json:"note,omitempty"`
}

Migration contains migration information of a data source.

type Record

type Record struct {
	Path string `json:"path"` // record path relative to history folder
	*dom.Project
	Manifest `json:"manifest"`
}

Record consists of a project definition and its manifest at one point in time. A record's path can be used to look up migration rules and scripts.

func ReadRecord

func ReadRecord(reg *lit.Regs, path string) (res Record, err error)

ReadProject reads the current project's unrecorded definition and manifest or an error.

The returned record represent the current malleable project state, and may contain unrecorded changes and preliminary versions, not representing the eventually recorded version definition.

func (*Record) Version

func (r *Record) Version() Version

type Stream

type Stream interface {
	// Scan returns the next value or an error, normal iteration must end with an eof error.
	Scan() (lit.Val, error)
	// Close closes the stream, possibly making it inoperable.
	Close() error
}

Stream is an iterator for a possibly large sequence of object literal data.

This abstraction allows us to choose an appropriate implementation for any situation, without being forced to load all the data into memory all at once.

func NewFileStream

func NewFileStream(f io.ReadCloser, name string, gzipped bool) (Stream, error)

NewFileStream creates and returns a new stream for the named reader f or an error.

func NewLitStream

func NewLitStream(l lit.Idxr) Stream

NewLitStream creates and returns a new stream for the idxr literal or an error.

func OpenFileStream

func OpenFileStream(path string) (Stream, error)

OpenFileStream opens a file and returns a new stream or an error.

type Vers

type Vers struct {
	Major, Minor, Patch int
}

Vers represents a parsed version string with int fields for major, minor and patch versions.

func ParseVers

func ParseVers(str string) (v Vers, err error)

ParseVers parses str and returns the result or an error.

func (Vers) String

func (v Vers) String() string

type Version

type Version struct {
	Name  string    `json:"name"`
	Vers  string    `json:"vers"`
	Date  time.Time `json:"date,omitempty"`
	Minor string    `json:"minor"`
	Patch string    `json:"patch"`
}

Version contains essential details for a node to derive a new version number.

The name is the node's qualified name, and date is an optional recording time. Vers is version string v1.23.4 for known versions or empty. The minor and patch are a lowercase hex sha256 hash strings of the node's details and its children.

func ReadVersion

func ReadVersion(r io.Reader) (v Version, err error)

ReadVersion returns a version read from r or and error.

func (Version) WriteTo

func (v Version) WriteTo(w io.Writer) (int64, error)

WriteTo writes the version to w and returns the written bytes or an error.

type Versioner

type Versioner map[string]*entry

Versioner sets and returns node version details, usually based on the last recorded manifest.

func NewVersioner

func NewVersioner(mf Manifest) Versioner

NewVersioner returns a new versioner based on the given manifest.

func (Versioner) Manifest

func (mv Versioner) Manifest(pr *dom.Project) (Manifest, error)

Manifest returns a update manifest for Project.

func (Versioner) Version

func (mv Versioner) Version(n dom.Node) (res Version, err error)

Version sets and returns the node version details or an error.