mig

Documentation

Overview

Package mig provides tools to version, record and migrate a project schema, and also provides rules to migrate the project data.

Project dom nodes are assigned sequential version numbers, that are automatically determined based on the node's content and its last known version. The version starts at one for new nodes and is incremented if the old and new definition differ. The schema and project versions work in a similar manner, only on based on the hashes of their children. This way we can avoid explicitly declaring versions.

A project manifest contains all details needed to determine version changes. It contains the version information for the project and all its nodes. The version information includes a sha256 hash. For models this hash is calculated based on the default string representation, which means that any change to the model results in a new hash. For schemas, the hash is based on the schema name and all its model hashes, and for projects similarly the name and schema hashes. Effectively only changes to models are automatically increment the version. Users should be able to force version updates for any dom node.

All datasets like backups or databases should store at least the project versions. Programs involved with data migration have the full project history to calculate any new versions, other programs only need the project manifest.

The schema history and manifest are managed by the daql command and are written to files. Changes need to be explicitly recorded into the project history and manifest. Data migration rules should also be recorded for each version as part of the history. Simple migration rules can be expressed as xelf expressions and interpreted by the daql command. Complex migration rules call any command, usually a simple go script that migrates one or more model changes for a dataset. The daql command should be able to generate simple rules and migration templates.

Index

• Constants
• Variables
• func DiscoverProject(path string) (string, error)
• func ResolveProject(path string) (*dom.Project, error)
• func WriteDataset(path string, d Dataset) error
• func WriteIter(it Iter, w io.Writer) error
• func WriteZip(z *zip.Writer, d Dataset) error
• type Dataset
  • func ReadDataset(path string) (Dataset, error)
  • func ReadZip(r *zip.Reader) (Dataset, error)
• type History
  • func ReadHistory(path string) (_ History, err error)
• type IODataset
• type Iter
  • func NewFileIter(f io.ReadCloser, gzipped bool) (Iter, error)
  • func OpenFileIter(path string) (Iter, error)
• type Manifest
  • func ReadManifest(r io.Reader) (mf Manifest, err error)
  • func (mf Manifest) Diff(old Manifest) map[string]byte
  • func (mf Manifest) First() (v Version)
  • func (mf Manifest) Get(name string) (Version, bool)
  • func (mf Manifest) Len() int
  • func (mf Manifest) Less(i, j int) bool
  • func (mf Manifest) Set(v Version) Manifest
  • func (mf Manifest) Sort() Manifest
  • func (mf Manifest) Swap(i, j int)
  • func (mf Manifest) Update(pr *dom.Project) (Manifest, error)
  • func (mf Manifest) WriteTo(w io.Writer) (nn int64, err error)
• type Record
  • func ReadProject(path string) (res Record, err error)
  • func (r *Record) Version() Version
• type Version
  • func ReadVersion(r io.Reader) (v Version, err error)
  • func (v Version) WriteTo(w io.Writer) (int64, error)
• type Versioner
  • func NewVersioner(mf Manifest) Versioner

Constants

const ProjectFileName = "project.daql"

Variables

var ErrNoChanges = cor.StrError("no changes")

var ErrNoHistory = cor.StrError("no history")

Functions

func DiscoverProject

func DiscoverProject(path string) (string, error)

DiscoverProject looks for a project file based on path and returns a cleaned path.

If path points to a file it check whether the file has a project file name. If path points to a directory, we try to look for a project file in the current and then in all its parents.

func ResolveProject

func ResolveProject(path string) (*dom.Project, error)

ResolveProject returns the project schema read from the project file path or an error.

This function will resolve schema includes. For now it only supports includes relative to the project directory are supported.

We want to be able to include schema definitions from imported go packages. This however means, that resolution of those path is specific to the host language. We need to call the go tool to determine the file path to include from. Package versioning is probably enough, but we may need a way to specify include version and reference a project, to lookup the schema history. And we later need the included schema history for migration rules and scripts, so ...

We should require project definitions even for library schemas, to reuse the same versioning and migration machinery. We then need to import the go package containing the included project file and declare the use of the external project in the project declaration, and can then include schemas from those external projects.

func WriteDataset

func WriteDataset(path string, d Dataset) error

WriteDataset writes a dataset to path or returns an error. If the path ends in '.zip' a zip file is written, otherwise the dataset is written as individual gzipped file to the directory at path.

func WriteIter

func WriteIter(it Iter, w io.Writer) error

func WriteZip

func WriteZip(z *zip.Writer, d Dataset) error

WriteZip writes a dataset to the given zip file or returns an error. It is the caller's responsibility to close the zip writer.

Types

type Dataset

type Dataset interface {
	// Version returns the project version of this dataset.
	Version() Version
	Keys() []string
	Iter(key string) (Iter, error)
	Close() error
}

func ReadDataset

func ReadDataset(path string) (Dataset, error)

ReadDataset returns a dataset with the manifest and data streams found at path or an error.

Path must either point to directory or a zip file containing individual files for the project version and data steams. The version file must be named 'version.json' and the individual data streams use the qualified model name with the '.json' extension for the format and optional a '.gz' extension, for gzipped files. The returned data streams are first read when iterated. We only allow the json format, because the files are usually machine written and read, and to make working with backups easier in other language.

func ReadZip

func ReadZip(r *zip.Reader) (Dataset, error)

ReadZip returns a dataset read from the given zip reader as described in ReadDataset or an error. It is the caller's responsibility to close a zip read closer or any underlying reader.

type History

type History interface {
	Path() string
	Curr() Record
	Last() Manifest
	Versions() []Version
	Manifest(v int64) (Manifest, error)
	Record(v int64) (Record, error)
	Commit(string) error
}

History provides project records.

func ReadHistory

func ReadHistory(path string) (_ History, err error)

ReadHistory returns the prepared project history based on a project path or an error.

The history folder path defaults to '$project/hist', but can changed in the project definition. The folder contains a link to, or copy of the last recorded manifest and record folders each containing a project definition and manifest file, encoded as optionally gzipped JSON streams. The record folders can also contain migration rules and scripts, to migration data to that version. The history folder acts as staging area for migrations for unrecorded project changes. Record folders can have any name starting with a 'v', the daql tool uses the padded version number and an optional record note that only acts as memory aid. The actual record version should always be read from the included manifest file.

type IODataset

type IODataset interface {
	Dataset
	Open(key string) (io.ReadCloser, error)
}

type Iter

type Iter interface {
	Scan() (lit.Lit, error)
	Close() error
}

Iter is an iterator for a possibly large sequence of object literal data.

This abstraction allows us to choose an appropriate implementation for any situation, without being forced to load all the data into memory at once.

func NewFileIter

func NewFileIter(f io.ReadCloser, gzipped bool) (Iter, error)

func OpenFileIter

func OpenFileIter(path string) (Iter, error)

type Manifest

type Manifest []Version

Manifest is set of versions sorted by name, usually for all nodes of one project. Manifest usually contain exactly one project version as first element, due to the project name qualifier prefix.

func ReadManifest

func ReadManifest(r io.Reader) (mf Manifest, err error)

ReadManifest returns a manifest read from r or an error. Manifests are read as JSON stream of version objects.

func (Manifest) Diff

func (mf Manifest) Diff(old Manifest) map[string]byte

Diff returns a map of all changed version names to a byte indicating the kind of change. The byte is the '+' for addition, '-' for deletion or '*' for modification.

func (Manifest) First

func (mf Manifest) First() (v Version)

First returns the first version of the manifest or a zero version if empty. The first version is the project version, by the nature of the sort order, unless the manifest is unnaturally contains none or many project versions.

func (Manifest) Get

func (mf Manifest) Get(name string) (Version, bool)

Get returns the version for the qualified name or false if no version was found.

func (Manifest) Len

func (mf Manifest) Len() int

func (Manifest) Less

func (mf Manifest) Less(i, j int) bool

func (Manifest) Set

func (mf Manifest) Set(v Version) Manifest

Set inserts a version into the manifest and returns the result.

func (Manifest) Sort

func (mf Manifest) Sort() Manifest

func (Manifest) Swap

func (mf Manifest) Swap(i, j int)

func (Manifest) Update

func (mf Manifest) Update(pr *dom.Project) (Manifest, error)

Update sets the node versions in project and returns the updated manifest or an error.

func (Manifest) WriteTo

func (mf Manifest) WriteTo(w io.Writer) (nn int64, err error)

WriteTo writes the manifest to w and returns the written bytes or an error. Manifests are written as JSON stream of version objects.

type Record

type Record struct {
	Path string // record path relative to history folder
	*dom.Project
	Manifest
}

Record consists of a project definition and its manifest at one point in time. A record's path can be used to look up migration rules and scripts.

func ReadProject

func ReadProject(path string) (res Record, err error)

ReadProject reads the current project's unrecorded definition and manifest or an error.

The returned record represent the current malleable project state, and may contain unrecorded changes and preliminary versions, not representing the eventually recorded version definition.

func (*Record) Version

func (r *Record) Version() Version

type Version

type Version struct {
	Name string    `json:"name"`
	Vers int64     `json:"vers"`
	Date time.Time `json:"date,omitempty"`
	Hash string    `json:"hash"`
}

Version contains essential details for a node to derive a new version number.

The name is the node's qualified name, and date is an optional recording time. Vers is a positive integer for known versions or zero if unknown. The hash is a lowercase hex string of an sha256 hash of the node's qualified name and its contents. For models the default string representation is used as content, for schemas each model hash and for projects each schema hash.

func ReadVersion

func ReadVersion(r io.Reader) (v Version, err error)

ReadVersion returns a version read from r or and error.

func (Version) WriteTo

func (v Version) WriteTo(w io.Writer) (int64, error)

WriteTo writes the version to w and returns the written bytes or an error.

type Versioner

type Versioner interface {
	// Manifest returns a fresh manifest with updated versions.
	Manifest() Manifest
	// Version sets and returns the node version details or an error.
	Version(dom.Node) (Version, error)
}

Versioner sets and returns node version details, usually based on the last recorded manifest.

func NewVersioner

func NewVersioner(mf Manifest) Versioner

NewVersioner returns a new versioner based on the given manifest.