debefix

package module
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 17, 2024 License: MIT Imports: 18 Imported by: 6

README

debefix - Database seeding and fixtures

GoDoc

debefix is a Go library (and a cli in the future) to seed database data and/or create fixtures for DB tests.

Tables can reference each other using string ids (called "refid"), generated fields (like database auto increment or generated UUID) are supported and can be resolved and used by other table's references.

Dependencies between tables can be detected automatically by reference ids, or manually. This is used to generate a dependency graph and output the insert statements in the correct order.

Using the yaml tag !dbfexpr it is possible to define expressions on field values.

Tables with rows can be declared at the top-level on inside a parent row using a special !dbfdeps tag. In this case, values from the parent row can be used, using the parent:<fieldname> expression.

Install

go get github.com/rrgmc/debefix

Goals

  • For developer seeding or test fixtures, not for inserting a huge amount of records.
  • YAML files are to be manually edited, so they must be easy to read and write.
  • Input can be files or memory, to allow creating simple tests.

Field value expressions

  • !dbfexpr "refid:<table>:<refid>:<fieldname>": reference a refid field value in a table. This id is declared using a config: !dbfconfig {"refid": <refid>} special tagged field in the row.
  • !dbfexpr "parent<:level>:<fieldname>": reference a field in the parent table. This can only be used inside a !dbfdeps block. Level is the number of parent levels, if not specified the default value is 1.
  • !dbfexpr "generated<:type>": indicates that this is a generated field that must be supplied at resolve time, and can later be used by other references once resolved. If type is specified, the value is parsed/cast to this type after db retrieval. The default types are 'int', 'float', 'str' and 'timestamp', using the YAML formats.

Special fields

All field with tags starting with !dbf are special fields. The name of the field is ignored.

  • config: !dbfconfig {"refid": "", "tags": ["", ""]}
  • deps: !dbfdeps {<tableID>: {...table config...}}

Generating SQL

SQL can be generated using github.com/rrgmc/debefix/db/sql/<dbtype>.

import (
    "sql"

    dbsql "github.com/rrgmc/debefix/db/sql"
    "github.com/rrgmc/debefix/db/sql/postgres"
)

func main() {
    db, err := sql.Open("postgres", "dsn://postgres")
    if err != nil {
        panic(err)
    }

    // will send an INSERT SQL for each row to the db, taking table dependency in account for the correct order. 
    resolvedValues, err := postgres.GenerateDirectory(context.Background(), "/x/y", dbsql.NewSQLQueryInterface(db))
    if err != nil {
        panic(err)
    }
    
    // resolvedValues will contain all data that was inserted, including any generated fields like autoincrement.
}

Generating Non-SQL

The import github.com/rrgmc/debefix/db contains a ResolverFunc that is not directly tied to SQL, it can be used to insert data in any database that has the concepts of "tables" with a list of field/values.

As inner maps/arrays are supported by YAML, data with more complex structure should work without any problems.

Sample input

The configuration can be in a single or multiple files, the file itself doesn't matter. The file names/directories are sorted alphabetically, so the order can be deterministic.

The same table can also be present in multiple files, given that the config section is equal (or only set in one of them).

Only files that have the extension .dbf.yaml are loaded by the directory loader.

# all_data.dbf.yaml
tags:
  config:
    table_name: "public.tag" # database table name. If not set, will use the table id (tags) as the table name.
  rows:
    - tag_id: !dbfexpr "generated:int" # means that this will be generated, for example as a database autoincrement
      name: "Go"
      created_at: !!timestamp 2023-01-01T12:30:12Z
      updated_at: !!timestamp 2023-01-01T12:30:12Z
      config:
        !dbfconfig
        refid: "go" # refid to be targeted by '!dbfexpr "refid:tags:go:tag_id"'
    - tag_id: !dbfexpr "generated:int"
      name: "JavaScript"
      created_at: !!timestamp 2023-01-01T12:30:12Z
      updated_at: !!timestamp 2023-01-01T12:30:12Z
      config:
        !dbfconfig
        refid: "javascript"
    - tag_id: !dbfexpr "generated:int"
      name: "C++"
      created_at: !!timestamp 2023-01-01T12:30:12Z
      updated_at: !!timestamp 2023-01-01T12:30:12Z
      config:
        !dbfconfig
        refid: "cpp"
users:
  config:
    table_name: "public.user"
  rows:
    - user_id: 1
      name: "John Doe"
      email: "john@example.com"
      created_at: !!timestamp 2023-01-01T12:30:12Z
      updated_at: !!timestamp 2023-01-01T12:30:12Z
      config:
        !dbfconfig
        refid: "johndoe" # refid to be targeted by '!dbfexpr "refid:users:johndoe:user_id"'
    - user_id: 2
      name: "Jane Doe"
      email: "jane@example.com"
      created_at: !!timestamp 2023-01-04T12:30:12Z
      updated_at: !!timestamp 2023-01-04T12:30:12Z
      config:
        !dbfconfig
        refid: "janedoe"
posts:
  config:
    table_name: "public.post"
  rows:
    - post_id: 1
      title: "Post 1"
      text: "This is the text of the first post"
      user_id: !dbfexpr "refid:users:johndoe:user_id"
      created_at: !!timestamp 2023-01-01T12:30:12Z
      updated_at: !!timestamp 2023-01-01T12:30:12Z
      deps:
        !dbfdeps
        posts_tags: # declaring tables in !dbfdeps is exactly the same as declaring top-level, but allows using "parent" expression to get parent info
          rows:
            - post_id: !dbfexpr "parent:post_id"
              tag_id: !dbfexpr "refid:tags:go:tag_id"
      config:
        !dbfconfig
        refid: "post_1"
    - post_id: 2
      parent_post_id: !dbfexpr "refid:posts:post_1:post_id" # order matters, so self-referential fields must be set in order
      title: "Post 2"
      text: "This is the text of the seco d post"
      user_id: !dbfexpr "refid:users:johndoe:user_id"
      created_at: !!timestamp 2023-01-02T12:30:12Z
      updated_at: !!timestamp 2023-01-02T12:30:12Z
      deps:
        !dbfdeps
        posts_tags:
          rows:
            - post_id: !dbfexpr "parent:post_id"
              tag_id: !dbfexpr "refid:tags:javascript:tag_id" # tag_id is generated so the value will be resolved before being set here 
        comments:
          rows:
            - comment_id: 3
              post_id: !dbfexpr "parent:post_id"
              user_id: !dbfexpr "refid:users:janedoe:user_id"
              text: "I liked this post!"
posts_tags:
  config:
    table_name: "public.post_tag"
comments:
  config:
    depends:
      - posts # add a manual dependency if there is no refid linking the tables
  rows:
    - comment_id: 1
      post_id: 1
      user_id: !dbfexpr "refid:users:janedoe:user_id"
      text: "Good post!"
      created_at: !!timestamp 2023-01-01T12:31:12Z
      updated_at: !!timestamp 2023-01-01T12:31:12Z
    - comment_id: 2
      post_id: 1
      user_id: !dbfexpr "refid:users:johndoe:user_id"
      text: "Thanks!"
      created_at: !!timestamp 2023-01-01T12:35:12Z
      updated_at: !!timestamp 2023-01-01T12:35:12Z

Samples

Extra

Sub-packages
  • filter: simple methods to find and extract data from parsed or resolved data, and doing transformations to objects, like entities. Can be used to get test data from fixtures.
  • value: value parsers both for "Load" and "Resolve", like UUID.
External

License

MIT

Author

Rangel Reale (rangelreale@gmail.com)

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	ValueError           = errors.New("value error")
	ResolveValueError    = errors.New("resolve value error")
	ResolveError         = errors.New("resolve error")
	ResolveCallbackError = errors.New("resolve callback error")
	RowNotFound          = errors.New("row not found in data")
)

Functions

func DefaultDirectoryTagFunc 

func DefaultDirectoryTagFunc(dirs []string) []string

DefaultDirectoryTagFunc joins directories using a dot (.).

func DefaultParseResolvedValue added in v0.13.0 

func DefaultParseResolvedValue(typ string, value any) (bool, any, error)

func ResolveCheck 

func ResolveCheck(data *Data, options ...ResolveOption) error

ResolveCheck checks if all dependencies between rows are resolvable.

func ResolveCheckCallback 

func ResolveCheckCallback(ctx ResolveContext, fields map[string]any) error

ResolveCheckCallback is the callback for the ResolveCheck function.

func StripNumberPunctuationPrefixDirectoryTagFunc added in v0.10.2 

func StripNumberPunctuationPrefixDirectoryTagFunc(dirs []string) []string

StripNumberPunctuationPrefixDirectoryTagFunc strips number and punctuation prefixes from each dir (like "01-") and joins directories using a dot (.).

Types

type Data 

type Data struct {
	Tables map[string]*Table
}

Data stores the entire collection of parsed Table information.

func Generate 

func Generate(fileProvider FileProvider, resolver ResolveCallback, options ...GenerateOption) (*Data, error)

Generate loads files and calls a resolver callback to resolve the values, and returns the resolved data. It is a combination of Load and Resolve.

func GenerateDirectory 

func GenerateDirectory(rootDir string, resolver ResolveCallback, options ...GenerateOption) (*Data, error)

GenerateDirectory is a version of Generate that loads from a directory name.

func GenerateFS 

func GenerateFS(fs fs.FS, resolver ResolveCallback, options ...GenerateOption) (*Data, error)

GenerateFS is a version of Generate that loads from a fs.FS.

func Load 

func Load(fileProvider FileProvider, options ...LoadOption) (*Data, error)

Load loads the files from the fileProvider and returns the list of loaded tables. Rows dependencies are not resolved, use ResolveCheck to check for them.

func MergeData added in v0.9.2 

func MergeData(list ...*Data) (*Data, error)

MergeData merge a list of Data objects into a new instance. The data is deep-copied, the source Data instances are never modified in any way.

func Resolve 

func Resolve(data *Data, f ResolveCallback, options ...ResolveOption) (*Data, error)

Resolve calls a callback for each table row, taking table dependency in account, and returns the resolved data.

func (*Data) Clone added in v0.12.0 

func (d *Data) Clone() (*Data, error)

Clone creates a deep-copy of the source. The source Data is never modified.

func (*Data) ExtractRows 

func (d *Data) ExtractRows(f func(table *Table, row Row) (bool, error)) (*Data, error)

ExtractRows extract rows matched by the callback.

func (*Data) ExtractRowsNamed 

func (d *Data) ExtractRowsNamed(f func(table *Table, row Row) (bool, string, error)) (map[string]Row, error)

ExtractRowsNamed extract rows matched by the callback into a named map.

func (*Data) Merge added in v0.9.2 

func (d *Data) Merge(source *Data) error

Merge merges source into the current instance. A deep copy is done to ensure source is never modified.

func (*Data) WalkRows 

func (d *Data) WalkRows(f func(table *Table, row Row) bool)

WalkRows calls a callback for each row in each table.

func (*Data) WalkTableData 

func (d *Data) WalkTableData(tableID string, f func(row Row) (bool, any, error)) (any, error)

WalkTableData searches for rows in tables.

type FSFileProviderOption 

type FSFileProviderOption interface {
	GenerateOption
	// contains filtered or unexported methods
}

func WithDirectoryAsTag 

func WithDirectoryAsTag() FSFileProviderOption

WithDirectoryAsTag creates tags for each directory. Inner directories will be concatenated by a dot (.).

func WithDirectoryIncludeFunc 

func WithDirectoryIncludeFunc(include func(path string, entry os.DirEntry) bool) FSFileProviderOption

WithDirectoryIncludeFunc sets a callback to allow choosing files that will be read. Check entry os.DirEntry.IsDir to detect files or directories.

func WithDirectoryTagFunc 

func WithDirectoryTagFunc(tagFunc func(dirs []string) []string) FSFileProviderOption

WithDirectoryTagFunc allows returning custom tags for each directory entry.

type FileInfo 

type FileInfo struct {
	Name string
	File io.Reader
	Tags []string
}

type FileProvider 

type FileProvider interface {
	Load(FileProviderCallback) error
}

FileProvider provides files and tags to Load. The order matters, so it should be deterministic.

func NewDirectoryFileProvider 

func NewDirectoryFileProvider(rootDir string, options ...FSFileProviderOption) FileProvider

NewDirectoryFileProvider creates a FileProvider that list files from a directory, sorted by name. Only files with the ".dbf.yaml" extension are returned. Returned file names are relative to the rootDir.

func NewFSFileProvider 

func NewFSFileProvider(fs fs.FS, options ...FSFileProviderOption) FileProvider

NewFSFileProvider creates a FileProvider that list files from a fs.FS, sorted by name. Only files with the ".dbf.yaml" extension are returned.

func NewStringFileProvider added in v0.12.0 

func NewStringFileProvider(files []string, options ...StringFileProviderOption) FileProvider

NewStringFileProvider creates a FileProvider that simulates a file for each string field, in the array order.

type FileProviderCallback 

type FileProviderCallback func(info FileInfo) error

type GenerateOption 

type GenerateOption interface {
	// contains filtered or unexported methods
}

func WithGenerateResolveCheck 

func WithGenerateResolveCheck(check bool) GenerateOption

WithGenerateResolveCheck sets whether to check the data using ResolveCheck. Default is false.

type IsGenerateOption 

type IsGenerateOption struct {
}

type LoadOption 

type LoadOption interface {
	GenerateOption
	// contains filtered or unexported methods
}

func WithLoadInitialData added in v0.12.0 

func WithLoadInitialData(initialData *Data) LoadOption

WithLoadInitialData sets the initial data. Can be used to merge more items into an existing data. The instance WILL be modified, call Data.Clone if you want to load into a copy.

func WithLoadProgress 

func WithLoadProgress(progress func(filename string)) LoadOption

WithLoadProgress sets a callback to report load progress.

func WithLoadRowsSetIgnoreTags added in v0.12.1 

func WithLoadRowsSetIgnoreTags(rowsSetIgnoreTags bool) LoadOption

WithLoadRowsSetIgnoreTags sets "IgnoreTags" on all rows loaded. This is mainly used in tests to be sure the rows will be included.

func WithLoadValueParser added in v0.15.0 

func WithLoadValueParser(parser ValueParser) LoadOption

WithLoadValueParser adds a YAML tag value parser.

type ParseError 

type ParseError struct {
	ErrorMessage string
	Path         string
	Position     *TokenPosition
}

func NewParseError 

func NewParseError(msg string, path string, position *TokenPosition) ParseError

func (ParseError) Error 

func (e ParseError) Error() string

type ResolveCallback 

type ResolveCallback func(ctx ResolveContext, fields map[string]any) error

type ResolveContext 

type ResolveContext interface {
	TableID() string
	TableName() string
	ResolveField(fieldName string, value any)
}

ResolveContext is the context used to resolve values.

type ResolveGenerate 

type ResolveGenerate struct {
	Type string
}

ResolveGenerate is a ResolveValue that indicates a value will be generated and must be returned.

type ResolveIncludeTagsFunc 

type ResolveIncludeTagsFunc func(tableID string, rowTags []string) bool

ResolveIncludeTagsFunc is the function signature for WithResolveTagsFunc

func DefaultResolveIncludeTagFunc 

func DefaultResolveIncludeTagFunc(tags []string) ResolveIncludeTagsFunc

DefaultResolveIncludeTagFunc returns a ResolveIncludeTagsFunc check checks if at least one tags is contained.

type ResolveOption 

type ResolveOption interface {
	GenerateOption
	// contains filtered or unexported methods
}

func WithResolveProgress 

func WithResolveProgress(progress func(tableID, tableName string)) ResolveOption

WithResolveProgress sets a function to receive resolve progress.

func WithResolveRowProgress 

func WithResolveRowProgress(rowProgress func(tableID, tableName string, current, amount int, isIncluded bool)) ResolveOption

WithResolveRowProgress sets a function to receive resolve row progress.

func WithResolveTags 

func WithResolveTags(tags []string) ResolveOption

WithResolveTags set Resolve to only resolve rows that contains at least one of these tags. If nil or 0 length, no row filtering is performed.

func WithResolveTagsFunc 

func WithResolveTagsFunc(f ResolveIncludeTagsFunc) ResolveOption

WithResolveTagsFunc sets a row tag filter function.

func WithResolvedValueParser added in v0.13.0 

func WithResolvedValueParser(f ResolvedValueParser) ResolveOption

type ResolveValue 

type ResolveValue interface {
	// contains filtered or unexported methods
}

ResolveValue indicates a field value that must be resolved.

type ResolvedValueParser added in v0.13.0 

type ResolvedValueParser interface {
	ParseResolvedValue(typ string, value any) (bool, any, error)
}

ResolvedValueParser parses resolved value types, like generated fields.

type ResolvedValueParserFunc added in v0.15.0 

type ResolvedValueParserFunc func(typ string, value any) (bool, any, error)

ResolvedValueParserFunc is a func wrapper for ResolvedValueParser.

func (ResolvedValueParserFunc) ParseResolvedValue added in v0.15.0 

func (p ResolvedValueParserFunc) ParseResolvedValue(typ string, value any) (bool, any, error)

type Row 

type Row struct {
	InternalID uuid.UUID
	Config     RowConfig
	Fields     map[string]any
}

func (Row) Clone added in v0.9.2 

func (r Row) Clone() Row

Clone does a deep copy of the row, to ensure source is never modified.

type RowConfig 

type RowConfig struct {
	RefID      string   `yaml:"refid"`
	Tags       []string `yaml:"tags"`
	IgnoreTags bool     `yaml:"ignoreTags"` // if true, always include row ignoring any tag filter.
}

func (RowConfig) Clone added in v0.9.2 

func (r RowConfig) Clone() RowConfig

type Rows 

type Rows []Row

type StringFileProviderOption added in v0.12.0 

type StringFileProviderOption func(*stringFileProvider)

func WithStringFileProviderTags added in v0.12.0 

func WithStringFileProviderTags(tags [][]string) StringFileProviderOption

WithStringFileProviderTags sets tags using the same array indexes as the files parameter.

type Table 

type Table struct {
	ID     string
	Config TableConfig
	Rows   Rows
}

func (*Table) AppendDeps 

func (t *Table) AppendDeps(deps ...string)

AppendDeps adds table dependencies checking duplicates.

func (*Table) Merge added in v0.9.2 

func (t *Table) Merge(source *Table) error

Merge merges source into d. A deep copy is done to ensure source is never modified.

func (*Table) WalkData 

func (t *Table) WalkData(f func(row Row) (bool, any, error)) (any, error)

WalkData searches for rows in the table.

type TableConfig 

type TableConfig struct {
	TableName     string         `yaml:"table_name"`
	Depends       []string       `yaml:"depends"`
	DefaultValues map[string]any `yaml:"default_values"`
}

func (*TableConfig) Merge 

func (c *TableConfig) Merge(other *TableConfig) error

Merge checks if merging is allowed before merging.

type TokenPosition 

type TokenPosition = token.Position

type Value 

type Value interface {
	// contains filtered or unexported methods
}

Value indicates a field value requires processing.

type ValueGenerated 

type ValueGenerated struct {
	Type string
}

ValueGenerated is a Value that will be generated in the future (possibly by a database).

type ValueInternalID 

type ValueInternalID struct {
	TableID    string
	InternalID uuid.UUID
	FieldName  string
}

ValueInternalID is a Value that references a field value in a table using the internal ID.

func (ValueInternalID) TableDepends 

func (v ValueInternalID) TableDepends() string

TableDepends indicates a dependency on another table.

type ValueParser added in v0.15.0 

type ValueParser interface {
	ParseValue(tag *ast.TagNode) (bool, any, error)
}

ValueParser is used to parse YAML tag values.

type ValueParserFunc added in v0.15.0 

type ValueParserFunc func(tag *ast.TagNode) (bool, any, error)

ValueParserFunc is a func adapter for ValueParser

func (ValueParserFunc) ParseValue added in v0.15.0 

func (p ValueParserFunc) ParseValue(tag *ast.TagNode) (bool, any, error)

type ValueRefID 

type ValueRefID struct {
	TableID   string
	RefID     string
	FieldName string
}

ValueRefID is a Value that references a field value in a table using the RefID (string ID).

func (ValueRefID) TableDepends 

func (v ValueRefID) TableDepends() string

TableDepends indicates a dependency on another table.

Source Files

View all Source files

Directories

Path Synopsis
db
sql
sql/postgres
filter module
internal
external/depgraph
value module

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.