dmlgen

Documentation

Overview

Package dmlgen generates Go code and protocol buffer files from database tables.

TODO check for https://github.com/improbable-eng/ts-protoc-gen and https://github.com/improbable-eng/grpc-web

Join Preload Preload loads the association data in a separate query, Join Preload will loads association data using inner join It can handle null association

Index

• func GenerateJSON(fileNameOrDirectory, buildTags string, g *bootstrap.Generator) error
• func RunProtoc(protoFilesPath string, po *ProtocOptions) error
• type FeatureToggle
  • func (f FeatureToggle) String() string
• type ForeignKeyOptions
• type Generator
  • func NewGenerator(packageImportPath string, opts ...Option) (*Generator, error)
  • func (g *Generator) GenerateGo(wMain, wTest io.Writer) error
  • func (g *Generator) GenerateSerializer(wMain, wTest io.Writer) error
• type Option
  • func WithBuildTags(lines ...string) (opt Option)
  • func WithCustomCode(marker, code string) (opt Option)
  • func WithCustomCodeFunc(marker string, fn func(g *Generator, t *Table, w io.Writer)) (opt Option)
  • func WithForeignKeyRelationships(ctx context.Context, db dml.Querier, o ForeignKeyOptions) (opt Option)
  • func WithProtobuf(sc *SerializerConfig) (opt Option)
  • func WithTable(tableName string, columns ddl.Columns, options ...string) (opt Option)
  • func WithTableConfig(tableName string, opt *TableConfig) (o Option)
  • func WithTableConfigDefault(opt TableConfig) (o Option)
  • func WithTablesFromDB(ctx context.Context, db *dml.ConnPool, tables ...string) (opt Option)
• type ProtocOptions
• type SerializerConfig
• type Table
  • func (t *Table) CollectionName() string
  • func (t *Table) EntityName() string
  • func (t *Table) EntityNameLCFirst() string
  • func (t *Table) GoCamelMaybePrivate(fieldName string) string
  • func (t *Table) IsFieldPrivate(dbColumnName string) bool
  • func (t *Table) IsFieldPublic(dbColumnName string) bool
• type TableConfig
• type TypeDef

Functions

func GenerateJSON

func GenerateJSON(fileNameOrDirectory, buildTags string, g *bootstrap.Generator) error

GenerateJSON creates the easysjon code for a specific file or a whole directory. argument `g` can be nil. deprecated as there are stdlib compatible packages which have the same speed without code gen.

func RunProtoc

func RunProtoc(protoFilesPath string, po *ProtocOptions) error

RunProtoc searches all *.proto files in the given path and calls protoc to generate the Go source code.

Types

type FeatureToggle

type FeatureToggle uint64

FeatureToggle allows certain generated code blocks to be switched off or on.

const (
	FeatureCollectionAppend FeatureToggle = 1 << iota
	FeatureCollectionClear
	FeatureCollectionCut
	FeatureCollectionDelete
	FeatureCollectionEach
	FeatureCollectionFilter
	FeatureCollectionInsert
	FeatureCollectionStruct // creates the struct type
	FeatureCollectionSwap
	FeatureCollectionUniqueGetters
	FeatureCollectionUniquifiedGetters
	FeatureCollectionValidate
	FeatureDB
	FeatureDBAssignLastInsertID
	FeatureDBDelete
	FeatureDBInsert
	FeatureDBMapColumns
	FeatureDBSelect
	FeatureDBTracing // opentelemetry tracing
	FeatureDBUpdate
	FeatureDBUpsert
	FeatureDBTableColumnNames
	FeatureEntityCopy
	FeatureEntityEmpty
	FeatureEntityGetSetPrivateFields
	FeatureEntityIsSet
	FeatureEntityRelationships
	FeatureEntityStruct // creates the struct type
	FeatureEntityValidate
	FeatureEntityWriteTo
)

List of available features

func (FeatureToggle) String

func (f FeatureToggle) String() string

type ForeignKeyOptions

type ForeignKeyOptions struct {
	IncludeRelationShips []string
	ExcludeRelationships []string
}

ForeignKeyOptions applies to WithForeignKeyRelationships

type Generator

type Generator struct {
	Package                     string // Name of the package
	PackageImportPath           string // Name of the package
	PackageSerializer           string // Name of the package, if empty uses field Package
	PackageSerializerImportPath string // Name of the package, if empty uses field PackageImportPath
	BuildTags                   []string
	ImportPaths                 []string
	ImportPathsTesting          []string
	// Tables uses the table name as map key and the table description as value.
	Tables map[string]*Table
	// Serializer defines the de/serializing method to use. Either
	// `empty`=default (none) or proto=protocol buffers or fbs=flatbuffers. JSON
	// marshaling is not affected with this config. Only one serializer can be
	// defined because some Go types depends on the types of serializer. E.g.
	// protobuf cannot use int16/int8 whereas in flatbuffers they are available.
	// Using a common denominator for every serializer like an int32 can cause
	// data loss when communicating with the database table which has different
	// (smaller) column types than an int32.
	Serializer string
	// SerializerHeaderOptions defines custom headers to use in the .proto or .fbs file.
	// For proto, sane defaults are available.
	SerializerHeaderOptions []string
	// TestSQLDumpGlobPath contains the path and glob pattern to load a SQL dump
	// containing the table schemas to run integration tests. If empty no dumps
	// get loaded and the test program assumes that the tables already exists.
	TestSQLDumpGlobPath string
	// contains filtered or unexported fields
}

Generator can generated Go source for for database tables once correctly configured.

func NewGenerator

func NewGenerator(packageImportPath string, opts ...Option) (*Generator, error)

NewGenerator creates a new instance of the SQL table code generator. The order of the applied options does not matter as they are getting sorted internally.

func (*Generator) GenerateGo

func (g *Generator) GenerateGo(wMain, wTest io.Writer) error

GenerateGo writes the Go source code into `w` and the test code into wTest.

func (*Generator) GenerateSerializer

func (g *Generator) GenerateSerializer(wMain, wTest io.Writer) error

GenerateSerializer writes the protocol buffer specifications into `w` and its test sources into wTest, if there are any tests.

type Option

type Option struct {
	// contains filtered or unexported fields
}

Option represents a sortable option for the NewGenerator function. Each option function can be applied in a mixed order.

func WithBuildTags

func WithBuildTags(lines ...string) (opt Option)

WithBuildTags adds your build tags to the file header. Each argument represents a build tag line.

func WithCustomCode

func WithCustomCode(marker, code string) (opt Option)

WithCustomCode inserts at the marker position your custom Go code. For available markers search these .go files for the map access of field *Generator.customCode. An example got written in TestGenerate_Tables_Protobuf_Json. If the marker does not exists or has a typo, no error gets reported and no code gets written.

func WithCustomCodeFunc

func WithCustomCodeFunc(marker string, fn func(g *Generator, t *Table, w io.Writer)) (opt Option)

WithCustomCodeFunc same as WithCustomCode but allows access to meta data. The func fn takes as first argument the main Generator where access to package global configuration is possible. If the scope of the marker is within a table, then argument t gets set, otherwise it is nil. The output must be written to w.

func WithForeignKeyRelationships

func WithForeignKeyRelationships(ctx context.Context, db dml.Querier, o ForeignKeyOptions) (opt Option)

WithForeignKeyRelationships analyzes the foreign keys which points to a table and adds them as a struct field name. For example: customer_address_entity.parent_id is a foreign key to customer_entity.entity_id hence the generated struct CustomerEntity has a new field which gets named CustomerAddressEntityCollection, pointing to type CustomerAddressEntityCollection. includeRelationShips and excludeRelationships must be a balanced slice in the notation of "table1.column1","table2.column2". For example:

"customer_entity.store_id", "store.store_id" which means that the

struct CustomerEntity won't have a field to the Store struct (1:1 relationship) (in case of excluding). The reverse case can also be added

"store.store_id", "customer_entity.store_id" which means that the

Store struct won't or will have a field pointing to the CustomerEntityCollection (1:M relationship). Setting includeRelationShips to nil will include all relationships. Wildcards are supported.

func WithProtobuf

func WithProtobuf(sc *SerializerConfig) (opt Option)

WithProtobuf enables protocol buffers as a serialization method. Argument headerOptions is optional. Heads up: This function also sets the internal Serializer field and all types will get adjusted to the minimum protobuf types. E.g. uint32 minimum instead of uint8/uint16. So if the Generator gets created multiple times to separate the creation of code, the WithProtobuf function must get set for Generator objects. See package store.

func WithTable

func WithTable(tableName string, columns ddl.Columns, options ...string) (opt Option)

WithTable sets a table and its columns. Allows to overwrite a table fetched with function WithTablesFromDB. Argument `options` can be set to "overwrite" and/or "view". Each option is its own slice entry.

func WithTableConfig

func WithTableConfig(tableName string, opt *TableConfig) (o Option)

WithTableConfig applies options to an existing table, identified by the table name used as map key. Options are custom struct or different encoders. Returns a not-found error if the table cannot be found in the `Generator` map.

func WithTableConfigDefault

func WithTableConfigDefault(opt TableConfig) (o Option)

WithTableConfigDefault sets for all tables the same configuration but can be overwritten on a per table basis with function WithTableConfig. Current behaviour: Does not apply the default config to all tables but only to those which have set an empty `WithTableConfig("table_name",&dmlgen.TableConfig{})`.

func WithTablesFromDB

func WithTablesFromDB(ctx context.Context, db *dml.ConnPool, tables ...string) (opt Option)

WithTablesFromDB queries the information_schema table and loads the column definition of the provided `tables` slice. It adds the tables to the `Generator` map. Once added a call to WithTableConfig can add additional configurations.

type ProtocOptions

type ProtocOptions struct {
	WorkingDirectory string
	ProtoGen         string // default go, options: gofast, gogo, gogofast, gogofaster and other installed proto generators

	ProtoPath []string // where to find other *.proto files; if empty the usual defaults apply
	GoOutPath string   // where to write the generated proto files; default "."
	GoOpt     []string // each entry generates a "--go_opt" argument
	// GoSourceRelative if the paths=source_relative flag is specified, the output
	// file is placed in the same relative directory as the input file. For
	// example, an input file protos/buzz.proto results in an output file at
	// protos/buzz.pb.go.
	GoSourceRelative bool

	GRPC               bool
	GRPCOpt            []string // each entry generates a "--go-grpc_opt" argument
	GRPCGatewayOutMap  []string // GRPC must be enabled in the above field
	GRPCGatewayOutPath string   // GRPC must be enabled in the above field

	SwaggerOutPath string
	CustomArgs     []string
}

ProtocOptions allows to modify the protoc CLI command.

type SerializerConfig

type SerializerConfig struct {
	PackageImportPath string
	AdditionalHeaders []string
}

SerializerConfig applies various optional settings to WithProtobuf and/or WithFlatbuffers and/or WithTypeScript.

type Table

type Table struct {
	Package              string // Name of the package
	Table                *ddl.Table
	Comment              string // Comment above the struct type declaration
	HasAutoIncrement     uint8  // 0=nil,1=false (has NO auto increment),2=true has auto increment
	HasEasyJSONMarshaler bool
	HasSerializer        bool // writes the .proto file if true
	// contains filtered or unexported fields
}

table writes one database table into Go source code.

func (*Table) CollectionName

func (t *Table) CollectionName() string

func (*Table) EntityName

func (t *Table) EntityName() string

func (*Table) EntityNameLCFirst

func (t *Table) EntityNameLCFirst() string

func (*Table) GoCamelMaybePrivate

func (t *Table) GoCamelMaybePrivate(fieldName string) string

func (*Table) IsFieldPrivate

func (t *Table) IsFieldPrivate(dbColumnName string) bool

func (*Table) IsFieldPublic

func (t *Table) IsFieldPublic(dbColumnName string) bool

type TableConfig

type TableConfig struct {
	// Encoders add method receivers for, each struct, compatible with the
	// interface declarations in the various encoding packages. Supported
	// encoder names are: json resp. easyjson.
	Encoders []string
	// StructTags enables struct tags proactively for the whole struct. Allowed
	// values are: bson, db, env, json, protobuf, toml, yaml and xml. For bson,
	// json, yaml and xml the omitempty attribute has been set. If you need a
	// different struct tag for a specifiv column you must set the option
	// CustomStructTags.
	StructTags []string
	// CustomStructTags allows to specify custom struct tags for a specific
	// column. The slice must be balanced, means index i sets to the column name
	// and index i+1 to the desired struct tag.
	// 		[]string{"column_a",`json: ",omitempty"`,"column_b","`xml:,omitempty`"}
	// In case when foreign keys should be referenced:
	// 		[]string{"FieldNameX",`faker: "-"`,"FieldNameY","`xml:field_name_y,omitempty`"}
	// TODO CustomStructTags should be appended to StructTags
	CustomStructTags []string // balanced slice
	// AppendCustomStructTags []string // balanced slice TODO maybe this additionally
	// Comment adds custom comments to each struct type. Useful when relying on
	// 3rd party JSON marshaler code generators like easyjson or ffjson. If
	// comment spans over multiple lines each line will be checked if it starts
	// with the comment identifier (//). If not, the identifier will be
	// prepended.
	Comment string
	// ColumnAliases specifies different names used for a column. For example
	// customer_entity.entity_id can also be sales_order.customer_id, hence a
	// Foreign Key. The alias would be just: entity_id:[]string{"customer_id"}.
	ColumnAliases map[string][]string // key=column name value a list of aliases
	// UniquifiedColumns specifies columns which are non primary/unique key one
	// but should have a dedicated function to extract their unique primitive
	// values as a slice. Not allowed are text, blob and binary.
	UniquifiedColumns []string
	// PrivateFields list struct field names which should be private to avoid
	// accidentally leaking through encoders. Appropriate getter/setter methods
	// get generated.
	PrivateFields []string
	// FeaturesInclude if set includes only those features, otherwise
	// everything. Some features can only be included on Default level and not
	// on a per table level.
	FeaturesInclude FeatureToggle
	// FeaturesExclude excludes features while field FeaturesInclude is empty.
	// Some features can only be excluded on Default level and not on a per
	// table level.
	FeaturesExclude FeatureToggle
	// FieldMapFn can map a dbIdentifier (database identifier) of the current
	// table to a new name. dbIdentifier is in most cases the column name and in
	// cases of foreign keys, it is the table name.
	FieldMapFn func(dbIdentifier string) (newName string)
	// contains filtered or unexported fields
}

TableConfig used in conjunction with WithTableConfig and WithTableConfigDefault to apply different configurations for a generated struct and its struct collection.

type TypeDef

type TypeDef struct {
	GoUNull    string // unsigned null
	GoUNotNull string // unsigned not null
	GoNull     string // A go type which can be null/nil
	GoNotNull  string // A usually native Go primitive type

	SerializerUNull    string
	SerializerUNotNull string
	SerializerNull     string
	SerializerNotNull  string
}

TypeDef used in variable `mysqlTypeToGo` to map a MySQL/MariaDB type to its appropriate Go and serializer type. Those types are getting printed in the generated files.