column

README

Column Mapping

introduction

column mapping is a library to provide a simple and unified way to mapping columns of table:

• add prefix for one char/varchar/text column

• add suffix for one char/varchar/text column

• partition ID (used for sharding schema/table, would partition these tables with a custom ID), only for id(int64)

column mapping rule

we define a rule Rule to show how to map column

type Rule struct {
	PatternSchema    string   `yaml:"schema-pattern" json:"schema-pattern" toml:"schema-pattern"`
	PatternTable     string   `yaml:"table-pattern" json:"table-pattern" toml:"table-pattern"`
	SourceColumn     string   `yaml:"source-column" json:"source-column" toml:"source-column"` // modify, add refer column, ignore
	TargetColumn     string   `yaml:"target-column" json:"target-column" toml:"target-column"` // add column, modify
	Expression       Expr     `yaml:"expression" json:"expression" toml:"expression"`
	Arguments        []string `yaml:"arguments" json:"arguments" toml:"arguments"`
	CreateTableQuery string   `yaml:"create-table-query" json:"create-table-query" toml:"create-table-query"`
}

now we support following expressions

add prefix, with arguments[prefix]

add suffix, with arguments[suffix]

partition id, with arguments [instance_id, prefix of schema, prefix of table, separator]
[1:1 bit][2:9 bits][3:10 bits][4:44 bits] int64  (using default bits length)
- 1: useless, no reason
- 2: schema ID (schema suffix)
- 3: table ID (table suffix)
- 4: origin ID (>= 0, <= 17592186044415)
And
- schema = arguments[1] + arguments[3] + schema suffix    or    arguments[1]
- table  = arguments[2] + arguments[3] + table suffix     or    arguments[2]
The separator argument defaults to an empty string if omitted.

notice

• only support above poor expressions now
• column mapping doesn't change column type and table structure now

Documentation

Index

• Variables
• func SetPartitionRule(instanceIDSize, schemaIDSize, tableIDSize int)
• type Expr
• type Mapping
  • func NewMapping(caseSensitive bool, rules []*Rule) (*Mapping, error)
  • func (m *Mapping) AddRule(rule *Rule) error
  • func (m *Mapping) HandleDDL(schema, table string, columns []string, statement string) (string, []int, error)
  • func (m *Mapping) HandleRowValue(schema, table string, columns []string, vals []any) ([]any, []int, error)
  • func (m *Mapping) RemoveRule(rule *Rule) error
  • func (m *Mapping) UpdateRule(rule *Rule) error
• type Rule
  • func (r *Rule) Adjust()
  • func (r *Rule) ToLower()
  • func (r *Rule) Valid() error

Variables

var Exprs = map[Expr]func(*mappingInfo, []any) ([]any, error){
	AddPrefix: addPrefix,
	AddSuffix: addSuffix,

	PartitionID: partitionID,
}

Exprs is some built-in expression for column mapping only supports some poor expressions now, we would unify tableInfo later and support more

Functions

func SetPartitionRule

func SetPartitionRule(instanceIDSize, schemaIDSize, tableIDSize int)

SetPartitionRule sets bit size of schema ID and table ID

Types

type Expr

type Expr string

Expr indicates how to handle column mapping

const (
	AddPrefix   Expr = "add prefix"
	AddSuffix   Expr = "add suffix"
	PartitionID Expr = "partition id"
)

poor Expr

type Mapping

type Mapping struct {
	selector.Selector
	// contains filtered or unexported fields
}

Mapping maps column to something by rules

func NewMapping

func NewMapping(caseSensitive bool, rules []*Rule) (*Mapping, error)

NewMapping returns a column mapping

func (*Mapping) AddRule

func (m *Mapping) AddRule(rule *Rule) error

AddRule adds a rule into mapping

func (*Mapping) HandleDDL

func (m *Mapping) HandleDDL(schema, table string, columns []string, statement string) (string, []int, error)

HandleDDL handles ddl

func (*Mapping) HandleRowValue

func (m *Mapping) HandleRowValue(schema, table string, columns []string, vals []any) ([]any, []int, error)

HandleRowValue handles row value

func (*Mapping) RemoveRule

func (m *Mapping) RemoveRule(rule *Rule) error

RemoveRule removes a rule from mapping

func (*Mapping) UpdateRule

func (m *Mapping) UpdateRule(rule *Rule) error

UpdateRule updates mapping rule

type Rule

type Rule struct {
	PatternSchema    string   `yaml:"schema-pattern" json:"schema-pattern" toml:"schema-pattern"`
	PatternTable     string   `yaml:"table-pattern" json:"table-pattern" toml:"table-pattern"`
	SourceColumn     string   `yaml:"source-column" json:"source-column" toml:"source-column"` // modify, add refer column, ignore
	TargetColumn     string   `yaml:"target-column" json:"target-column" toml:"target-column"` // add column, modify
	Expression       Expr     `yaml:"expression" json:"expression" toml:"expression"`
	Arguments        []string `yaml:"arguments" json:"arguments" toml:"arguments"`
	CreateTableQuery string   `yaml:"create-table-query" json:"create-table-query" toml:"create-table-query"`
}

Rule is a rule to map column TODO: we will do it later, if we need to implement a real column mapping, we need table structure of source and target system

func (*Rule) Adjust

func (r *Rule) Adjust()

Adjust normalizes the rule into an easier-to-process form, e.g. filling in optional arguments with the default values.

func (*Rule) ToLower

func (r *Rule) ToLower()

ToLower covert schema/table parttern to lower case

func (*Rule) Valid

func (r *Rule) Valid() error

Valid checks validity of rule. add prefix/suffix: it should have target column and one argument partition id: it should have 3 to 4 arguments