README
¶
typesystem
Package typesystem enables Data Transfer to be compatible with different type system versions and provides definitions of a Data Transfer type system.
See CHANGELOG.md for details on what each fallback does.
A version of a type system is a set of rules of conversion of specific source database types to Data Transfer types, as well as rule of their conversion to specific target database types. Type system is thus only about heterogenous transfers.
This package consists of the typesystem and document methods which define typesystem rules (the mentioned rules of conversion) and the fallback facility methods.
Fallbacks enable users of older versions of transfers (with typesystem version, a property of the transfer, set to some value different from the "current" one) to convert types differently than the newly created transfers do. This feature enables to introduce some non-backwards-compatible changes to Data Transfer without any extra settings.
Use AddFallbackSource and AddFallbackTarget methods to add fallbacks. For example, a fallback for PostgreSQL source is added as follows:
// In file `pkg/dataagent/pg/fallback_date_as_string.go`
func init() {
typesystem.AddFallbackSource(typesystem.Fallback{
To: 1,
ProviderType: abstract.ProviderTypePostgreSQL,
Function: func(ci *abstract.ChangeItem) (*abstract.ChangeItem, error) {
if !ci.IsRowEvent() {
switch ci.Kind {
case abstract.InitTableLoad, abstract.DoneTableLoad:
// perform fallback
default:
return ci, nil
}
}
for i := 0; i < len(ci.TableSchema); i++ {
switch ci.TableSchema[i].DataType {
case schema.TypeDate.String(), schema.TypeDatetime.String(), schema.TypeTimestamp.String():
ci.TableSchema[i].DataType = schema.TypeString.String()
default:
// do nothing
}
}
return ci, nil
},
})
}
// ...
// In `typesystem` package:
const LatestVersion int = 2 // previously was `1`
See function descriptions and actual use examples for extra details.
Each type system change (update) requires an increment of the global "latest" type system version constant, located inside this package.
Documentation
¶
Index ¶
- Constants
- Variables
- func AddFallbackSourceFactory(factory FallbackFactory)
- func AddFallbackTargetFactory(factory FallbackFactory)
- func BuildSourceDoc(typ abstract.ProviderType, title string) string
- func BuildTargetDoc(typ abstract.ProviderType, title string) string
- func Doc(typ abstract.ProviderType, title string) string
- func SourceRules(provider abstract.ProviderType, rules map[schema.Type][]string)
- func SupportedTypes() []schema.Type
- func TargetRule(provider abstract.ProviderType, rules map[schema.Type]string)
- type Fallback
- type FallbackFactory
- type Rule
Constants ¶
const ( RestPlaceholder = "REST..." NotSupportedPlaceholder = "N/A" )
const LatestVersion int = 9
LatestVersion is the current (most recent) version of the typesystem. Increment this when adding a new fallback.
At any moment, fallbacks can only be "to" any version preceding the latest.
Zero value is reserved and MUST NOT be used.
When incrementing this value, DO ADD a link to the function(s) implementing this fallback to CHANGELOG.md in the current directory
const NewTransfersVersion int = 9
NewTransfersVersion is the version of the typesystem set for new transfers. It must be less or equal to the LatestVersion.
To upgrade typesystem version, the following process should be applied: 1. LatestVersion is increased & fallbacks are introduced in the first PR. NewTransfersVersion stays the same! 2. Controlplane and dataplane are deployed and dataplane now contains the fallbacks for a new version. 3. The second PR increases NewTransfersVersion. When a controlplane with this change is deployed, dataplanes already have the required fallbacks.
Variables ¶
var FallbackDoesNotApplyErr = xerrors.NewSentinel("this fallback does not apply")
var SourceFallbackFactories = make([]FallbackFactory, 0)
var TargetFallbackFactories = make([]FallbackFactory, 0)
Functions ¶
func AddFallbackSourceFactory ¶
func AddFallbackSourceFactory(factory FallbackFactory)
AddFallbackSourceFactory registers a fallbacks for a source of some type
This method is expected to be called in the `init()` function of a module which introduces a fallback when some additional behaviour is expected
func AddFallbackTargetFactory ¶
func AddFallbackTargetFactory(factory FallbackFactory)
AddFallbackTargetFactory registers a fallbacks for a target of some type
This method is expected to be called in the `init()` function of a module which introduces a fallback when some additional behaviour is expected
func BuildSourceDoc ¶
func BuildSourceDoc(typ abstract.ProviderType, title string) string
func BuildTargetDoc ¶
func BuildTargetDoc(typ abstract.ProviderType, title string) string
func SourceRules ¶
func SourceRules(provider abstract.ProviderType, rules map[schema.Type][]string)
func SupportedTypes ¶
func TargetRule ¶
func TargetRule(provider abstract.ProviderType, rules map[schema.Type]string)
Types ¶
type Fallback ¶
type Fallback struct {
// To is the target typesystem version of this fallback
To int
// ProviderType limits the providers to which this fallback applies
ProviderType abstract.ProviderType
// Function defines the transformation. Input is an item of any kind.
//
// If a fallback does not apply, this method MUST return an error containing FallbackDoesNotApplyErr
Function func(ci *abstract.ChangeItem) (*abstract.ChangeItem, error)
}
Fallback defines a transformation from a newer version of typesystem to an older one.
type FallbackFactory ¶
type FallbackFactory func() Fallback
type Rule ¶
type Rule struct {
// `Target` stores the mapping of `schema.Type`-s to provider-specific types for each `Target` provider.
// example:
// {schema.TypeString: "TEXT"}
Target map[schema.Type]string
// `Source` stores the conversion rules from the provider-specific type to the `schema.Type` for each `Source` provider.
// example:
// {"TINYTEXT": schema.TypeString}
Source map[string]schema.Type
}
func RuleFor ¶
func RuleFor(provider abstract.ProviderType) Rule
Source Files
Directories