processor

package
v0.0.8 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 8, 2024 License: Apache-2.0 Imports: 31 Imported by: 0

Documentation

Overview

Copyright © 2020 Marvin

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright © 2020 Marvin

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright © 2020 Marvin

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright © 2020 Marvin

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright © 2020 Marvin

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright © 2020 Marvin

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright © 2020 Marvin

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright © 2020 Marvin

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright © 2020 Marvin

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright © 2020 Marvin

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright © 2020 Marvin

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright © 2020 Marvin

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright © 2020 Marvin

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright © 2020 Marvin

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright © 2020 Marvin

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright © 2020 Marvin

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright © 2020 Marvin

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright © 2020 Marvin

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed p. in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright © 2020 Marvin

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright © 2020 Marvin

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright © 2020 Marvin

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright © 2020 Marvin

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright © 2020 Marvin

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright © 2020 Marvin

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Cmp added in v0.0.7 

func Cmp(src map[string]int64, dest map[string]int64) (map[string]int64, map[string]int64)

Cmp used for the src and dest store key-value pair , and key data row and value data row counts

func DivideDatabaseTableColumnStatisticsBucket added in v0.0.7 

func DivideDatabaseTableColumnStatisticsBucket(database database.IDatabase, schemaName, tableName string, cons *structure.HighestBucket, chunkRange *structure.Range, divideCountCnt int) ([]*structure.Range, error)

DivideDatabaseTableColumnStatisticsBucket splits a chunk to multiple chunks by random Notice: If the `count <= 1`, it will skip splitting and return `chunk` as a slice directly.

func ExtractDatabaseTableStatisticsValuesFromBuckets added in v0.0.7 

func ExtractDatabaseTableStatisticsValuesFromBuckets(divideDbType, valueString string, columnNames []string) ([]string, error)

ExtractDatabaseTableStatisticsValuesFromBuckets analyze upperBound or lowerBound to string for each column. upperBound and lowerBound are looks like '(123, abc)' for multiple fields, or '123' for one field.

func GenMYSQLCompatibleDatabaseDeleteStmtSQL added in v0.0.7 

func GenMYSQLCompatibleDatabaseDeleteStmtSQL(schemaName, tableName, sqlHint string, columnDetailSlice []string, columnDataString []string, columnDataCounts int) string

func GenMYSQLCompatibleDatabaseInsertStmtSQL added in v0.0.7 

func GenMYSQLCompatibleDatabaseInsertStmtSQL(schemaName, tableName, sqlHint string, columnDetailSlice []string, columnDataString string, columnDataCounts int, safeMode bool) string

func GenMYSQLCompatibleDatabasePrepareStmt added in v0.0.7 

func GenMYSQLCompatibleDatabasePrepareStmt(
	schemaName, tableName string, sqlHint, columnDetailSlice string, insertBatchSize int, safeMode bool) string

func GenOracleCompatibleDatabaseDeleteStmtSQL added in v0.0.7 

func GenOracleCompatibleDatabaseDeleteStmtSQL(schemaName, tableName, sqlHint string, columnDetailSlice []string, columnDataString []string, columnDataCounts int) string

func GenOracleCompatibleDatabaseInsertStmtSQL added in v0.0.7 

func GenOracleCompatibleDatabaseInsertStmtSQL(schemaName, tableName, sqlHint string, columnDetailSlice []string, columnDataString string, columnDataCounts int, safeMode bool) string

func GetDownstreamDatabaseTableColumnDatatype added in v0.0.7 

func GetDownstreamDatabaseTableColumnDatatype(schemaNameT, tableNameT string, databaseT database.IDatabase, originColumnNameSli []string, columnNameRouteRule map[string]string) ([]string, error)

func InspectOracleMigrateTask added in v0.0.7 

func InspectOracleMigrateTask(taskName, taskFlow, taskMode string, databaseS database.IDatabase, connectDBCharsetS, connectDBCharsetT string) (string, string, error)

func OptimizerMYSQLCompatibleDataMigrateColumnS added in v0.0.7 

func OptimizerMYSQLCompatibleDataMigrateColumnS(columnName, datatype, datetimePrecision string) (string, error)

func OptimizerMYSQLCompatibleMigrateOracleDataCompareColumnST added in v0.0.7 

func OptimizerMYSQLCompatibleMigrateOracleDataCompareColumnST(columnNameS, datatypeS string, datetimePrecisionS, dataLengthS int64, dataPrecisionS, dataScaleS, dbCharsetSDest, columnNameT, dbCharsetTFrom, dbCharsetTDest string) (string, string, error)

func OptimizerOracleDataMigrateColumnS added in v0.0.7 

func OptimizerOracleDataMigrateColumnS(columnName, datatype, dataScale string) (string, error)

func OptimizerOracleMigrateMYSQLCompatibleDataCompareColumnST added in v0.0.7 

func OptimizerOracleMigrateMYSQLCompatibleDataCompareColumnST(columnNameS, datatypeS string, dataLengthS int64, dataPrecisionS, dataScaleS, dbCharsetSFrom, dbCharsetSDest, columnNameT, dbCharsetTDest string) (string, string, error)

func ProcessDownstreamDatabaseTableColumnStatisticsBucket added in v0.0.7 

func ProcessDownstreamDatabaseTableColumnStatisticsBucket(dbTypeT, dbCharsetT string, bs []*structure.Range, r *structure.Rule) ([]*structure.Range, error)

func ProcessUpstreamDatabaseTableColumnStatisticsBucket added in v0.0.7 

func ProcessUpstreamDatabaseTableColumnStatisticsBucket(dbTypeS, dbCharsetS string, caseFieldRule string,
	database database.IDatabase, schemaName, tableName string, cons *structure.HighestBucket, chunkSize int64, enableCollation bool) (*structure.HighestBucket, []*structure.Range, error)

func ReverseUpstreamHighestBucketDownstreamRule added in v0.0.7 

func ReverseUpstreamHighestBucketDownstreamRule(taskFlow, dbTypeT, dbCharsetS string, columnDatatypeT []string, cons *structure.HighestBucket, columnRouteRule map[string]string) (*structure.Rule, error)

Types

type CsvMigrateRow added in v0.0.7 

type CsvMigrateRow struct {
	Ctx        context.Context
	TaskMode   string
	TaskFlow   string
	BufioSize  int
	Dmt        *task.DataMigrateTask
	DatabaseS  database.IDatabase
	DBCharsetS string
	DBCharsetT string
	TaskParams *pb.CsvMigrateParam
	ReadChan   chan []string
	WriteChan  chan string
}

func (*CsvMigrateRow) MigrateApply added in v0.0.7 

func (r *CsvMigrateRow) MigrateApply() error

func (*CsvMigrateRow) MigrateProcess added in v0.0.7 

func (r *CsvMigrateRow) MigrateProcess() error

func (*CsvMigrateRow) MigrateRead added in v0.0.7 

func (r *CsvMigrateRow) MigrateRead() error

type DataCompareFile added in v0.0.7 

type DataCompareFile struct {
	Ctx        context.Context `json:"-"`
	Mutex      *sync.Mutex     `json:"-"`
	CompFile   *os.File        `json:"-"`
	CompWriter *bufio.Writer   `json:"-"`
	TaskName   string          `json:"taskName"`
	TaskFlow   string          `json:"taskFlow"`
	OutputDir  string          `json:"outputDir"`
}

func NewDataCompareFile added in v0.0.7 

func NewDataCompareFile(ctx context.Context,
	taskName, taskFlow, outputDir string) *DataCompareFile

func (*DataCompareFile) Close added in v0.0.7 

func (s *DataCompareFile) Close() error

func (*DataCompareFile) InitFile added in v0.0.7 

func (s *DataCompareFile) InitFile() error

func (*DataCompareFile) SyncFile added in v0.0.7 

func (s *DataCompareFile) SyncFile() error

type DataCompareRow added in v0.0.7 

type DataCompareRow struct {
	Ctx            context.Context
	StartTime      time.Time
	TaskMode       string
	TaskFlow       string
	Dmt            *task.DataCompareTask
	DatabaseS      database.IDatabase
	DatabaseT      database.IDatabase
	BatchSize      int
	WriteThread    int
	CallTimeout    int
	DBCharsetS     string
	DBCharsetT     string
	RepairStmtFlow string
}

func (*DataCompareRow) CompareCRC32 added in v0.0.7 

func (r *DataCompareRow) CompareCRC32() error

func (*DataCompareRow) CompareMd5ORCrc32 added in v0.0.8 

func (r *DataCompareRow) CompareMd5ORCrc32() error

func (*DataCompareRow) CompareMethod added in v0.0.7 

func (r *DataCompareRow) CompareMethod() string

func (*DataCompareRow) CompareRows added in v0.0.7 

func (r *DataCompareRow) CompareRows() error

type DataCompareRule added in v0.0.7 

type DataCompareRule struct {
	Ctx                         context.Context    `json:"-"`
	TaskName                    string             `json:"taskName"`
	TaskMode                    string             `json:"taskMode"`
	TaskFlow                    string             `json:"taskFlow"`
	SchemaNameS                 string             `json:"schemaNameS"`
	TableNameS                  string             `json:"tableNameS"`
	SchemaNameT                 string             `json:"schemNameT"`
	TableNameT                  string             `json:"tableNameT"`
	GlobalSqlHintS              string             `json:"globalSqlHintS"`
	GlobalSqlHintT              string             `json:"globalSqlHintT"`
	TableTypeS                  map[string]string  `json:"tableTypeS"`
	ColumnNameSliS              []string           `json:"columnNameSliS"`
	IgnoreSelectFields          []string           `json:"ignoreSelectFields"`
	GlobalIgnoreConditionFields []string           `json:"globalIgnoreConditionFields"`
	OnlyDatabaseCompareRow      bool               `json:"onlyDatabaseCompareRow"`
	OnlyProgramCompareCRC32     bool               `json:"onlyProgramCompareCRC32"`
	DisableDatabaseCompareMd5   bool               `json:"disableDatabaseCompareMd5"`
	DatabaseS                   database.IDatabase `json:"-"`
	DatabaseT                   database.IDatabase `json:"-"`
	DBCharsetS                  string             `json:"DBCharsetS"`
	DBCharsetT                  string             `json:"DBCharsetT"`
	CaseFieldRuleS              string             `json:"caseFieldRuleS"`
	CaseFieldRuleT              string             `json:"caseFieldRuleT"`
}

func (*DataCompareRule) GenSchemaNameRule added in v0.0.7 

func (r *DataCompareRule) GenSchemaNameRule() (string, string, error)

func (*DataCompareRule) GenSchemaTableColumnSelectRule added in v0.0.7 

func (r *DataCompareRule) GenSchemaTableColumnSelectRule() (string, string, string, string, error)

func (*DataCompareRule) GenSchemaTableCompareMethodRule added in v0.0.7 

func (r *DataCompareRule) GenSchemaTableCompareMethodRule() string

func (*DataCompareRule) GenSchemaTableCustomRule added in v0.0.7 

func (r *DataCompareRule) GenSchemaTableCustomRule() (string, string, []string, string, string, error)

func (*DataCompareRule) GenSchemaTableNameRule added in v0.0.7 

func (r *DataCompareRule) GenSchemaTableNameRule() (string, string, error)

func (*DataCompareRule) GenSchemaTableTypeRule added in v0.0.7 

func (r *DataCompareRule) GenSchemaTableTypeRule() string

func (*DataCompareRule) GetSchemaTableColumnNameRule added in v0.0.7 

func (r *DataCompareRule) GetSchemaTableColumnNameRule() (map[string]string, error)

type DataMigrateRule added in v0.0.7 

type DataMigrateRule struct {
	Ctx            context.Context    `json:"-"`
	TaskName       string             `json:"taskName"`
	TaskMode       string             `json:"taskMode"`
	TaskFlow       string             `json:"taskFlow"`
	SchemaNameS    string             `json:"schemaNameS"`
	TableNameS     string             `json:"tableNameS"`
	GlobalSqlHintS string             `json:"globalSqlHintS"`
	TableTypeS     map[string]string  `json:"tableTypeS"`
	DatabaseS      database.IDatabase `json:"databaseS"`
	DBCharsetS     string             `json:"DBCharsetS"`
	CaseFieldRuleS string             `json:"caseFieldRuleS"`
	CaseFieldRuleT string             `json:"caseFieldRuleT"`
}

func (*DataMigrateRule) GenSchemaNameRule added in v0.0.7 

func (r *DataMigrateRule) GenSchemaNameRule() (string, string, error)

func (*DataMigrateRule) GenSchemaTableColumnSelectRule added in v0.0.7 

func (r *DataMigrateRule) GenSchemaTableColumnSelectRule() (string, string, string, string, error)

func (*DataMigrateRule) GenSchemaTableCustomRule added in v0.0.7 

func (r *DataMigrateRule) GenSchemaTableCustomRule() (bool, string, string, error)

func (*DataMigrateRule) GenSchemaTableNameRule added in v0.0.7 

func (r *DataMigrateRule) GenSchemaTableNameRule() (string, string, error)

func (*DataMigrateRule) GenSchemaTableTypeRule added in v0.0.7 

func (r *DataMigrateRule) GenSchemaTableTypeRule() string

func (*DataMigrateRule) GetSchemaTableColumnNameRule added in v0.0.7 

func (r *DataMigrateRule) GetSchemaTableColumnNameRule() (map[string]string, error)

type DataScanFile added in v0.0.7 

type DataScanFile struct {
	Ctx        context.Context `json:"-"`
	Mutex      *sync.Mutex     `json:"-"`
	CompFile   *os.File        `json:"-"`
	CompWriter *bufio.Writer   `json:"-"`
	TaskName   string          `json:"taskName"`
	TaskMode   string          `json:"taskMode"`
	TaskFlow   string          `json:"taskFlow"`
	OutputDir  string          `json:"outputDir"`
}

func NewDataScanFile added in v0.0.7 

func NewDataScanFile(ctx context.Context,
	taskName, taskMode, taskFlow, outputDir string) *DataScanFile

func (*DataScanFile) Close added in v0.0.7 

func (s *DataScanFile) Close() error

func (*DataScanFile) InitFile added in v0.0.7 

func (s *DataScanFile) InitFile() error

func (*DataScanFile) SyncFile added in v0.0.7 

func (s *DataScanFile) SyncFile() error

type DataScanRow added in v0.0.7 

type DataScanRow struct {
	Ctx        context.Context
	StartTime  time.Time
	TaskName   string
	TaskMode   string
	TaskFlow   string
	Dst        *task.DataScanTask
	DatabaseS  database.IDatabase
	DBCharsetS string
}

func (*DataScanRow) ScanRows added in v0.0.7 

func (r *DataScanRow) ScanRows() error

type DataScanRule added in v0.0.7 

type DataScanRule struct {
	Ctx               context.Context    `json:"-"`
	TaskName          string             `json:"taskName"`
	TaskMode          string             `json:"taskMode"`
	TaskFlow          string             `json:"taskFlow"`
	SchemaNameS       string             `json:"schemaNameS"`
	TableNameS        string             `json:"tableNameS"`
	GlobalSamplerateS string             `json:"globalSamplerateS"`
	GlobalSqlHintS    string             `json:"globalSqlHintS"`
	TableTypeS        map[string]string  `json:"tableTypeS"`
	DatabaseS         database.IDatabase `json:"-"`
	DBCharsetS        string             `json:"DBCharsetS"`
}

func (*DataScanRule) GenSchemaNameRule added in v0.0.7 

func (r *DataScanRule) GenSchemaNameRule() (string, error)

func (*DataScanRule) GenSchemaTableCustomRule added in v0.0.7 

func (r *DataScanRule) GenSchemaTableCustomRule() (string, string, error)

func (*DataScanRule) GenSchemaTableTypeRule added in v0.0.7 

func (r *DataScanRule) GenSchemaTableTypeRule() string

func (*DataScanRule) GenTableColumnNameRule added in v0.0.7 

func (r *DataScanRule) GenTableColumnNameRule() (string, string, error)

func (*DataScanRule) GenTableNameRule added in v0.0.7 

func (r *DataScanRule) GenTableNameRule() (string, error)

type Datasource added in v0.0.7 

type Datasource struct {
	DatabaseS   database.IDatabase `json:"-"`
	SchemaNameS string             `json:"schemaNameS"`
	TableNameS  string             `json:"tableNameS"`
	TableTypeS  string             `json:"tableTypeS"`
}

func (*Datasource) GetTableCharsetCollation added in v0.0.7 

func (d *Datasource) GetTableCharsetCollation() (string, string, error)

func (*Datasource) GetTableCheckKey added in v0.0.7 

func (d *Datasource) GetTableCheckKey() ([]map[string]string, error)

func (*Datasource) GetTableColumnComment added in v0.0.7 

func (d *Datasource) GetTableColumnComment() ([]map[string]string, error)

func (*Datasource) GetTableColumns added in v0.0.7 

func (d *Datasource) GetTableColumns() ([]map[string]string, error)

func (*Datasource) GetTableComment added in v0.0.7 

func (d *Datasource) GetTableComment() ([]map[string]string, error)

func (*Datasource) GetTableForeignKey added in v0.0.7 

func (d *Datasource) GetTableForeignKey() ([]map[string]string, error)

func (*Datasource) GetTableNormalIndex added in v0.0.7 

func (d *Datasource) GetTableNormalIndex() ([]map[string]string, error)

func (*Datasource) GetTableOriginStruct added in v0.0.7 

func (d *Datasource) GetTableOriginStruct() (string, error)

func (*Datasource) GetTablePrimaryKey added in v0.0.7 

func (d *Datasource) GetTablePrimaryKey() ([]map[string]string, error)

func (*Datasource) GetTableUniqueIndex added in v0.0.7 

func (d *Datasource) GetTableUniqueIndex() ([]map[string]string, error)

func (*Datasource) GetTableUniqueKey added in v0.0.7 

func (d *Datasource) GetTableUniqueKey() ([]map[string]string, error)

func (*Datasource) String added in v0.0.7 

func (d *Datasource) String() string

type MySQLProcessor 

type MySQLProcessor struct {
	Ctx        context.Context `json:"-"`
	TaskName   string          `json:"taskName"`
	TaskFlow   string          `json:"taskFlow"`
	SchemaName string          `json:"schemaName"`
	TableName  string          `json:"tableName"`

	DBCharset                string                           `json:"dbCharset"`
	BuildinDatatypeRules     []*buildin.BuildinDatatypeRule   `json:"-"`
	BuildinDefaultValueRules []*buildin.BuildinDefaultvalRule `json:"-"`
	ColumnRouteRules         map[string]string                `json:"columnRouteRules"`
	IsBaseline               bool                             `json:"isBaseline"`
	Database                 database.IDatabase               `json:"-"`
}

func (*MySQLProcessor) GenDatabaseSchemaTable 

func (p *MySQLProcessor) GenDatabaseSchemaTable() (string, string)

func (*MySQLProcessor) GenDatabaseTableCharset 

func (p *MySQLProcessor) GenDatabaseTableCharset() (string, error)

func (*MySQLProcessor) GenDatabaseTableCheckConstraintDetail 

func (p *MySQLProcessor) GenDatabaseTableCheckConstraintDetail() (map[string]structure.ConstraintCheck, error)

func (*MySQLProcessor) GenDatabaseTableCollation 

func (p *MySQLProcessor) GenDatabaseTableCollation() (string, error)

func (*MySQLProcessor) GenDatabaseTableColumnDetail 

func (p *MySQLProcessor) GenDatabaseTableColumnDetail() (map[string]structure.NewColumn, map[string]map[string]structure.OldColumn, error)

func (*MySQLProcessor) GenDatabaseTableComment 

func (p *MySQLProcessor) GenDatabaseTableComment() (string, error)

func (*MySQLProcessor) GenDatabaseTableForeignConstraintDetail 

func (p *MySQLProcessor) GenDatabaseTableForeignConstraintDetail() (map[string]structure.ConstraintForeign, error)

func (*MySQLProcessor) GenDatabaseTableIndexDetail 

func (p *MySQLProcessor) GenDatabaseTableIndexDetail() (map[string]structure.Index, error)

func (*MySQLProcessor) GenDatabaseTablePartitionDetail 

func (p *MySQLProcessor) GenDatabaseTablePartitionDetail() ([]structure.Partition, error)

func (*MySQLProcessor) GenDatabaseTablePrimaryConstraintDetail 

func (p *MySQLProcessor) GenDatabaseTablePrimaryConstraintDetail() (map[string]structure.ConstraintPrimary, error)

func (*MySQLProcessor) GenDatabaseTableUniqueConstraintDetail 

func (p *MySQLProcessor) GenDatabaseTableUniqueConstraintDetail() (map[string]structure.ConstraintUnique, error)

type OracleProcessor 

type OracleProcessor struct {
	Ctx                      context.Context                  `json:"-"`
	TaskName                 string                           `json:"taskName"`
	TaskFlow                 string                           `json:"taskFlow"`
	SchemaName               string                           `json:"schemaName"`
	TableName                string                           `json:"tableName"`
	DBCharset                string                           `json:"dbCharset"`
	Database                 database.IDatabase               `json:"-"`
	BuildinDatatypeRules     []*buildin.BuildinDatatypeRule   `json:"-"`
	BuildinDefaultValueRules []*buildin.BuildinDefaultvalRule `json:"-"`
	ColumnRouteRules         map[string]string                `json:"columnRouteRules"`
	IsBaseline               bool                             `json:"isBaseline"`
	TableCollation           string                           `json:"tableCollation"`
	SchemaCollation          string                           `json:"schemaCollation"`
	NLSComp                  string                           `json:"NLSComp"`
	DBCollation              bool                             `json:"DBCollation"`
}

func (*OracleProcessor) GenDatabaseSchemaTable 

func (p *OracleProcessor) GenDatabaseSchemaTable() (string, string)

func (*OracleProcessor) GenDatabaseTableCharset 

func (p *OracleProcessor) GenDatabaseTableCharset() (string, error)

func (*OracleProcessor) GenDatabaseTableCheckConstraintDetail 

func (p *OracleProcessor) GenDatabaseTableCheckConstraintDetail() (map[string]structure.ConstraintCheck, error)

func (*OracleProcessor) GenDatabaseTableCollation 

func (p *OracleProcessor) GenDatabaseTableCollation() (string, error)

func (*OracleProcessor) GenDatabaseTableColumnDetail 

func (p *OracleProcessor) GenDatabaseTableColumnDetail() (map[string]structure.NewColumn, map[string]map[string]structure.OldColumn, error)

func (*OracleProcessor) GenDatabaseTableComment 

func (p *OracleProcessor) GenDatabaseTableComment() (string, error)

func (*OracleProcessor) GenDatabaseTableForeignConstraintDetail 

func (p *OracleProcessor) GenDatabaseTableForeignConstraintDetail() (map[string]structure.ConstraintForeign, error)

func (*OracleProcessor) GenDatabaseTableIndexDetail 

func (p *OracleProcessor) GenDatabaseTableIndexDetail() (map[string]structure.Index, error)

func (*OracleProcessor) GenDatabaseTablePartitionDetail 

func (p *OracleProcessor) GenDatabaseTablePartitionDetail() ([]structure.Partition, error)

func (*OracleProcessor) GenDatabaseTablePrimaryConstraintDetail 

func (p *OracleProcessor) GenDatabaseTablePrimaryConstraintDetail() (map[string]structure.ConstraintPrimary, error)

func (*OracleProcessor) GenDatabaseTableUniqueConstraintDetail 

func (p *OracleProcessor) GenDatabaseTableUniqueConstraintDetail() (map[string]structure.ConstraintUnique, error)

type ScanResultMYSQLCompatible added in v0.0.7 

type ScanResultMYSQLCompatible struct {
	ColumnName     string
	Bigint         string
	BigintUnsigned string
	DecimalInt     string
	DecimalPoint   string
	Unknown        string
}

type SequenceMigrateDatabase added in v0.0.7 

type SequenceMigrateDatabase struct {
	Ctx           context.Context    `json:"-"`
	TaskName      string             `json:"taskName"`
	TaskFlow      string             `json:"taskFlow"`
	SchemaNameS   string             `json:"schemaNameS"`
	SchemaNameT   string             `json:"schemaNameT"`
	TaskStartTime time.Time          `json:"-"`
	DatasourceT   database.IDatabase `json:"-"`
	Sequences     []string           `json:"sequences"`
}

func NewSequenceMigrateDatabase added in v0.0.7 

func NewSequenceMigrateDatabase(ctx context.Context,
	taskName, taskFlow, schemaNameS, schemaNameT string, datasourceT database.IDatabase,
	taskStartTime time.Time, seqs []string) *SequenceMigrateDatabase

func (*SequenceMigrateDatabase) GenSequenceDigest added in v0.0.7 

func (s *SequenceMigrateDatabase) GenSequenceDigest() (string, error)

func (*SequenceMigrateDatabase) SyncSequenceDatabase added in v0.0.7 

func (s *SequenceMigrateDatabase) SyncSequenceDatabase() error

func (*SequenceMigrateDatabase) WriteSequenceDatabase added in v0.0.7 

func (s *SequenceMigrateDatabase) WriteSequenceDatabase() error

type SqlMigrateRow added in v0.0.7 

type SqlMigrateRow struct {
	Ctx           context.Context
	TaskMode      string
	TaskFlow      string
	Smt           *task.SqlMigrateTask
	DatabaseS     database.IDatabase
	DatabaseT     database.IDatabase
	DatabaseTStmt *sql.Stmt
	DBCharsetS    string
	DBCharsetT    string
	SqlThreadT    int
	BatchSize     int
	CallTimeout   int
	SafeMode      bool
	ReadChan      chan []interface{}
	WriteChan     chan []interface{}
}

func (*SqlMigrateRow) MigrateApply added in v0.0.7 

func (r *SqlMigrateRow) MigrateApply() error

func (*SqlMigrateRow) MigrateProcess added in v0.0.7 

func (r *SqlMigrateRow) MigrateProcess() error

func (*SqlMigrateRow) MigrateRead added in v0.0.7 

func (r *SqlMigrateRow) MigrateRead() error

type SqlMigrateRule added in v0.0.7 

type SqlMigrateRule struct {
	Ctx             context.Context    `json:"-"`
	TaskName        string             `json:"taskName"`
	TaskMode        string             `json:"taskMode"`
	TaskFlow        string             `json:"taskFlow"`
	SchemaNameT     string             `json:"schemaNameT"`
	TableNameT      string             `json:"tableNameT"`
	SqlHintT        string             `json:"sqlHintT"`
	GlobalSqlHintT  string             `json:"globalSqlHintT"`
	DatabaseS       database.IDatabase `json:"databaseS"`
	DBCharsetS      string             `json:"DBCharsetS"`
	SqlQueryS       string             `json:"sqlQueryS"`
	ColumnRouteRule map[string]string  `json:"columnRouteRule"`
	CaseFieldRuleS  string             `json:"caseFieldRuleS"`
	CaseFieldRuleT  string             `json:"caseFieldRuleT"`
}

func (*SqlMigrateRule) GenSqlMigrateSchemaNameRule added in v0.0.7 

func (r *SqlMigrateRule) GenSqlMigrateSchemaNameRule() (string, error)

func (*SqlMigrateRule) GenSqlMigrateTableColumnRule added in v0.0.7 

func (r *SqlMigrateRule) GenSqlMigrateTableColumnRule() (string, string, string, error)

func (*SqlMigrateRule) GenSqlMigrateTableCustomRule added in v0.0.7 

func (r *SqlMigrateRule) GenSqlMigrateTableCustomRule() (string, string)

func (*SqlMigrateRule) GenSqlMigrateTableNameRule added in v0.0.7 

func (r *SqlMigrateRule) GenSqlMigrateTableNameRule() (string, error)

type StmtMigrateRow added in v0.0.7 

type StmtMigrateRow struct {
	Ctx           context.Context
	TaskMode      string
	TaskFlow      string
	Dmt           *task.DataMigrateTask
	DatabaseS     database.IDatabase
	DatabaseT     database.IDatabase
	DatabaseTStmt *sql.Stmt
	DBCharsetS    string
	DBCharsetT    string
	SqlThreadT    int
	BatchSize     int
	CallTimeout   int
	SafeMode      bool
	ReadChan      chan []interface{}
	WriteChan     chan []interface{}
}

func (*StmtMigrateRow) MigrateApply added in v0.0.7 

func (r *StmtMigrateRow) MigrateApply() error

func (*StmtMigrateRow) MigrateProcess added in v0.0.7 

func (r *StmtMigrateRow) MigrateProcess() error

func (*StmtMigrateRow) MigrateRead added in v0.0.7 

func (r *StmtMigrateRow) MigrateRead() error

type StructCompareFile added in v0.0.7 

type StructCompareFile struct {
	Ctx        context.Context `json:"-"`
	Mutex      *sync.Mutex     `json:"-"`
	CompFile   *os.File        `json:"-"`
	CompWriter *bufio.Writer   `json:"-"`
	TaskName   string          `json:"taskName"`
	TaskFlow   string          `json:"taskFlow"`
	OutputDir  string          `json:"outputDir"`
}

func NewStructCompareFile added in v0.0.7 

func NewStructCompareFile(ctx context.Context,
	taskName, taskFlow, outputDir string) *StructCompareFile

func (*StructCompareFile) Close added in v0.0.7 

func (s *StructCompareFile) Close() error

func (*StructCompareFile) InitFile added in v0.0.7 

func (s *StructCompareFile) InitFile() error

func (*StructCompareFile) SyncFile added in v0.0.7 

func (s *StructCompareFile) SyncFile() error

type StructMigrateDatabase added in v0.0.7 

type StructMigrateDatabase struct {
	Ctx           context.Context       `json:"-"`
	TaskName      string                `json:"taskName"`
	TaskFlow      string                `json:"taskFlow"`
	TaskStartTime time.Time             `json:"-"`
	DatasourceT   database.IDatabase    `json:"-"`
	TableStruct   *database.TableStruct `json:"tableStruct"`
}

func NewStructMigrateDatabase added in v0.0.7 

func NewStructMigrateDatabase(ctx context.Context,
	taskName, taskFlow string, datasourceT database.IDatabase,
	taskStartTime time.Time, tableStruct *database.TableStruct) *StructMigrateDatabase

func (*StructMigrateDatabase) GenTableStructDDL added in v0.0.7 

func (s *StructMigrateDatabase) GenTableStructDDL() ([]string, []string, error)

func (*StructMigrateDatabase) GenTableStructDigest added in v0.0.7 

func (s *StructMigrateDatabase) GenTableStructDigest() (string, string, string, error)

func (*StructMigrateDatabase) SyncStructDatabase added in v0.0.7 

func (s *StructMigrateDatabase) SyncStructDatabase() error

func (*StructMigrateDatabase) WriteStructDatabase added in v0.0.7 

func (s *StructMigrateDatabase) WriteStructDatabase() error

WriteStructDatabase used for sync file, current only write database, but not sync target database

type StructMigrateFile added in v0.0.7 

type StructMigrateFile struct {
	Ctx              context.Context `json:"-"`
	Mutex            *sync.Mutex     `json:"-"`
	CompFile         *os.File        `json:"-"`
	InCompFile       *os.File        `json:"-"`
	CompWriter       *bufio.Writer   `json:"-"`
	InCompWriter     *bufio.Writer   `json:"-"`
	TaskName         string          `json:"taskName"`
	TaskFlow         string          `json:"taskFlow"`
	DBCharsetS       string          `json:"dbCharsetS"`
	DBCollationS     bool            `json:"dbCollationS"`
	SchemaCollationS string          `json:"schemaCollationS"`
	DBNlsComp        string          `json:"DBNlsComp"`
	SchemaNameS      string          `json:"schemaNameS"`
	SchemaNameT      string          `json:"schemaNameT"`
	OutputDir        string          `json:"outputDir"`
}

func NewStructMigrateFile added in v0.0.7 

func NewStructMigrateFile(ctx context.Context,
	taskName, taskFlow, schemaNameS string, outputDir string) *StructMigrateFile

func (*StructMigrateFile) Close added in v0.0.7 

func (s *StructMigrateFile) Close() error

func (*StructMigrateFile) InitOutputFile added in v0.0.7 

func (s *StructMigrateFile) InitOutputFile() error

func (*StructMigrateFile) SyncSequenceFile added in v0.0.7 

func (s *StructMigrateFile) SyncSequenceFile() error

func (*StructMigrateFile) SyncStructFile added in v0.0.7 

func (s *StructMigrateFile) SyncStructFile() error

type StructMigrateRule added in v0.0.7 

type StructMigrateRule struct {
	Ctx                      context.Context                  `json:"-"`
	TaskName                 string                           `json:"taskName"`
	TaskMode                 string                           `json:"taskMode"`
	TaskFlow                 string                           `json:"taskFlow"`
	SchemaNameS              string                           `json:"schemaNameS"`
	TableNameS               string                           `json:"tableNameS"`
	TablePrimaryAttrs        []map[string]string              `json:"tablePrimaryAttrs"`
	TableColumnsAttrs        []map[string]string              `json:"tableColumnsAttrs"`
	TableCommentAttrs        []map[string]string              `json:"tableCommentAttrs"`
	CaseFieldRuleT           string                           `json:"caseFieldRuleT"`
	CreateIfNotExist         bool                             `json:"createIfNotExist"`
	DBCollationS             bool                             `json:"DBCollationS"`
	DBCharsetS               string                           `json:"dbCharsetS"`
	DBCharsetT               string                           `json:"dbCharsetT"`
	BuildinDatatypeRules     []*buildin.BuildinDatatypeRule   `json:"-"`
	BuildinDefaultValueRules []*buildin.BuildinDefaultvalRule `json:"-"`
}

func (*StructMigrateRule) GetCaseFieldRule added in v0.0.7 

func (r *StructMigrateRule) GetCaseFieldRule() string

func (*StructMigrateRule) GetCreatePrefixRule added in v0.0.7 

func (r *StructMigrateRule) GetCreatePrefixRule() string

func (*StructMigrateRule) GetSchemaNameRule added in v0.0.7 

func (r *StructMigrateRule) GetSchemaNameRule() (map[string]string, error)

func (*StructMigrateRule) GetTableAttributesRule added in v0.0.7 

func (r *StructMigrateRule) GetTableAttributesRule() (string, error)

func (*StructMigrateRule) GetTableColumnCollationRule added in v0.0.7 

func (r *StructMigrateRule) GetTableColumnCollationRule() (map[string]string, error)

func (*StructMigrateRule) GetTableColumnCommentRule added in v0.0.7 

func (r *StructMigrateRule) GetTableColumnCommentRule() (map[string]string, error)

func (*StructMigrateRule) GetTableColumnRule added in v0.0.7 

func (r *StructMigrateRule) GetTableColumnRule() (map[string]string, map[string]string, map[string]string, error)

GetTableColumnRule used for get custom table column rule column datatype rule priority: - column level - table level - task level - default level

func (*StructMigrateRule) GetTableCommentRule added in v0.0.7 

func (r *StructMigrateRule) GetTableCommentRule() (string, error)

func (*StructMigrateRule) GetTableNameRule added in v0.0.7 

func (r *StructMigrateRule) GetTableNameRule() (map[string]string, error)

func (*StructMigrateRule) String added in v0.0.7 

func (r *StructMigrateRule) String() string

type Table 

type Table struct {
	TaskName string
	TaskFlow string
	Source   *structure.Table
	Target   *structure.Table
}

Table structure comparison Based on the upstream Oracle table structure information, compare the downstream MySQL table structure 1. If the upstream exists and the downstream does not exist, the record will be output. If the upstream does not exist and the downstream exists, the record will not be output by default. 2. Ignore the comparison of different index names and constraint names between the upstream and downstream, and only compare whether the same fields exist under the same constraints downstream. 3. Partitions only compare partition types, partition keys, partition expressions, etc., and do not compare the specific conditions of each partition.

func (*Table) ComparePartitionTableType 

func (t *Table) ComparePartitionTableType() string

func (*Table) CompareTableCharsetCollation 

func (t *Table) CompareTableCharsetCollation() string

func (*Table) CompareTableCheckConstraint 

func (t *Table) CompareTableCheckConstraint() (string, error)

func (*Table) CompareTableColumnCharsetCollation 

func (t *Table) CompareTableColumnCharsetCollation() string

func (*Table) CompareTableColumnCounts 

func (t *Table) CompareTableColumnCounts() string

func (*Table) CompareTableColumnDetail 

func (t *Table) CompareTableColumnDetail() (string, error)

func (*Table) CompareTableComment 

func (t *Table) CompareTableComment() string

func (*Table) CompareTableForeignConstraint 

func (t *Table) CompareTableForeignConstraint() (string, error)

func (*Table) CompareTableIndexDetail 

func (t *Table) CompareTableIndexDetail() (string, error)

func (*Table) CompareTablePartitionDetail 

func (t *Table) CompareTablePartitionDetail() (string, error)

func (*Table) CompareTablePrimaryConstraint 

func (t *Table) CompareTablePrimaryConstraint() (string, error)

func (*Table) CompareTableUniqueConstraint 

func (t *Table) CompareTableUniqueConstraint() (string, error)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.