processor

package
v0.0.16 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 12, 2024 License: Apache-2.0 Imports: 35 Imported by: 0

Documentation

Overview

Copyright © 2020 Marvin

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed p. in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Cmp added in v0.0.7

func Cmp(src map[string]int64, dest map[string]int64) (map[string]int64, map[string]int64)

Cmp used for the src and dest store key-value pair , and key data row and value data row counts

func DivideDatabaseTableColumnStatisticsBucket added in v0.0.7

func DivideDatabaseTableColumnStatisticsBucket(database database.IDatabase, schemaName, tableName string, cons *structure.Selectivity, chunkRange *structure.Range, divideCountCnt int) ([]*structure.Range, error)

DivideDatabaseTableColumnStatisticsBucket splits a chunk to multiple chunks by random Notice: If the `count <= 1`, it will skip splitting and return `chunk` as a slice directly.

func ExtractDatabaseTableStatisticsValuesFromBuckets added in v0.0.7

func ExtractDatabaseTableStatisticsValuesFromBuckets(divideDbType, valueString string, columnNames []string) ([]string, error)

ExtractDatabaseTableStatisticsValuesFromBuckets analyze upperBound or lowerBound to string for each column. upperBound and lowerBound are looks like '(123, abc)' for multiple fields, or '123' for one field.

func GenMYSQLCompatibleDatabaseDeleteStmtSQL added in v0.0.7

func GenMYSQLCompatibleDatabaseDeleteStmtSQL(schemaName, tableName, sqlHint string, columnDetailSlice []string, columnDataString []string, columnDataCounts int) string

func GenMYSQLCompatibleDatabaseInsertStmtSQL added in v0.0.7

func GenMYSQLCompatibleDatabaseInsertStmtSQL(schemaName, tableName, sqlHint string, columnDetailSlice []string, columnDataString []string, safeMode bool, columnDataCounts ...int) string

func GenMYSQLCompatibleDatabasePrepareStmt added in v0.0.7

func GenMYSQLCompatibleDatabasePrepareStmt(
	schemaName, tableName string, sqlHint, columnDetailSlice string, insertBatchSize int, safeMode bool) string

func GenOracleCompatibleDatabaseDeleteStmtSQL added in v0.0.7

func GenOracleCompatibleDatabaseDeleteStmtSQL(schemaName, tableName, sqlHint string, columnDetailSlice []string, columnDataString []string, columnDataCounts int) string

func GenOracleCompatibleDatabaseInsertStmtSQL added in v0.0.7

func GenOracleCompatibleDatabaseInsertStmtSQL(schemaName, tableName, sqlHint string, columnDetailSlice []string, columnDataString []string, safeMode bool, columnDataCounts ...int) string

func GenPostgresCompatibleDatabaseDeleteStmtSQL added in v0.0.15

func GenPostgresCompatibleDatabaseDeleteStmtSQL(schemaName, tableName, sqlHint string, columnDetailSlice []string, columnDataString []string, columnDataCounts int) string

func GenPostgresCompatibleDatabaseInsertStmtSQL added in v0.0.15

func GenPostgresCompatibleDatabaseInsertStmtSQL(schemaName, tableName, sqlHint string, columnDetailSlice []string, columnDataString []string, safeMode bool, columnDataCounts ...int) string

func GetDownstreamTableColumnDatatype added in v0.0.9

func GetDownstreamTableColumnDatatype(schemaNameT, tableNameT string, databaseT database.IDatabase, originColumnNameSli []string, columnNameRouteRule map[string]string) ([]string, error)

func InspectOracleMigrateTask added in v0.0.7

func InspectOracleMigrateTask(taskName, taskFlow, taskMode string, databaseS database.IDatabase, connectDBCharsetS, connectDBCharsetT string) (string, error)

func InspectPostgresMigrateTask added in v0.0.15

func InspectPostgresMigrateTask(taskName, taskFlow, taskMode string, databaseS database.IDatabase, connectDBCharsetS, connectDBCharsetT string) (string, error)

func OptimizerMYSQLCompatibleDataMigrateColumnS added in v0.0.7

func OptimizerMYSQLCompatibleDataMigrateColumnS(columnName, datatype, datetimePrecision string) (string, error)

func OptimizerMYSQLCompatibleMigrateOracleDataCompareColumnST added in v0.0.7

func OptimizerMYSQLCompatibleMigrateOracleDataCompareColumnST(columnNameS, datatypeS string, datetimePrecisionS, dataLengthS int64, dataPrecisionS, dataScaleS, dbCharsetSDest, columnNameT, dbCharsetTFrom, dbCharsetTDest string) (string, string, error)

func OptimizerMYSQLCompatibleMigratePostgresDataCompareColumnST added in v0.0.15

func OptimizerMYSQLCompatibleMigratePostgresDataCompareColumnST(columnNameS, datatypeS string, datatypeIsBooleanT bool, datetimePrecisionS int64, dataPrecisionS, dataScaleS, dbCharsetSDest, columnNameT, dbCharsetTDest string) (string, string, error)

func OptimizerOracleDataMigrateColumnS added in v0.0.7

func OptimizerOracleDataMigrateColumnS(columnName, datatype, dataScale string) (string, error)

func OptimizerOracleMigrateMYSQLCompatibleDataCompareColumnST added in v0.0.7

func OptimizerOracleMigrateMYSQLCompatibleDataCompareColumnST(columnNameS, datatypeS string, dataLengthS int64, dataPrecisionS, dataScaleS, dbCharsetSFrom, dbCharsetSDest, columnNameT, dbCharsetTDest string) (string, string, error)

func OptimizerPostgresCompatibleMigrateMySQLCompatibleDataCompareColumnST added in v0.0.15

func OptimizerPostgresCompatibleMigrateMySQLCompatibleDataCompareColumnST(columnNameS, datatypeS string, datetimePrecisionS int64, dataPrecisionS, dataScaleS, dbCharsetSDest, columnNameT, dbCharsetTDest string) (string, string, error)

func OptimizerPostgresDataMigrateColumnS added in v0.0.15

func OptimizerPostgresDataMigrateColumnS(columnName, datatype, datetimePrecision string) (string, error)

func ProcessDownstreamStatisticsBucket added in v0.0.9

func ProcessDownstreamStatisticsBucket(dbTypeT, dbCharsetT string, bs []*structure.Range, r *structure.Rule) ([]*structure.Range, error)

Types

type CsvMigrateRow added in v0.0.7

type CsvMigrateRow struct {
	Ctx        context.Context
	TaskMode   string
	TaskFlow   string
	BufioSize  int
	Dmt        *task.DataMigrateTask
	DatabaseS  database.IDatabase
	DBCharsetS string
	DBCharsetT string
	TaskParams *pb.CsvMigrateParam
	ReadChan   chan []string
	WriteChan  chan string
}

func (*CsvMigrateRow) MigrateApply added in v0.0.7

func (r *CsvMigrateRow) MigrateApply() error

func (*CsvMigrateRow) MigrateProcess added in v0.0.7

func (r *CsvMigrateRow) MigrateProcess() error

func (*CsvMigrateRow) MigrateRead added in v0.0.7

func (r *CsvMigrateRow) MigrateRead() error

type CsvMigrateTask added in v0.0.14

type CsvMigrateTask struct {
	Ctx             context.Context
	Task            *task.Task
	DBRoleS         string
	DBVersionS      string
	DBCharsetS      string
	DBCharsetT      string
	DatabaseS       database.IDatabase
	DatabaseT       database.IDatabase
	SchemaNameS     string
	SchemaNameT     string
	GlobalSnapshotS string

	CsvParams *pb.CsvMigrateParam

	WaiterC chan *WaitingRecs
	ResumeC chan *WaitingRecs
}

func (*CsvMigrateTask) Init added in v0.0.14

func (cmt *CsvMigrateTask) Init() error

func (*CsvMigrateTask) Last added in v0.0.14

func (cmt *CsvMigrateTask) Last() error

func (*CsvMigrateTask) PrepareStatisticsRange added in v0.0.14

func (cmt *CsvMigrateTask) PrepareStatisticsRange(globalScn string, attsRule *database.DataMigrateAttributesRule, r *structure.Range, chunkID int) (*task.DataMigrateTask, error)

func (*CsvMigrateTask) Process added in v0.0.14

func (cmt *CsvMigrateTask) Process(s *WaitingRecs) error

func (*CsvMigrateTask) ProcessChunkScan added in v0.0.14

func (cmt *CsvMigrateTask) ProcessChunkScan(ctx context.Context, globalScn string, tableRows uint64, tableSize float64, attsRule *database.DataMigrateAttributesRule) error

func (*CsvMigrateTask) ProcessStatisticsScan added in v0.0.14

func (cmt *CsvMigrateTask) ProcessStatisticsScan(ctx context.Context, dbTypeS, globalScn string, tableRows uint64, tableSize float64, attsRule *database.DataMigrateAttributesRule) error

func (*CsvMigrateTask) ProcessTableScan added in v0.0.14

func (cmt *CsvMigrateTask) ProcessTableScan(ctx context.Context, globalScn string, tableRows uint64, tableSize float64, attsRule *database.DataMigrateAttributesRule) error

func (*CsvMigrateTask) Resume added in v0.0.14

func (cmt *CsvMigrateTask) Resume() error

func (*CsvMigrateTask) Run added in v0.0.14

func (cmt *CsvMigrateTask) Run() error

type DataCompareFile added in v0.0.7

type DataCompareFile struct {
	Ctx         context.Context `json:"-"`
	Mutex       *sync.Mutex     `json:"-"`
	CompFile    *os.File        `json:"-"`
	CompWriter  *bufio.Writer   `json:"-"`
	TaskName    string          `json:"taskName"`
	TaskFlow    string          `json:"taskFlow"`
	SchemaNameS string          `json:"schemaNameS"`
	TableNameS  string          `json:"tableNameS"`
	OutputDir   string          `json:"outputDir"`
}

func NewDataCompareFile added in v0.0.7

func NewDataCompareFile(ctx context.Context,
	taskName, taskFlow, schemaName, tableName, outputDir string) *DataCompareFile

func (*DataCompareFile) Close added in v0.0.7

func (s *DataCompareFile) Close() error

func (*DataCompareFile) InitFile added in v0.0.7

func (s *DataCompareFile) InitFile() error

func (*DataCompareFile) SyncFile added in v0.0.7

func (s *DataCompareFile) SyncFile() error

type DataCompareRow added in v0.0.7

type DataCompareRow struct {
	Ctx            context.Context
	StartTime      time.Time
	TaskMode       string
	TaskFlow       string
	Dmt            *task.DataCompareTask
	DatabaseS      database.IDatabase
	DatabaseT      database.IDatabase
	BatchSize      int
	WriteThread    int
	CallTimeout    int
	DBCharsetS     string
	DBCharsetT     string
	RepairStmtFlow string
	Separator      string
}

func (*DataCompareRow) CompareCRC32 added in v0.0.7

func (r *DataCompareRow) CompareCRC32() error

func (*DataCompareRow) CompareMd5ORCrc32 added in v0.0.8

func (r *DataCompareRow) CompareMd5ORCrc32() error

func (*DataCompareRow) CompareMethod added in v0.0.7

func (r *DataCompareRow) CompareMethod() string

func (*DataCompareRow) CompareRows added in v0.0.7

func (r *DataCompareRow) CompareRows() error

type DataCompareRule added in v0.0.7

type DataCompareRule struct {
	Ctx                         context.Context    `json:"-"`
	TaskName                    string             `json:"taskName"`
	TaskMode                    string             `json:"taskMode"`
	TaskFlow                    string             `json:"taskFlow"`
	SchemaNameS                 string             `json:"schemaNameS"`
	TableNameS                  string             `json:"tableNameS"`
	SchemaNameT                 string             `json:"schemNameT"`
	TableNameT                  string             `json:"tableNameT"`
	GlobalSqlHintS              string             `json:"globalSqlHintS"`
	GlobalSqlHintT              string             `json:"globalSqlHintT"`
	TableTypeS                  map[string]string  `json:"tableTypeS"`
	ColumnNameSliS              []string           `json:"columnNameSliS"`
	IgnoreSelectFields          []string           `json:"ignoreSelectFields"`
	GlobalIgnoreConditionFields []string           `json:"globalIgnoreConditionFields"`
	OnlyDatabaseCompareRow      bool               `json:"onlyDatabaseCompareRow"`
	OnlyProgramCompareCRC32     bool               `json:"onlyProgramCompareCRC32"`
	DisableDatabaseCompareMd5   bool               `json:"disableDatabaseCompareMd5"`
	DatabaseS                   database.IDatabase `json:"-"`
	DatabaseT                   database.IDatabase `json:"-"`
	DBCharsetS                  string             `json:"DBCharsetS"`
	DBCharsetT                  string             `json:"DBCharsetT"`
	CaseFieldRuleS              string             `json:"caseFieldRuleS"`
	CaseFieldRuleT              string             `json:"caseFieldRuleT"`
}

func (*DataCompareRule) GenSchemaNameRule added in v0.0.7

func (r *DataCompareRule) GenSchemaNameRule() (string, string, error)

func (*DataCompareRule) GenSchemaTableColumnSelectRule added in v0.0.7

func (r *DataCompareRule) GenSchemaTableColumnSelectRule() (string, string, string, string, error)

func (*DataCompareRule) GenSchemaTableCompareMethodRule added in v0.0.7

func (r *DataCompareRule) GenSchemaTableCompareMethodRule() string

func (*DataCompareRule) GenSchemaTableCustomRule added in v0.0.7

func (r *DataCompareRule) GenSchemaTableCustomRule() (string, string, string, []string, string, string, error)

func (*DataCompareRule) GenSchemaTableNameRule added in v0.0.7

func (r *DataCompareRule) GenSchemaTableNameRule() (string, string, error)

func (*DataCompareRule) GenSchemaTableTypeRule added in v0.0.7

func (r *DataCompareRule) GenSchemaTableTypeRule() string

func (*DataCompareRule) GetSchemaTableColumnNameRule added in v0.0.7

func (r *DataCompareRule) GetSchemaTableColumnNameRule() (map[string]string, error)

type DataCompareTask added in v0.0.9

type DataCompareTask struct {
	Ctx         context.Context
	Task        *task.Task
	DatabaseS   database.IDatabase
	DatabaseT   database.IDatabase
	SchemaNameS string

	DBCharsetS string
	DBCharsetT string

	GlobalSnapshotS string
	GlobalSnapshotT string
	TaskParams      *pb.DataCompareParam

	WaiterC chan *WaitingRecs
	ResumeC chan *WaitingRecs
}

func (*DataCompareTask) Init added in v0.0.9

func (dmt *DataCompareTask) Init() error

func (*DataCompareTask) Last added in v0.0.11

func (dmt *DataCompareTask) Last() error

func (*DataCompareTask) PrepareStatisticsRange added in v0.0.9

func (dmt *DataCompareTask) PrepareStatisticsRange(globalScnS, globalScnT string, attsRule *database.DataCompareAttributesRule, upRanges, downRanges []*structure.Range) ([]*task.DataCompareTask, error)

func (*DataCompareTask) Process added in v0.0.9

func (dmt *DataCompareTask) Process(s *WaitingRecs) error

func (*DataCompareTask) ProcessStatisticsScan added in v0.0.9

func (dmt *DataCompareTask) ProcessStatisticsScan(ctx context.Context, dbTypeS, dbTypeT, globalScnS, globalScnT string, tableRows uint64, tableSize float64, attsRule *database.DataCompareAttributesRule) error

func (*DataCompareTask) ProcessTableScan added in v0.0.9

func (dmt *DataCompareTask) ProcessTableScan(ctx context.Context, globalScnS, globalScnT string, tableRows uint64, tableSize float64, attsRule *database.DataCompareAttributesRule) error

func (*DataCompareTask) Resume added in v0.0.9

func (dmt *DataCompareTask) Resume() error

func (*DataCompareTask) Run added in v0.0.9

func (dmt *DataCompareTask) Run() error

type DataMigrateRule added in v0.0.7

type DataMigrateRule struct {
	Ctx            context.Context    `json:"-"`
	TaskName       string             `json:"taskName"`
	TaskMode       string             `json:"taskMode"`
	TaskFlow       string             `json:"taskFlow"`
	SchemaNameS    string             `json:"schemaNameS"`
	TableNameS     string             `json:"tableNameS"`
	GlobalSqlHintS string             `json:"globalSqlHintS"`
	TableTypeS     map[string]string  `json:"tableTypeS"`
	DatabaseS      database.IDatabase `json:"databaseS"`
	DBCharsetS     string             `json:"DBCharsetS"`
	CaseFieldRuleS string             `json:"caseFieldRuleS"`
	CaseFieldRuleT string             `json:"caseFieldRuleT"`
}

func (*DataMigrateRule) GenSchemaNameRule added in v0.0.7

func (r *DataMigrateRule) GenSchemaNameRule() (string, string, error)

func (*DataMigrateRule) GenSchemaTableColumnSelectRule added in v0.0.7

func (r *DataMigrateRule) GenSchemaTableColumnSelectRule() (string, string, string, string, error)

func (*DataMigrateRule) GenSchemaTableCustomRule added in v0.0.7

func (r *DataMigrateRule) GenSchemaTableCustomRule() (bool, string, string, error)

func (*DataMigrateRule) GenSchemaTableNameRule added in v0.0.7

func (r *DataMigrateRule) GenSchemaTableNameRule() (string, string, error)

func (*DataMigrateRule) GenSchemaTableTypeRule added in v0.0.7

func (r *DataMigrateRule) GenSchemaTableTypeRule() string

func (*DataMigrateRule) GetSchemaTableColumnNameRule added in v0.0.7

func (r *DataMigrateRule) GetSchemaTableColumnNameRule() (map[string]string, error)

func (*DataMigrateRule) String added in v0.0.15

func (r *DataMigrateRule) String() string

type DataMigrateTask added in v0.0.9

type DataMigrateTask struct {
	Ctx             context.Context
	Task            *task.Task
	DBRoleS         string
	DBVersionS      string
	DBCharsetS      string
	DBCharsetT      string
	DatabaseS       database.IDatabase
	DatabaseT       database.IDatabase
	SchemaNameS     string
	SchemaNameT     string
	GlobalSnapshotS string
	StmtParams      *pb.StatementMigrateParam

	WaiterC chan *WaitingRecs
	ResumeC chan *WaitingRecs
}

func (*DataMigrateTask) Init added in v0.0.9

func (cmt *DataMigrateTask) Init() error

func (*DataMigrateTask) Last added in v0.0.11

func (cmt *DataMigrateTask) Last() error

func (*DataMigrateTask) PrepareStatisticsRange added in v0.0.9

func (cmt *DataMigrateTask) PrepareStatisticsRange(globalScn string, attsRule *database.DataMigrateAttributesRule, r *structure.Range, chunkID int) (*task.DataMigrateTask, error)

func (*DataMigrateTask) Process added in v0.0.9

func (cmt *DataMigrateTask) Process(s *WaitingRecs) error

func (*DataMigrateTask) ProcessChunkScan added in v0.0.9

func (cmt *DataMigrateTask) ProcessChunkScan(ctx context.Context, globalScn string, tableRows uint64, tableSize float64, attsRule *database.DataMigrateAttributesRule) error

func (*DataMigrateTask) ProcessStatisticsScan added in v0.0.9

func (cmt *DataMigrateTask) ProcessStatisticsScan(ctx context.Context, dbTypeS, globalScn string, tableRows uint64, tableSize float64, attsRule *database.DataMigrateAttributesRule) error

func (*DataMigrateTask) ProcessTableScan added in v0.0.9

func (cmt *DataMigrateTask) ProcessTableScan(ctx context.Context, globalScn string, tableRows uint64, tableSize float64, attsRule *database.DataMigrateAttributesRule) error

func (*DataMigrateTask) Resume added in v0.0.9

func (cmt *DataMigrateTask) Resume() error

func (*DataMigrateTask) Run added in v0.0.9

func (cmt *DataMigrateTask) Run() error

type DataScanFile added in v0.0.7

type DataScanFile struct {
	Ctx        context.Context `json:"-"`
	Mutex      *sync.Mutex     `json:"-"`
	CompFile   *os.File        `json:"-"`
	CompWriter *bufio.Writer   `json:"-"`
	TaskName   string          `json:"taskName"`
	TaskMode   string          `json:"taskMode"`
	TaskFlow   string          `json:"taskFlow"`
	OutputDir  string          `json:"outputDir"`
}

func NewDataScanFile added in v0.0.7

func NewDataScanFile(ctx context.Context,
	taskName, taskMode, taskFlow, outputDir string) *DataScanFile

func (*DataScanFile) Close added in v0.0.7

func (s *DataScanFile) Close() error

func (*DataScanFile) InitFile added in v0.0.7

func (s *DataScanFile) InitFile() error

func (*DataScanFile) SyncFile added in v0.0.7

func (s *DataScanFile) SyncFile() error

type DataScanRow added in v0.0.7

type DataScanRow struct {
	Ctx        context.Context
	StartTime  time.Time
	TaskName   string
	TaskMode   string
	TaskFlow   string
	Dst        *task.DataScanTask
	DatabaseS  database.IDatabase
	DBCharsetS string
}

func (*DataScanRow) ScanRows added in v0.0.7

func (r *DataScanRow) ScanRows() error

type DataScanRule added in v0.0.7

type DataScanRule struct {
	Ctx               context.Context    `json:"-"`
	TaskName          string             `json:"taskName"`
	TaskMode          string             `json:"taskMode"`
	TaskFlow          string             `json:"taskFlow"`
	SchemaNameS       string             `json:"schemaNameS"`
	TableNameS        string             `json:"tableNameS"`
	GlobalSamplerateS string             `json:"globalSamplerateS"`
	GlobalSqlHintS    string             `json:"globalSqlHintS"`
	TableTypeS        map[string]string  `json:"tableTypeS"`
	DatabaseS         database.IDatabase `json:"-"`
	DBCharsetS        string             `json:"DBCharsetS"`
}

func (*DataScanRule) GenSchemaNameRule added in v0.0.7

func (r *DataScanRule) GenSchemaNameRule() (string, error)

func (*DataScanRule) GenSchemaTableCustomRule added in v0.0.7

func (r *DataScanRule) GenSchemaTableCustomRule() (string, string, error)

func (*DataScanRule) GenSchemaTableTypeRule added in v0.0.7

func (r *DataScanRule) GenSchemaTableTypeRule() string

func (*DataScanRule) GenTableColumnNameRule added in v0.0.7

func (r *DataScanRule) GenTableColumnNameRule() (string, string, error)

func (*DataScanRule) GenTableNameRule added in v0.0.7

func (r *DataScanRule) GenTableNameRule() (string, error)

type DataScanTask added in v0.0.9

type DataScanTask struct {
	Ctx         context.Context
	Task        *task.Task
	DatabaseS   database.IDatabase
	SchemaNameS string
	DBRoleS     string
	DBCharsetS  string
	DBVersionS  string

	TaskParams *pb.DataScanParam
	WaiterC    chan *WaitingRecs
	ResumeC    chan *WaitingRecs
}

func (*DataScanTask) Init added in v0.0.9

func (dst *DataScanTask) Init() error

func (*DataScanTask) Last added in v0.0.11

func (dst *DataScanTask) Last() error

func (*DataScanTask) PrepareStatisticsRange added in v0.0.9

func (dst *DataScanTask) PrepareStatisticsRange(globalScn string, attsRule *database.DataScanAttributesRule, ranges []*structure.Range) ([]*task.DataScanTask, error)

func (*DataScanTask) Process added in v0.0.9

func (dst *DataScanTask) Process(s *WaitingRecs) error

func (*DataScanTask) ProcessChunkScan added in v0.0.9

func (dst *DataScanTask) ProcessChunkScan(ctx context.Context, schemaNameS, tableNameS, globalScn string, tableRows uint64, tableSize float64, attsRule *database.DataScanAttributesRule) error

func (*DataScanTask) ProcessStatisticsScan added in v0.0.9

func (dst *DataScanTask) ProcessStatisticsScan(ctx context.Context, dbTypeS, globalScn string, tableRows uint64, tableSize float64, attsRule *database.DataScanAttributesRule) error

func (*DataScanTask) ProcessTableScan added in v0.0.9

func (dst *DataScanTask) ProcessTableScan(ctx context.Context, globalScn string, tableRows uint64, tableSize float64, attsRule *database.DataScanAttributesRule) error

func (*DataScanTask) Resume added in v0.0.9

func (dst *DataScanTask) Resume() error

func (*DataScanTask) Run added in v0.0.9

func (dst *DataScanTask) Run() error

type Datasource added in v0.0.7

type Datasource struct {
	DBTypeS     string             `json:"dbTypeS"`
	DBVersionS  string             `json:"dbVersionS"`
	DatabaseS   database.IDatabase `json:"-"`
	SchemaNameS string             `json:"schemaNameS"`
	TableNameS  string             `json:"tableNameS"`
	TableTypeS  string             `json:"tableTypeS"`
}

func (*Datasource) GetTableCharsetCollation added in v0.0.7

func (d *Datasource) GetTableCharsetCollation() (string, string, error)

func (*Datasource) GetTableCheckKey added in v0.0.7

func (d *Datasource) GetTableCheckKey() ([]map[string]string, error)

func (*Datasource) GetTableColumnComment added in v0.0.7

func (d *Datasource) GetTableColumnComment() ([]map[string]string, error)

func (*Datasource) GetTableColumns added in v0.0.7

func (d *Datasource) GetTableColumns() ([]map[string]string, error)

func (*Datasource) GetTableComment added in v0.0.7

func (d *Datasource) GetTableComment() ([]map[string]string, error)

func (*Datasource) GetTableForeignKey added in v0.0.7

func (d *Datasource) GetTableForeignKey() ([]map[string]string, error)

func (*Datasource) GetTableNormalIndex added in v0.0.7

func (d *Datasource) GetTableNormalIndex() ([]map[string]string, error)

func (*Datasource) GetTableOriginStruct added in v0.0.7

func (d *Datasource) GetTableOriginStruct() (string, error)

func (*Datasource) GetTablePrimaryKey added in v0.0.7

func (d *Datasource) GetTablePrimaryKey() ([]map[string]string, error)

func (*Datasource) GetTableUniqueIndex added in v0.0.7

func (d *Datasource) GetTableUniqueIndex() ([]map[string]string, error)

func (*Datasource) GetTableUniqueKey added in v0.0.7

func (d *Datasource) GetTableUniqueKey() ([]map[string]string, error)

func (*Datasource) String added in v0.0.7

func (d *Datasource) String() string

type Divide added in v0.0.9

type Divide struct {
	DBTypeS     string
	DBCharsetS  string
	SchemaNameS string
	TableNameS  string
	ChunkSize   int64
	DatabaseS   database.IDatabase
	Cons        *structure.Selectivity
	RangeC      chan []*structure.Range
}

func (*Divide) ProcessUpstreamStatisticsBucket added in v0.0.9

func (d *Divide) ProcessUpstreamStatisticsBucket() error

type ImportTiDBDatabase added in v0.0.14

type ImportTiDBDatabase struct {
	CharsetSet          string `json:"charsetSet"`
	FieldsTerminatedBy  string `json:"fieldsTerminatedBy"`
	FieldsEnclosedBy    string `json:"fieldsEnclosedBy"`
	FieldsEscapedBy     string `json:"fieldsEscapedBy"`
	FieldsDefinedNullBy string `json:"fieldsDefinedNullBy"`
	LinesTerminatedBy   string `json:"linesTerminatedBy"`
	SkipRows            int    `json:"skipRows"`
	ImportTiDBParams
}

func (*ImportTiDBDatabase) Import added in v0.0.14

func (i *ImportTiDBDatabase) Import(ctx context.Context, db database.IDatabase, schemaNameT, tableNameT, columnNameT, outputCsvDir string) error

type ImportTiDBParams added in v0.0.14

type ImportTiDBParams struct {
	DiskQuota             string `json:"diskQuota"`
	Thread                string `json:"thread"`
	MaxWriteSpeed         string `json:"maxWriteSpeed"`
	ChecksumTable         string `json:"checksumTable"`
	CloudStorageUri       string `json:"cloudStorageUri"`
	Detached              string `json:"detached"`
	DisableTikvImportMode string `json:"disableTikvImportMode"`
	SplitFile             string `json:"splitFile"` // strict-format
}

func (*ImportTiDBParams) Builder added in v0.0.14

func (p *ImportTiDBParams) Builder() (string, error)

type MySQLProcessor

type MySQLProcessor struct {
	Ctx        context.Context `json:"-"`
	TaskName   string          `json:"taskName"`
	TaskFlow   string          `json:"taskFlow"`
	SchemaName string          `json:"schemaName"`
	TableName  string          `json:"tableName"`

	DBCharset                string                           `json:"dbCharset"`
	BuildinDatatypeRules     []*buildin.BuildinDatatypeRule   `json:"-"`
	BuildinDefaultValueRules []*buildin.BuildinDefaultvalRule `json:"-"`
	ColumnRouteRules         map[string]string                `json:"columnRouteRules"`
	IsBaseline               bool                             `json:"isBaseline"`
	Database                 database.IDatabase               `json:"-"`
}

func (*MySQLProcessor) GenDatabaseSchemaTable

func (p *MySQLProcessor) GenDatabaseSchemaTable() (string, string)

func (*MySQLProcessor) GenDatabaseTableCharset

func (p *MySQLProcessor) GenDatabaseTableCharset() (string, error)

func (*MySQLProcessor) GenDatabaseTableCheckConstraintDetail

func (p *MySQLProcessor) GenDatabaseTableCheckConstraintDetail() (map[string]structure.ConstraintCheck, error)

func (*MySQLProcessor) GenDatabaseTableCollation

func (p *MySQLProcessor) GenDatabaseTableCollation() (string, error)

func (*MySQLProcessor) GenDatabaseTableColumnDetail

func (p *MySQLProcessor) GenDatabaseTableColumnDetail() (map[string]structure.NewColumn, map[string]map[string]structure.OldColumn, error)

func (*MySQLProcessor) GenDatabaseTableComment

func (p *MySQLProcessor) GenDatabaseTableComment() (string, error)

func (*MySQLProcessor) GenDatabaseTableForeignConstraintDetail

func (p *MySQLProcessor) GenDatabaseTableForeignConstraintDetail() (map[string]structure.ConstraintForeign, error)

func (*MySQLProcessor) GenDatabaseTableIndexDetail

func (p *MySQLProcessor) GenDatabaseTableIndexDetail() (map[string]structure.Index, error)

func (*MySQLProcessor) GenDatabaseTablePartitionDetail

func (p *MySQLProcessor) GenDatabaseTablePartitionDetail() ([]structure.Partition, error)

func (*MySQLProcessor) GenDatabaseTablePrimaryConstraintDetail

func (p *MySQLProcessor) GenDatabaseTablePrimaryConstraintDetail() (map[string]structure.ConstraintPrimary, error)

func (*MySQLProcessor) GenDatabaseTableUniqueConstraintDetail

func (p *MySQLProcessor) GenDatabaseTableUniqueConstraintDetail() (map[string]structure.ConstraintUnique, error)

type OracleProcessor

type OracleProcessor struct {
	Ctx                      context.Context                  `json:"-"`
	TaskName                 string                           `json:"taskName"`
	TaskFlow                 string                           `json:"taskFlow"`
	SchemaName               string                           `json:"schemaName"`
	TableName                string                           `json:"tableName"`
	DBCharset                string                           `json:"dbCharset"`
	Database                 database.IDatabase               `json:"-"`
	BuildinDatatypeRules     []*buildin.BuildinDatatypeRule   `json:"-"`
	BuildinDefaultValueRules []*buildin.BuildinDefaultvalRule `json:"-"`
	ColumnRouteRules         map[string]string                `json:"columnRouteRules"`
	IsBaseline               bool                             `json:"isBaseline"`
	TableCollation           string                           `json:"tableCollation"`
	SchemaCollation          string                           `json:"schemaCollation"`
	NLSComp                  string                           `json:"NLSComp"`
	DBCollation              bool                             `json:"DBCollation"`
}

func (*OracleProcessor) GenDatabaseSchemaTable

func (p *OracleProcessor) GenDatabaseSchemaTable() (string, string)

func (*OracleProcessor) GenDatabaseTableCharset

func (p *OracleProcessor) GenDatabaseTableCharset() (string, error)

func (*OracleProcessor) GenDatabaseTableCheckConstraintDetail

func (p *OracleProcessor) GenDatabaseTableCheckConstraintDetail() (map[string]structure.ConstraintCheck, error)

func (*OracleProcessor) GenDatabaseTableCollation

func (p *OracleProcessor) GenDatabaseTableCollation() (string, error)

func (*OracleProcessor) GenDatabaseTableColumnDetail

func (p *OracleProcessor) GenDatabaseTableColumnDetail() (map[string]structure.NewColumn, map[string]map[string]structure.OldColumn, error)

func (*OracleProcessor) GenDatabaseTableComment

func (p *OracleProcessor) GenDatabaseTableComment() (string, error)

func (*OracleProcessor) GenDatabaseTableForeignConstraintDetail

func (p *OracleProcessor) GenDatabaseTableForeignConstraintDetail() (map[string]structure.ConstraintForeign, error)

func (*OracleProcessor) GenDatabaseTableIndexDetail

func (p *OracleProcessor) GenDatabaseTableIndexDetail() (map[string]structure.Index, error)

func (*OracleProcessor) GenDatabaseTablePartitionDetail

func (p *OracleProcessor) GenDatabaseTablePartitionDetail() ([]structure.Partition, error)

func (*OracleProcessor) GenDatabaseTablePrimaryConstraintDetail

func (p *OracleProcessor) GenDatabaseTablePrimaryConstraintDetail() (map[string]structure.ConstraintPrimary, error)

func (*OracleProcessor) GenDatabaseTableUniqueConstraintDetail

func (p *OracleProcessor) GenDatabaseTableUniqueConstraintDetail() (map[string]structure.ConstraintUnique, error)

type PostgresProcessor added in v0.0.16

type PostgresProcessor struct {
	Ctx                      context.Context                  `json:"-"`
	TaskName                 string                           `json:"taskName"`
	TaskFlow                 string                           `json:"taskFlow"`
	SchemaName               string                           `json:"schemaName"`
	TableName                string                           `json:"tableName"`
	DBCharset                string                           `json:"dbCharset"`
	Database                 database.IDatabase               `json:"-"`
	BuildinDatatypeRules     []*buildin.BuildinDatatypeRule   `json:"-"`
	BuildinDefaultValueRules []*buildin.BuildinDefaultvalRule `json:"-"`
	ColumnRouteRules         map[string]string                `json:"columnRouteRules"`
	IsBaseline               bool                             `json:"isBaseline"`
	TableCharset             string                           `json:"tableCharset"`
	TableCollation           string                           `json:"tableCollation"`
}

func (*PostgresProcessor) GenDatabaseSchemaTable added in v0.0.16

func (p *PostgresProcessor) GenDatabaseSchemaTable() (string, string)

func (*PostgresProcessor) GenDatabaseTableCharset added in v0.0.16

func (p *PostgresProcessor) GenDatabaseTableCharset() (string, error)

func (*PostgresProcessor) GenDatabaseTableCheckConstraintDetail added in v0.0.16

func (p *PostgresProcessor) GenDatabaseTableCheckConstraintDetail() (map[string]structure.ConstraintCheck, error)

func (*PostgresProcessor) GenDatabaseTableCollation added in v0.0.16

func (p *PostgresProcessor) GenDatabaseTableCollation() (string, error)

func (*PostgresProcessor) GenDatabaseTableColumnDetail added in v0.0.16

func (p *PostgresProcessor) GenDatabaseTableColumnDetail() (map[string]structure.NewColumn, map[string]map[string]structure.OldColumn, error)

func (*PostgresProcessor) GenDatabaseTableComment added in v0.0.16

func (p *PostgresProcessor) GenDatabaseTableComment() (string, error)

func (*PostgresProcessor) GenDatabaseTableForeignConstraintDetail added in v0.0.16

func (p *PostgresProcessor) GenDatabaseTableForeignConstraintDetail() (map[string]structure.ConstraintForeign, error)

func (*PostgresProcessor) GenDatabaseTableIndexDetail added in v0.0.16

func (p *PostgresProcessor) GenDatabaseTableIndexDetail() (map[string]structure.Index, error)

func (*PostgresProcessor) GenDatabaseTablePartitionDetail added in v0.0.16

func (p *PostgresProcessor) GenDatabaseTablePartitionDetail() ([]structure.Partition, error)

func (*PostgresProcessor) GenDatabaseTablePrimaryConstraintDetail added in v0.0.16

func (p *PostgresProcessor) GenDatabaseTablePrimaryConstraintDetail() (map[string]structure.ConstraintPrimary, error)

func (*PostgresProcessor) GenDatabaseTableUniqueConstraintDetail added in v0.0.16

func (p *PostgresProcessor) GenDatabaseTableUniqueConstraintDetail() (map[string]structure.ConstraintUnique, error)

type ScanResultMYSQLCompatible added in v0.0.7

type ScanResultMYSQLCompatible struct {
	ColumnName     string
	Bigint         string
	BigintUnsigned string
	DecimalInt     string
	DecimalPoint   string
	Unknown        string
}

type SequenceMigrateDatabase added in v0.0.7

type SequenceMigrateDatabase struct {
	Ctx            context.Context           `json:"-"`
	TaskName       string                    `json:"taskName"`
	TaskMode       string                    `json:"taskMode"`
	TaskFlow       string                    `json:"taskFlow"`
	TaskStartTime  time.Time                 `json:"-"`
	DatasourceT    database.IDatabase        `json:"-"`
	SeqMigrateTask *task.SequenceMigrateTask `json:"seqMigrateTask"`
}

func NewSequenceMigrateDatabase added in v0.0.7

func NewSequenceMigrateDatabase(ctx context.Context,
	taskName, taskMode, taskFlow string, datasourceT database.IDatabase,
	taskStartTime time.Time, seqs *task.SequenceMigrateTask) *SequenceMigrateDatabase

func (*SequenceMigrateDatabase) SyncSequenceDatabase added in v0.0.7

func (s *SequenceMigrateDatabase) SyncSequenceDatabase() error

func (*SequenceMigrateDatabase) WriteSequenceDatabase added in v0.0.7

func (s *SequenceMigrateDatabase) WriteSequenceDatabase() error

type SqlMigrateRow added in v0.0.7

type SqlMigrateRow struct {
	Ctx               context.Context
	TaskMode          string
	TaskFlow          string
	Smt               *task.SqlMigrateTask
	DatabaseS         database.IDatabase
	DatabaseT         database.IDatabase
	DatabaseTStmt     *sql.Stmt
	DBCharsetS        string
	DBCharsetT        string
	SqlThreadT        int
	BatchSize         int
	CallTimeout       int
	SafeMode          bool
	EnablePrepareStmt bool
	ReadChan          chan []interface{}
	WriteChan         chan []interface{}
}

func (*SqlMigrateRow) MigrateApply added in v0.0.7

func (r *SqlMigrateRow) MigrateApply() error

func (*SqlMigrateRow) MigrateProcess added in v0.0.7

func (r *SqlMigrateRow) MigrateProcess() error

func (*SqlMigrateRow) MigrateRead added in v0.0.7

func (r *SqlMigrateRow) MigrateRead() error

type SqlMigrateRule added in v0.0.7

type SqlMigrateRule struct {
	Ctx             context.Context    `json:"-"`
	TaskName        string             `json:"taskName"`
	TaskMode        string             `json:"taskMode"`
	TaskFlow        string             `json:"taskFlow"`
	SchemaNameT     string             `json:"schemaNameT"`
	TableNameT      string             `json:"tableNameT"`
	SqlHintT        string             `json:"sqlHintT"`
	GlobalSqlHintT  string             `json:"globalSqlHintT"`
	DatabaseS       database.IDatabase `json:"databaseS"`
	DBCharsetS      string             `json:"DBCharsetS"`
	SqlQueryS       string             `json:"sqlQueryS"`
	ColumnRouteRule map[string]string  `json:"columnRouteRule"`
	CaseFieldRuleS  string             `json:"caseFieldRuleS"`
	CaseFieldRuleT  string             `json:"caseFieldRuleT"`
}

func (*SqlMigrateRule) GenSqlMigrateSchemaNameRule added in v0.0.7

func (r *SqlMigrateRule) GenSqlMigrateSchemaNameRule() (string, error)

func (*SqlMigrateRule) GenSqlMigrateTableColumnRule added in v0.0.7

func (r *SqlMigrateRule) GenSqlMigrateTableColumnRule() (string, string, string, error)

func (*SqlMigrateRule) GenSqlMigrateTableCustomRule added in v0.0.7

func (r *SqlMigrateRule) GenSqlMigrateTableCustomRule() (string, string)

func (*SqlMigrateRule) GenSqlMigrateTableNameRule added in v0.0.7

func (r *SqlMigrateRule) GenSqlMigrateTableNameRule() (string, error)

type SqlMigrateTask added in v0.0.9

type SqlMigrateTask struct {
	Ctx        context.Context
	Task       *task.Task
	DatabaseS  database.IDatabase
	DatabaseT  database.IDatabase
	DBRoleS    string
	DBCharsetS string
	DBCharsetT string
	DBVersionS string

	TaskParams *pb.SqlMigrateParam
}

func (*SqlMigrateTask) Init added in v0.0.9

func (smt *SqlMigrateTask) Init() error

func (*SqlMigrateTask) Last added in v0.0.11

func (smt *SqlMigrateTask) Last() error

func (*SqlMigrateTask) Process added in v0.0.9

func (smt *SqlMigrateTask) Process() error

func (*SqlMigrateTask) Resume added in v0.0.9

func (smt *SqlMigrateTask) Resume() error

func (*SqlMigrateTask) Run added in v0.0.9

func (smt *SqlMigrateTask) Run() error

type StmtMigrateRow added in v0.0.7

type StmtMigrateRow struct {
	Ctx               context.Context
	TaskMode          string
	TaskFlow          string
	Dmt               *task.DataMigrateTask
	DatabaseS         database.IDatabase
	DatabaseT         database.IDatabase
	DatabaseTStmt     *sql.Stmt
	DBCharsetS        string
	DBCharsetT        string
	SqlThreadT        int
	BatchSize         int
	CallTimeout       int
	SafeMode          bool
	EnablePrepareStmt bool
	ReadChan          chan []interface{}
	WriteChan         chan []interface{}
}

func (*StmtMigrateRow) MigrateApply added in v0.0.7

func (r *StmtMigrateRow) MigrateApply() error

func (*StmtMigrateRow) MigrateProcess added in v0.0.7

func (r *StmtMigrateRow) MigrateProcess() error

func (*StmtMigrateRow) MigrateRead added in v0.0.7

func (r *StmtMigrateRow) MigrateRead() error

type StructCompareFile added in v0.0.7

type StructCompareFile struct {
	Ctx        context.Context `json:"-"`
	Mutex      *sync.Mutex     `json:"-"`
	CompFile   *os.File        `json:"-"`
	CompWriter *bufio.Writer   `json:"-"`
	TaskName   string          `json:"taskName"`
	TaskFlow   string          `json:"taskFlow"`
	OutputDir  string          `json:"outputDir"`
}

func NewStructCompareFile added in v0.0.7

func NewStructCompareFile(ctx context.Context,
	taskName, taskFlow, outputDir string) *StructCompareFile

func (*StructCompareFile) Close added in v0.0.7

func (s *StructCompareFile) Close() error

func (*StructCompareFile) InitFile added in v0.0.7

func (s *StructCompareFile) InitFile() error

func (*StructCompareFile) SyncFile added in v0.0.7

func (s *StructCompareFile) SyncFile() error

type StructCompareTask added in v0.0.9

type StructCompareTask struct {
	Ctx                       context.Context
	Task                      *task.Task
	DBTypeS                   string
	DBTypeT                   string
	DatabaseS                 database.IDatabase
	DatabaseT                 database.IDatabase
	DBCharsetS                string
	DBCharsetT                string
	SchemaNameS               string
	SchemaNameT               string
	StartTime                 time.Time
	TaskParams                *pb.StructCompareParam
	BuildInDatatypeRulesS     []*buildin.BuildinDatatypeRule
	BuildInDefaultValueRulesS []*buildin.BuildinDefaultvalRule
	BuildInDatatypeRulesT     []*buildin.BuildinDatatypeRule
	BuildInDefaultValueRulesT []*buildin.BuildinDefaultvalRule

	ReadyInit chan bool
}

func (*StructCompareTask) Init added in v0.0.9

func (dmt *StructCompareTask) Init() error

func (*StructCompareTask) Last added in v0.0.11

func (dmt *StructCompareTask) Last() error

func (*StructCompareTask) Process added in v0.0.9

func (dmt *StructCompareTask) Process() error

func (*StructCompareTask) Resume added in v0.0.9

func (dmt *StructCompareTask) Resume() error

func (*StructCompareTask) Run added in v0.0.9

func (dmt *StructCompareTask) Run() error

type StructMigrateDatabase added in v0.0.7

type StructMigrateDatabase struct {
	Ctx           context.Context       `json:"-"`
	TaskName      string                `json:"taskName"`
	TaskFlow      string                `json:"taskFlow"`
	TaskStartTime time.Time             `json:"-"`
	DatasourceT   database.IDatabase    `json:"-"`
	TableStruct   *database.TableStruct `json:"tableStruct"`
}

func NewStructMigrateDatabase added in v0.0.7

func NewStructMigrateDatabase(ctx context.Context,
	taskName, taskFlow string, datasourceT database.IDatabase,
	taskStartTime time.Time, tableStruct *database.TableStruct) *StructMigrateDatabase

func (*StructMigrateDatabase) GenTableStructDDL added in v0.0.7

func (s *StructMigrateDatabase) GenTableStructDDL() ([]string, []string, error)

func (*StructMigrateDatabase) GenTableStructDigest added in v0.0.7

func (s *StructMigrateDatabase) GenTableStructDigest() (string, string, string, error)

func (*StructMigrateDatabase) SyncStructDatabase added in v0.0.7

func (s *StructMigrateDatabase) SyncStructDatabase() error

func (*StructMigrateDatabase) WriteStructDatabase added in v0.0.7

func (s *StructMigrateDatabase) WriteStructDatabase() error

WriteStructDatabase used for sync file, current only write database, but not sync target database

type StructMigrateFile added in v0.0.7

type StructMigrateFile struct {
	Ctx              context.Context `json:"-"`
	Mutex            *sync.Mutex     `json:"-"`
	CompFile         *os.File        `json:"-"`
	InCompFile       *os.File        `json:"-"`
	CompWriter       *bufio.Writer   `json:"-"`
	InCompWriter     *bufio.Writer   `json:"-"`
	TaskName         string          `json:"taskName"`
	TaskFlow         string          `json:"taskFlow"`
	DBCharsetS       string          `json:"dbCharsetS"`
	DBCollationS     bool            `json:"dbCollationS"`
	SchemaCollationS string          `json:"schemaCollationS"`
	DBNlsComp        string          `json:"DBNlsComp"`
	SchemaNameS      string          `json:"schemaNameS"`
	SchemaNameT      string          `json:"schemaNameT"`
	OutputDir        string          `json:"outputDir"`
}

func NewStructMigrateFile added in v0.0.7

func NewStructMigrateFile(ctx context.Context,
	taskName, taskFlow, schemaNameS string, outputDir string) *StructMigrateFile

func (*StructMigrateFile) Close added in v0.0.7

func (s *StructMigrateFile) Close() error

func (*StructMigrateFile) InitOutputFile added in v0.0.7

func (s *StructMigrateFile) InitOutputFile() error

func (*StructMigrateFile) SyncSequenceFile added in v0.0.7

func (s *StructMigrateFile) SyncSequenceFile() error

func (*StructMigrateFile) SyncStructFile added in v0.0.7

func (s *StructMigrateFile) SyncStructFile() error

type StructMigrateRule added in v0.0.7

type StructMigrateRule struct {
	Ctx                      context.Context                  `json:"-"`
	TaskName                 string                           `json:"taskName"`
	TaskMode                 string                           `json:"taskMode"`
	TaskFlow                 string                           `json:"taskFlow"`
	SchemaNameS              string                           `json:"schemaNameS"`
	TableNameS               string                           `json:"tableNameS"`
	TablePrimaryAttrs        []map[string]string              `json:"tablePrimaryAttrs"`
	TableColumnsAttrs        []map[string]string              `json:"tableColumnsAttrs"`
	TableCommentAttrs        []map[string]string              `json:"tableCommentAttrs"`
	TableCharsetAttr         string                           `json:"tableCharsetAttr"`
	TableCollationAttr       string                           `json:"tableCollationAttr"`
	CaseFieldRuleT           string                           `json:"caseFieldRuleT"`
	CreateIfNotExist         bool                             `json:"createIfNotExist"`
	DBVersionS               string                           `json:"dbVersionS"`
	DBCharsetS               string                           `json:"dbCharsetS"` // the string data charset conversation
	DBCharsetT               string                           `json:"dbCharsetT"` // the string data charset conversation
	BuildinDatatypeRules     []*buildin.BuildinDatatypeRule   `json:"-"`
	BuildinDefaultValueRules []*buildin.BuildinDefaultvalRule `json:"-"`
}

func (*StructMigrateRule) GetCaseFieldRule added in v0.0.7

func (r *StructMigrateRule) GetCaseFieldRule() string

func (*StructMigrateRule) GetCreatePrefixRule added in v0.0.7

func (r *StructMigrateRule) GetCreatePrefixRule() string

func (*StructMigrateRule) GetSchemaNameRule added in v0.0.7

func (r *StructMigrateRule) GetSchemaNameRule() (map[string]string, error)

func (*StructMigrateRule) GetTableAttributesRule added in v0.0.7

func (r *StructMigrateRule) GetTableAttributesRule() (string, error)

func (*StructMigrateRule) GetTableCharsetRule added in v0.0.15

func (r *StructMigrateRule) GetTableCharsetRule() (string, error)

func (*StructMigrateRule) GetTableCollationRule added in v0.0.15

func (r *StructMigrateRule) GetTableCollationRule() (string, error)

func (*StructMigrateRule) GetTableColumnCharsetRule added in v0.0.15

func (r *StructMigrateRule) GetTableColumnCharsetRule() (map[string]string, error)

func (*StructMigrateRule) GetTableColumnCollationRule added in v0.0.7

func (r *StructMigrateRule) GetTableColumnCollationRule() (map[string]string, error)

func (*StructMigrateRule) GetTableColumnCommentRule added in v0.0.7

func (r *StructMigrateRule) GetTableColumnCommentRule() (map[string]string, error)

func (*StructMigrateRule) GetTableColumnRule added in v0.0.7

func (r *StructMigrateRule) GetTableColumnRule() (map[string]string, map[string]string, map[string]string, error)

GetTableColumnRule used for get custom table column rule column datatype rule priority: - column level - table level - task level - default level

func (*StructMigrateRule) GetTableCommentRule added in v0.0.7

func (r *StructMigrateRule) GetTableCommentRule() (string, error)

func (*StructMigrateRule) GetTableNameRule added in v0.0.7

func (r *StructMigrateRule) GetTableNameRule() (map[string]string, error)

func (*StructMigrateRule) String added in v0.0.7

func (r *StructMigrateRule) String() string

type StructMigrateTable added in v0.0.9

type StructMigrateTable struct {
	TaskName            string                                `json:"taskName"`
	TaskFlow            string                                `json:"taskFlow"`
	DatasourceS         *Datasource                           `json:"datasourceS"`
	TableAttributes     *database.StructMigrateAttributes     `json:"tableAttributes"`
	TableAttributesRule *database.StructMigrateAttributesRule `json:"tableAttributesRule"`
	DBCharsetT          string                                `json:"DBCharsetT"`
}

func (*StructMigrateTable) GenSchemaNameS added in v0.0.9

func (t *StructMigrateTable) GenSchemaNameS() string

func (*StructMigrateTable) GenSchemaNameT added in v0.0.9

func (t *StructMigrateTable) GenSchemaNameT() (string, error)

func (*StructMigrateTable) GenTableCheckKey added in v0.0.9

func (t *StructMigrateTable) GenTableCheckKey() ([]string, error)

func (*StructMigrateTable) GenTableColumnComment added in v0.0.9

func (t *StructMigrateTable) GenTableColumnComment() ([]string, error)

func (*StructMigrateTable) GenTableColumns added in v0.0.9

func (t *StructMigrateTable) GenTableColumns() ([]string, error)

func (*StructMigrateTable) GenTableComment added in v0.0.9

func (t *StructMigrateTable) GenTableComment() (string, error)

func (*StructMigrateTable) GenTableCreatePrefixT added in v0.0.9

func (t *StructMigrateTable) GenTableCreatePrefixT() string

func (*StructMigrateTable) GenTableForeignKey added in v0.0.9

func (t *StructMigrateTable) GenTableForeignKey() ([]string, error)

func (*StructMigrateTable) GenTableNameS added in v0.0.9

func (t *StructMigrateTable) GenTableNameS() string

func (*StructMigrateTable) GenTableNameT added in v0.0.9

func (t *StructMigrateTable) GenTableNameT() (string, error)

func (*StructMigrateTable) GenTableNormalIndex added in v0.0.9

func (t *StructMigrateTable) GenTableNormalIndex() ([]string, []string, error)

func (*StructMigrateTable) GenTableOriginDDlS added in v0.0.9

func (t *StructMigrateTable) GenTableOriginDDlS() string

func (*StructMigrateTable) GenTablePrimaryKey added in v0.0.9

func (t *StructMigrateTable) GenTablePrimaryKey() (string, error)

func (*StructMigrateTable) GenTableSuffix added in v0.0.9

func (t *StructMigrateTable) GenTableSuffix() (string, error)

func (*StructMigrateTable) GenTableTypeS added in v0.0.9

func (t *StructMigrateTable) GenTableTypeS() string

func (*StructMigrateTable) GenTableUniqueIndex added in v0.0.9

func (t *StructMigrateTable) GenTableUniqueIndex() ([]string, []string, error)

func (*StructMigrateTable) GenTableUniqueKey added in v0.0.9

func (t *StructMigrateTable) GenTableUniqueKey() ([]string, error)

func (*StructMigrateTable) String added in v0.0.9

func (t *StructMigrateTable) String() string

type StructMigrateTask added in v0.0.9

type StructMigrateTask struct {
	Ctx                      context.Context
	Task                     *task.Task
	SchemaNameS              string
	SchemaNameT              string
	DatabaseS                database.IDatabase
	DatabaseT                database.IDatabase
	DBTypeS                  string
	DBVersionS               string
	DBCharsetS               string
	DBCharsetT               string
	StartTime                time.Time
	BuildInDatatypeRules     []*buildin.BuildinDatatypeRule
	BuildInDefaultValueRules []*buildin.BuildinDefaultvalRule
	TaskParams               *pb.StructMigrateParam

	StructReadyInit   chan bool
	SequenceReadyInit chan bool
}

func (*StructMigrateTask) Init added in v0.0.9

func (st *StructMigrateTask) Init() error

func (*StructMigrateTask) Last added in v0.0.11

func (st *StructMigrateTask) Last() error

func (*StructMigrateTask) Resume added in v0.0.9

func (st *StructMigrateTask) Resume() error

func (*StructMigrateTask) Run added in v0.0.9

func (st *StructMigrateTask) Run() error

type Table

type Table struct {
	TaskName string
	TaskFlow string
	Source   *structure.Table
	Target   *structure.Table
}

Table structure comparison Based on the upstream Oracle table structure information, compare the downstream MySQL table structure 1. If the upstream exists and the downstream does not exist, the record will be output. If the upstream does not exist and the downstream exists, the record will not be output by default. 2. Ignore the comparison of different index names and constraint names between the upstream and downstream, and only compare whether the same fields exist under the same constraints downstream. 3. Partitions only compare partition types, partition keys, partition expressions, etc., and do not compare the specific conditions of each partition.

func (*Table) ComparePartitionTableType

func (t *Table) ComparePartitionTableType() string

func (*Table) CompareTableCharsetCollation

func (t *Table) CompareTableCharsetCollation() string

func (*Table) CompareTableCheckConstraint

func (t *Table) CompareTableCheckConstraint() (string, error)

func (*Table) CompareTableColumnCharsetCollation

func (t *Table) CompareTableColumnCharsetCollation() string

func (*Table) CompareTableColumnCounts

func (t *Table) CompareTableColumnCounts() string

func (*Table) CompareTableColumnDetail

func (t *Table) CompareTableColumnDetail() (string, error)

func (*Table) CompareTableComment

func (t *Table) CompareTableComment() string

func (*Table) CompareTableForeignConstraint

func (t *Table) CompareTableForeignConstraint() (string, error)

func (*Table) CompareTableIndexCounts added in v0.0.12

func (t *Table) CompareTableIndexCounts() string

func (*Table) CompareTableIndexDetail

func (t *Table) CompareTableIndexDetail() (string, error)

func (*Table) CompareTablePartitionDetail

func (t *Table) CompareTablePartitionDetail() (string, error)

func (*Table) CompareTablePrimaryConstraint

func (t *Table) CompareTablePrimaryConstraint() (string, error)

func (*Table) CompareTableUniqueConstraint

func (t *Table) CompareTableUniqueConstraint() (string, error)

type WaitingRecs added in v0.0.9

type WaitingRecs struct {
	TaskName    string
	SchemaNameS string
	TableNameS  string
	SchemaNameT string
	TableNameT  string
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL