profiles

package
v1.0.1-0...-1811ebb Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 28, 2023 License: Apache-2.0 Imports: 13 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	SourceProfileTypeUnset = iota
	SourceProfileTypeFile
	SourceProfileTypeConnection
	SourceProfileTypeConfig
	SourceProfileTypeCsv
)
View Source 
const (
	SourceProfileConnectionTypeUnset = iota
	SourceProfileConnectionTypeMySQL
	SourceProfileConnectionTypePostgreSQL
	SourceProfileConnectionTypeDynamoDB
	SourceProfileConnectionTypeSqlServer
	SourceProfileConnectionTypeOracle
)
View Source 
const (
	TargetProfileTypeUnset = iota
	TargetProfileTypeConnection
)
View Source 
const (
	TargetProfileConnectionTypeUnset = iota
	TargetProfileConnectionTypeSpanner
)

Variables

This section is empty.

Functions

func GenerateMYSQLConnectionStr 

func GenerateMYSQLConnectionStr() (string, error)

func GeneratePGSQLConnectionStr 

func GeneratePGSQLConnectionStr() (string, error)

func GetSQLConnectionStr 

func GetSQLConnectionStr(sourceProfile SourceProfile) string

func GetSchemaSampleSize 

func GetSchemaSampleSize(sourceProfile SourceProfile) int64

func ParseMap 

func ParseMap(s string) (map[string]string, error)

Parses input string `s` as a map of key-value pairs. It's expected that the input string is of the form "key1=value1,key2=value2,..." etc. Return error otherwise.

Types

type DataShard 

type DataShard struct {
	DataShardId          string                `json:"dataShardId"`
	SrcConnectionProfile DatastreamConnProfile `json:"srcConnectionProfile"`
	DstConnectionProfile DatastreamConnProfile `json:"dstConnectionProfile"`
	DataflowConfig       DataflowConfig        `json:"dataflowConfig"`
	TmpDir               string                `json:"tmpDir"`
	StreamLocation       string                `json:"streamLocation"`
	LogicalShards        []LogicalShard        `json:"databases"`
}

type DataflowConfig 

type DataflowConfig struct {
	Location      string `json:"location"`
	Network       string `json:"network"`
	Subnetwork    string `json:"subnetwork"`
	HostProjectId string `json:"hostProjectId"`
}

type DatastreamConnProfile 

type DatastreamConnProfile struct {
	Name     string `json:"name"`
	Location string `json:"location"`
}

type DirectConnectionConfig 

type DirectConnectionConfig struct {
	DataShardId string `json:"dataShardId"`
	Host        string `json:"host"`
	User        string `json:"user"`
	Password    string `json:"password"`
	Port        string `json:"port"`
	DbName      string `json:"dbName"`
}

type LogicalShard 

type LogicalShard struct {
	DbName         string `json:"dbName"`
	LogicalShardId string `json:"databaseId"`
	RefDataShardId string `json:"refDataShardId"`
}

type ShardConfigurationBulk 

type ShardConfigurationBulk struct {
	SchemaSource DirectConnectionConfig   `json:"schemaSource"`
	DataShards   []DirectConnectionConfig `json:"dataShards"`
}

type ShardConfigurationDMS 

type ShardConfigurationDMS struct {
}

TODO: Define the sharding structure for DMS migrations here.

type ShardConfigurationDataflow 

type ShardConfigurationDataflow struct {
	SchemaSource   DirectConnectionConfig `json:"schemaSource"`
	DataShards     []*DataShard           `json:"dataShards"`
	DataflowConfig DataflowConfig         `json:"dataflowConfig"`
}

type SourceProfile 

type SourceProfile struct {
	Driver string
	Ty     SourceProfileType
	File   SourceProfileFile
	Conn   SourceProfileConnection
	Config SourceProfileConfig
	Csv    SourceProfileCsv
}

func NewSourceProfile 

func NewSourceProfile(s string, source string) (SourceProfile, error)

Flag source-profile is passed as a list of key value pairs on the command line. Following 3 formats are supported as a valid source-profile.

Format 1. Specify file path and file format. File path can be a local file path or a gcs file path. Support for more file path types can be added in future. File format can be "dump" e.g., when specifying a mysqldump or pgdump etc. Support for more formats e.g., "csv", "avro" etc can be added in future.

Example: -source-profile="file=/tmp/abc, format=dump" Example: -source-profile="file=gcs://bucket_name/cart.txt, format=dump"

Format 2. Specify source connection parameters. If none specified, then read from envrironment variables.

Format 3. Specify a config file that specifies source connection profile.

func (SourceProfile) ToLegacyDriver 

func (src SourceProfile) ToLegacyDriver(source string) (string, error)

ToLegacyDriver converts source-profile to equivalent legacy global flags e.g., -driver, -dump-file etc since the rest of the codebase still uses the same. TODO: Deprecate this function and pass around SourceProfile across the codebase wherever information about source connection is required.

func (SourceProfile) UseTargetSchema 

func (src SourceProfile) UseTargetSchema() bool

UseTargetSchema returns true if the driver expects an existing schema to use in the target database.

type SourceProfileConfig 

type SourceProfileConfig struct {
	ConfigType                 string                     `json:"configType"`
	ShardConfigurationBulk     ShardConfigurationBulk     `json:"shardConfigurationBulk"`
	ShardConfigurationDataflow ShardConfigurationDataflow `json:"shardConfigurationDataflow"`
	ShardConfigurationDMS      ShardConfigurationDMS      `json:"shardConfigurationDMS"`
}

func NewSourceProfileConfig 

func NewSourceProfileConfig(source string, path string) (SourceProfileConfig, error)

type SourceProfileConnection 

type SourceProfileConnection struct {
	Ty        SourceProfileConnectionType
	Streaming bool
	Mysql     SourceProfileConnectionMySQL
	Pg        SourceProfileConnectionPostgreSQL
	Dydb      SourceProfileConnectionDynamoDB
	SqlServer SourceProfileConnectionSqlServer
	Oracle    SourceProfileConnectionOracle
}

func NewSourceProfileConnection 

func NewSourceProfileConnection(source string, params map[string]string) (SourceProfileConnection, error)

type SourceProfileConnectionDynamoDB 

type SourceProfileConnectionDynamoDB struct {
	// These connection params are not used currently because the SDK reads directly from the env variables.
	// These are still kept around as reference when we refactor passing
	// SourceProfile instead of sqlConnectionStr around.
	AwsAccessKeyID     string // Same as AWS_ACCESS_KEY_ID environment variable
	AwsSecretAccessKey string // Same as AWS_SECRET_ACCESS_KEY environment variable
	AwsRegion          string // Same as AWS_REGION environment variable
	DydbEndpoint       string // Same as DYNAMODB_ENDPOINT_OVERRIDE environment variable
	SchemaSampleSize   int64  // Number of rows to use for inferring schema (default 100,000)
	// contains filtered or unexported fields
}

func NewSourceProfileConnectionDynamoDB 

func NewSourceProfileConnectionDynamoDB(params map[string]string) (SourceProfileConnectionDynamoDB, error)

type SourceProfileConnectionMySQL 

type SourceProfileConnectionMySQL struct {
	Host            string // Same as MYSQLHOST environment variable
	Port            string // Same as MYSQLPORT environment variable
	User            string // Same as MYSQLUSER environment variable
	Db              string // Same as MYSQLDATABASE environment variable
	Pwd             string // Same as MYSQLPWD environment variable
	StreamingConfig string
}

func NewSourceProfileConnectionMySQL 

func NewSourceProfileConnectionMySQL(params map[string]string) (SourceProfileConnectionMySQL, error)

type SourceProfileConnectionOracle 

type SourceProfileConnectionOracle struct {
	Host            string
	Port            string
	User            string
	Db              string
	Pwd             string
	StreamingConfig string
}

func NewSourceProfileConnectionOracle 

func NewSourceProfileConnectionOracle(params map[string]string) (SourceProfileConnectionOracle, error)

type SourceProfileConnectionPostgreSQL 

type SourceProfileConnectionPostgreSQL struct {
	Host            string // Same as PGHOST environment variable
	Port            string // Same as PGPORT environment variable
	User            string // Same as PGUSER environment variable
	Db              string // Same as PGDATABASE environment variable
	Pwd             string // Same as PGPASSWORD environment variable
	StreamingConfig string
}

func NewSourceProfileConnectionPostgreSQL 

func NewSourceProfileConnectionPostgreSQL(params map[string]string) (SourceProfileConnectionPostgreSQL, error)

type SourceProfileConnectionSqlServer 

type SourceProfileConnectionSqlServer struct {
	Host string
	Port string
	User string
	Db   string
	Pwd  string
}

func NewSourceProfileConnectionSqlServer 

func NewSourceProfileConnectionSqlServer(params map[string]string) (SourceProfileConnectionSqlServer, error)

type SourceProfileConnectionType 

type SourceProfileConnectionType int

type SourceProfileCsv 

type SourceProfileCsv struct {
	Manifest  string
	Delimiter string
	NullStr   string
}

func NewSourceProfileCsv 

func NewSourceProfileCsv(params map[string]string) SourceProfileCsv

type SourceProfileFile 

type SourceProfileFile struct {
	Path   string
	Format string
}

func NewSourceProfileFile 

func NewSourceProfileFile(params map[string]string) SourceProfileFile

type SourceProfileType 

type SourceProfileType int

type TargetProfile 

type TargetProfile struct {
	Ty   TargetProfileType
	Conn TargetProfileConnection
}

func NewTargetProfile 

func NewTargetProfile(s string) (TargetProfile, error)

Target profile is passed as a list of key value pairs on the command line. Today we support only direct connection as a valid target profile type, but in future we can support writing to CSV or AVRO as valid targets.

Among direct connection targets, today we only support Spanner database. TargetProfileConnectionType can be extended to add more databases. Users can specify the database dialect, instance, database name etc when connecting to Spanner.

Database dialect can take 2 values: GoogleSQL or PostgreSQL and the same correspond to regular Cloud Spanner database and PG Cloud Spanner database respectively.

If dbName is not specified, then HarbourBridge will autogenerate the same and create a database with the same name.

Example: -target-profile="instance=my-instance1,dbName=my-new-db1" Example: -target-profile="instance=my-instance1,dbName=my-new-db1,dialect=PostgreSQL"

func (TargetProfile) FetchTargetDialect 

func (trg TargetProfile) FetchTargetDialect(ctx context.Context) (string, error)

This expects that GetResourceIds has already been called once and the project, instance and dbName fields in target profile are populated.

func (*TargetProfile) GetResourceIds 

func (targetProfile *TargetProfile) GetResourceIds(ctx context.Context, now time.Time, driverName string, out *os.File) (string, string, string, error)

type TargetProfileConnection 

type TargetProfileConnection struct {
	Ty TargetProfileConnectionType
	Sp TargetProfileConnectionSpanner
}

type TargetProfileConnectionSpanner 

type TargetProfileConnectionSpanner struct {
	Endpoint string // Same as SPANNER_API_ENDPOINT environment variable
	Project  string // Same as GCLOUD_PROJECT environment variable
	Instance string
	Dbname   string
	Dialect  string
}

type TargetProfileConnectionType 

type TargetProfileConnectionType int

type TargetProfileType 

type TargetProfileType int

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.