vreplication

package
v0.20.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 3, 2024 License: Apache-2.0 Imports: 64 Imported by: 3

Documentation

Index

Constants

View Source 
const (

	// LogStreamCreate is used when a row in the vreplication table is inserted via VReplicationExec.
	LogStreamCreate = "Stream Created"
	// LogStreamUpdate is used when a row in the vreplication table is updated via VReplicationExec.
	LogStreamUpdate = "Stream Updated"
	// LogStreamDelete is used when a row in the vreplication table is deleted via VReplicationExec.
	LogStreamDelete = "Stream Deleted"
	// LogMessage is used for generic log messages.
	LogMessage = "Message"
	// LogCopyStart is used when the copy phase is started.
	LogCopyStart = "Started Copy Phase"
	// LogCopyEnd is used when the copy phase is done.
	LogCopyEnd = "Ended Copy Phase"
	// LogStateChange is used when the state of the stream changes.
	LogStateChange = "State Changed"

	// LogError indicates that there is an error from which we cannot recover and the operator needs to intervene.
	LogError = "Error"
)
View Source 
const (

	// SQLMode should be used whenever performing a schema change as part of a vreplication
	// workflow to ensure that you set a permissive SQL mode as defined by
	// VReplication. We follow MySQL's model for recreating database objects
	// on a target -- using SQL statements generated from a source -- which
	// ensures that we can recreate them regardless of the sql_mode that was
	// in effect on the source when it was created:
	//   https://github.com/mysql/mysql-server/blob/3290a66c89eb1625a7058e0ef732432b6952b435/client/mysqldump.cc#L795-L818
	SQLMode       = "NO_AUTO_VALUE_ON_ZERO"
	StrictSQLMode = "STRICT_ALL_TABLES,NO_AUTO_VALUE_ON_ZERO"
)
View Source 
const AllowUnsafeWriteCommentDirective = "ALLOW_UNSAFE_VREPLICATION_WRITE"

A comment directive that you can include in your VReplication write statements if you want to bypass the safety checks that ensure you are being selective. The full comment directive looks like this: delete /*vt+ ALLOW_UNSAFE_VREPLICATION_WRITE */ from _vt.vreplication

View Source 
const ExcludeStr = "exclude"

ExcludeStr is the filter value for excluding tables that match a rule. TODO(sougou): support this on vstreamer side also.

View Source 
const IncludeNonServingTabletsCommentDirective = "INCLUDE_NON_SERVING_TABLETS_IN_PLAN"

A comment directive that you need to include in your VReplication statements if you want the controller to include non-serving tablets in the execution plan (via tablet picker options). The full comment directive looks like this: update /*vt+ INCLUDE_NON_SERVING_TABLETS_IN_PLAN=1 */ _vt.vreplication set ...

View Source 
const NoForeignKeyCheckFlagBitmask uint32 = 1 << 1

NoForeignKeyCheckFlagBitmask is the bitmask for the 2nd bit (least significant) of the flags in a binlog row event. This bit is set if foreign key checks are disabled.

Variables

This section is empty.

Functions

func AddStatusPart 

func AddStatusPart()

AddStatusPart adds the vreplication status to the status page.

func MatchTable 

func MatchTable(tableName string, filter *binlogdatapb.Filter) (*binlogdatapb.Rule, error)

MatchTable is similar to tableMatches and buildPlan defined in vstreamer/planbuilder.go.

func StatusSummary 

func StatusSummary() (maxReplicationLagSeconds int64, binlogPlayersCount int32)

StatusSummary returns the summary status of vreplication.

Types

type ColumnInfo added in v0.11.0 

type ColumnInfo struct {
	Name        string
	CharSet     string
	Collation   string
	DataType    string
	ColumnType  string
	IsPK        bool
	IsGenerated bool
}

ColumnInfo is used to store charset and collation

type ControllerStatus 

type ControllerStatus struct {
	Index                 int32
	Source                string
	SourceShard           string
	StopPosition          string
	LastPosition          string
	Heartbeat             int64
	ReplicationLagSeconds int64
	Counts                map[string]int64
	Rates                 map[string][]float64
	State                 string
	SourceTablet          *topodatapb.TabletAlias
	Messages              []string
	QueryCounts           map[string]int64
	BulkQueryCounts       map[string]int64
	TrxQueryBatchCounts   map[string]int64
	PhaseTimings          map[string]int64
	CopyRowCount          int64
	CopyLoopCount         int64
	NoopQueryCounts       map[string]int64
	TableCopyTimings      map[string]int64
}

ControllerStatus contains a renderable status of a controller.

type Engine 

type Engine struct {
	// contains filtered or unexported fields
}

Engine is the engine for handling vreplication.

func NewEngine 

func NewEngine(env *vtenv.Environment, config *tabletenv.TabletConfig, ts *topo.Server, cell string, mysqld mysqlctl.MysqlDaemon, lagThrottler *throttle.Throttler) *Engine

NewEngine creates a new Engine. A nil ts means that the Engine is disabled.

func NewSimpleTestEngine added in v0.16.0 

func NewSimpleTestEngine(ts *topo.Server, cell string, mysqld mysqlctl.MysqlDaemon, dbClientFactoryFiltered func() binlogplayer.DBClient, dbClientFactoryDba func() binlogplayer.DBClient, dbname string, externalConfig map[string]*dbconfigs.DBConfigs) *Engine

NewSimpleTestEngine creates a new Engine for testing that can also short circuit functions as needed.

func NewTestEngine 

func NewTestEngine(ts *topo.Server, cell string, mysqld mysqlctl.MysqlDaemon, dbClientFactoryFiltered func() binlogplayer.DBClient, dbClientFactoryDba func() binlogplayer.DBClient, dbname string, externalConfig map[string]*dbconfigs.DBConfigs) *Engine

NewTestEngine creates a new Engine for testing.

func (*Engine) Close 

func (vre *Engine) Close()

Close closes the Engine service.

func (*Engine) Exec 

func (vre *Engine) Exec(query string) (*sqltypes.Result, error)

Exec runs the specified query as the Filtered user.

func (*Engine) ExecWithDBA added in v0.11.0 

func (vre *Engine) ExecWithDBA(query string) (*sqltypes.Result, error)

ExecWithDBA runs the specified query as the DBA user.

func (*Engine) InitDBConfig 

func (vre *Engine) InitDBConfig(dbcfgs *dbconfigs.DBConfigs)

InitDBConfig should be invoked after the db name is computed.

func (*Engine) IsOpen 

func (vre *Engine) IsOpen() bool

IsOpen returns true if Engine is open.

func (*Engine) Open 

func (vre *Engine) Open(ctx context.Context)

Open starts the Engine service.

func (*Engine) WaitForPos 

func (vre *Engine) WaitForPos(ctx context.Context, id int32, pos string) error

WaitForPos waits for the replication to reach the specified position.

type EngineStatus 

type EngineStatus struct {
	IsOpen      bool
	Controllers []*ControllerStatus
}

EngineStatus contains a renderable status of the Engine.

type HTTPStreamWriterMock 

type HTTPStreamWriterMock struct {
	// contains filtered or unexported fields
}

HTTPStreamWriterMock implements http.ResponseWriter and adds a channel to sync writes and reads

func NewHTTPStreamWriterMock 

func NewHTTPStreamWriterMock() *HTTPStreamWriterMock

NewHTTPStreamWriterMock returns a new HTTPStreamWriterMock

func (*HTTPStreamWriterMock) Flush 

func (w *HTTPStreamWriterMock) Flush()

Flush sends buffered data to the channel

func (*HTTPStreamWriterMock) Header 

func (w *HTTPStreamWriterMock) Header() http.Header

Header is a stub

func (*HTTPStreamWriterMock) Write 

func (w *HTTPStreamWriterMock) Write(data []byte) (int, error)

Write buffers sent data

func (*HTTPStreamWriterMock) WriteHeader 

func (w *HTTPStreamWriterMock) WriteHeader(statuscode int)

WriteHeader is a stub

type InsertGenerator 

type InsertGenerator struct {
	// contains filtered or unexported fields
}

InsertGenerator generates a vreplication insert statement.

func NewInsertGenerator 

func NewInsertGenerator(state binlogdatapb.VReplicationWorkflowState, dbname string) *InsertGenerator

NewInsertGenerator creates a new InsertGenerator.

func (*InsertGenerator) AddRow 

func (ig *InsertGenerator) AddRow(workflow string, bls *binlogdatapb.BinlogSource, pos, cell, tabletTypes string,
	workflowType binlogdatapb.VReplicationWorkflowType, workflowSubType binlogdatapb.VReplicationWorkflowSubType, deferSecondaryKeys bool)

AddRow adds a row to the insert statement.

func (*InsertGenerator) String 

func (ig *InsertGenerator) String() string

String returns the generated statement.

type PostCopyAction added in v0.16.0 

type PostCopyAction struct {
	Type PostCopyActionType `json:"type"`
	Task string             `json:"task"`
}

type PostCopyActionType added in v0.16.0 

type PostCopyActionType int
const (
	PostCopyActionNone PostCopyActionType = iota
	PostCopyActionSQL
)

type ReplicaConnector added in v0.13.0 

type ReplicaConnector struct {
	// contains filtered or unexported fields
}

func NewReplicaConnector 

func NewReplicaConnector(venv *vtenv.Environment, connParams *mysql.ConnParams) *ReplicaConnector

NewReplicaConnector returns replica connector

This is used by binlog server to make vstream connection using the vstream connection, it will parse the events from binglog to fetch the corresponding GTID for required recovery time

func (*ReplicaConnector) Close added in v0.13.0 

func (c *ReplicaConnector) Close() error

func (*ReplicaConnector) VStream added in v0.13.0 

func (c *ReplicaConnector) VStream(ctx context.Context, startPos string, filter *binlogdatapb.Filter, send func([]*binlogdatapb.VEvent) error) error

type ReplicatorPlan 

type ReplicatorPlan struct {
	VStreamFilter *binlogdatapb.Filter
	TargetTables  map[string]*TablePlan
	TablePlans    map[string]*TablePlan
	ColInfoMap    map[string][]*ColumnInfo

	Source *binlogdatapb.BinlogSource
	// contains filtered or unexported fields
}

ReplicatorPlan is the execution plan for the replicator. It contains plans for all the tables it's replicating. Every phase of vreplication builds its own instance of the ReplicatorPlan. This is because the plan depends on copyState, which changes on every iteration. The table plans within ReplicatorPlan will not be fully populated because all the information is not available initially. For simplicity, the ReplicatorPlan is immutable. Once we get the field info for a table from the stream response, we'll have all the necessary info to build the final plan. At that time, buildExecutionPlan is invoked, which will make a copy of the TablePlan from ReplicatorPlan, and fill the rest of the members, leaving the original plan unchanged. The constructor is buildReplicatorPlan in table_plan_builder.go

func (*ReplicatorPlan) MarshalJSON 

func (rp *ReplicatorPlan) MarshalJSON() ([]byte, error)

MarshalJSON performs a custom JSON Marshalling.

type ShardSorter 

type ShardSorter []string

ShardSorter implements a sort.Sort() function for sorting shard ranges

func (ShardSorter) Key 

func (s ShardSorter) Key(ind int) string

Key returns the prefix of a shard range

func (ShardSorter) Len 

func (s ShardSorter) Len() int

Len implements the required interface for a sorting function

func (ShardSorter) Less 

func (s ShardSorter) Less(i, j int) bool

Less implements the required interface for a sorting function

func (ShardSorter) Swap 

func (s ShardSorter) Swap(i, j int)

Swap implements the required interface for a sorting function

type TablePlan 

type TablePlan struct {
	// TargetName, SendRule will always be initialized.
	TargetName string
	SendRule   *binlogdatapb.Rule
	// Lastpk will be initialized if it was specified, and
	// will be used for building the final plan after field info
	// is received.
	Lastpk *sqltypes.Result
	// BulkInsertFront, BulkInsertValues and BulkInsertOnDup are used
	// by vcopier. These three parts are combined to build bulk insert
	// statements. This is functionally equivalent to generating
	// multiple statements using the "Insert" construct, but much more
	// efficient for the copy phase.
	BulkInsertFront  *sqlparser.ParsedQuery
	BulkInsertValues *sqlparser.ParsedQuery
	BulkInsertOnDup  *sqlparser.ParsedQuery
	// Insert, Update and Delete are used by vplayer.
	// If the plan is an insertIgnore type, then Insert
	// and Update contain 'insert ignore' statements and
	// Delete is nil.
	Insert           *sqlparser.ParsedQuery
	Update           *sqlparser.ParsedQuery
	Delete           *sqlparser.ParsedQuery
	MultiDelete      *sqlparser.ParsedQuery
	Fields           []*querypb.Field
	ConvertIntToEnum map[string]bool
	// PKReferences is used to check if an event changed
	// a primary key column (row move).
	PKReferences []string
	// PKIndices is an array, length = #columns, true if column is part of the PK
	PKIndices               []bool
	Stats                   *binlogplayer.Stats
	FieldsToSkip            map[string]bool
	ConvertCharset          map[string](*binlogdatapb.CharsetConversion)
	HasExtraSourcePkColumns bool

	TablePlanBuilder *tablePlanBuilder
	// PartialInserts is a dynamically generated cache of insert ParsedQueries, which update only some columns.
	// This is when we use a binlog_row_image which is not "full". The key is a serialized bitmap of data columns
	// which are sent as part of the RowEvent.
	PartialInserts map[string]*sqlparser.ParsedQuery
	// PartialUpdates are same as PartialInserts, but for update statements
	PartialUpdates map[string]*sqlparser.ParsedQuery

	CollationEnv *collations.Environment
}

TablePlan is the execution plan for a table within a replicator. If the column names are not known at the time of plan building (like select *), then only TargetName, SendRule and Lastpk are initialized. When the stream returns the field info, those are used as column names to build the final plan. Lastpk comes from copyState. If it's set, then the generated plans are significantly different because any events that fall beyond Lastpk must be excluded. If column names were known upfront, then all fields of TablePlan are built except for Fields. This member is populated only after the field info is received from the stream. The ParsedQuery objects assume that a map of before and after values will be built based on the streaming rows. Before image values will be prefixed with a "b_", and after image values will be prefixed with a "a_". The TablePlan structure is used during all the phases of vreplication: catchup, copy, fastforward, or regular replication.

func (*TablePlan) MarshalJSON 

func (tp *TablePlan) MarshalJSON() ([]byte, error)

MarshalJSON performs a custom JSON Marshalling.

type VStreamerClient 

type VStreamerClient interface {
	Open(context.Context) error
	Close(context.Context) error

	// VStream streams VReplication events based on the specified filter.
	VStream(ctx context.Context, startPos string, tablePKs []*binlogdatapb.TableLastPK, filter *binlogdatapb.Filter, send func([]*binlogdatapb.VEvent) error) error

	// VStreamRows streams rows of a table from the specified starting point.
	VStreamRows(ctx context.Context, query string, lastpk *querypb.QueryResult, send func(*binlogdatapb.VStreamRowsResponse) error) error

	// VStreamTables streams rows of a table from the specified starting point.
	VStreamTables(ctx context.Context, send func(*binlogdatapb.VStreamTablesResponse) error) error
}

VStreamerClient exposes the core interface of a vstreamer

type VrLogStats 

type VrLogStats struct {
	Type       string
	Detail     string
	StartTime  time.Time
	LogTime    string
	DurationNs int64
}

VrLogStats collects attributes of a vreplication event for logging

func NewVrLogStats 

func NewVrLogStats(eventType string) *VrLogStats

NewVrLogStats should be called at the start of the event to be logged

func (*VrLogStats) Send 

func (stats *VrLogStats) Send(detail string)

Send records the log event, should be called on a stats object constructed by NewVrLogStats()

Source Files

View all Source files

Directories

Path Synopsis
Package queryhistory provides tools for verifying that a SQL statement history conforms to a set of expectations.
 Package queryhistory provides tools for verifying that a SQL statement history conforms to a set of expectations.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.