vreplication

package
v1.0.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 25, 2020 License: Apache-2.0 Imports: 44 Imported by: 0

Documentation

Index

Constants

View Source 
const ExcludeStr = "exclude"

ExcludeStr is the filter value for excluding tables that match a rule. TODO(sougou): support this on vstreamer side also.

Variables

This section is empty.

Functions

func AddStatusPart 

func AddStatusPart()

AddStatusPart adds the vreplication status to the status page.

func InitVStreamerClient added in v1.0.1 

func InitVStreamerClient(cfg *dbconfigs.DBConfigs)

InitVStreamerClient initializes config for vstreamer client

func MatchTable added in v1.0.1 

func MatchTable(tableName string, filter *binlogdatapb.Filter) (*binlogdatapb.Rule, error)

MatchTable is similar to tableMatches and buildPlan defined in vstreamer/planbuilder.go.

func StatusSummary 

func StatusSummary() (maxSecondsBehindMaster int64, binlogPlayersCount int32)

StatusSummary returns the summary status of vreplication.

Types

type ControllerStatus 

type ControllerStatus struct {
	Index               uint32
	Source              string
	SourceShard         string
	StopPosition        string
	LastPosition        string
	SecondsBehindMaster int64
	Counts              map[string]int64
	Rates               map[string][]float64
	State               string
	SourceTablet        string
	Messages            []string
}

ControllerStatus contains a renderable status of a controller.

type Engine 

type Engine struct {
	// contains filtered or unexported fields
}

Engine is the engine for handling vreplication.

func NewEngine 

func NewEngine(ts *topo.Server, cell string, mysqld mysqlctl.MysqlDaemon, dbClientFactory func() binlogplayer.DBClient, dbName string) *Engine

NewEngine creates a new Engine. A nil ts means that the Engine is disabled.

func (*Engine) Close 

func (vre *Engine) Close()

Close closes the Engine service.

func (*Engine) Exec 

func (vre *Engine) Exec(query string) (*sqltypes.Result, error)

Exec executes the query and the related actions. Example insert statement: insert into _vt.vreplication 

(workflow, source, pos, max_tps, max_replication_lag, time_updated, transaction_timestamp, state)
values ('Resharding', 'keyspace:"ks" shard:"0" tables:"a" tables:"b" ', 'MariaDB/0-1-1083', 9223372036854775807, 9223372036854775807, 481823, 0, 'Running')`

Example update statement: update _vt.vreplication set state='Stopped', message='testing stop' where id=1 Example delete: delete from _vt.vreplication where id=1 Example select: select * from _vt.vreplication

func (*Engine) IsOpen 

func (vre *Engine) IsOpen() bool

IsOpen returns true if Engine is open.

func (*Engine) Open 

func (vre *Engine) Open(ctx context.Context) error

Open starts the Engine service.

func (*Engine) WaitForPos 

func (vre *Engine) WaitForPos(ctx context.Context, id int, pos string) error

WaitForPos waits for the replication to reach the specified position.

type EngineStatus 

type EngineStatus struct {
	IsOpen      bool
	Controllers []*ControllerStatus
}

EngineStatus contains a renderable status of the Engine.

type HTTPStreamWriterMock added in v1.0.1 

type HTTPStreamWriterMock struct {
	// contains filtered or unexported fields
}

HTTPStreamWriterMock implements http.ResponseWriter and adds a channel to sync writes and reads

func NewHTTPStreamWriterMock added in v1.0.1 

func NewHTTPStreamWriterMock() *HTTPStreamWriterMock

NewHTTPStreamWriterMock returns a new HTTPStreamWriterMock

func (*HTTPStreamWriterMock) Flush added in v1.0.1 

func (w *HTTPStreamWriterMock) Flush()

Flush sends buffered data to the channel

func (*HTTPStreamWriterMock) Header added in v1.0.1 

func (w *HTTPStreamWriterMock) Header() http.Header

Header is a stub

func (*HTTPStreamWriterMock) Write added in v1.0.1 

func (w *HTTPStreamWriterMock) Write(data []byte) (int, error)

Write buffers sent data

func (*HTTPStreamWriterMock) WriteHeader added in v1.0.1 

func (w *HTTPStreamWriterMock) WriteHeader(statuscode int)

WriteHeader is a stub

type InsertGenerator added in v1.0.1 

type InsertGenerator struct {
	// contains filtered or unexported fields
}

InsertGenerator generates a vreplication insert statement.

func NewInsertGenerator added in v1.0.1 

func NewInsertGenerator(state, dbname string) *InsertGenerator

NewInsertGenerator creates a new InsertGenerator.

func (*InsertGenerator) AddRow added in v1.0.1 

func (ig *InsertGenerator) AddRow(workflow string, bls *binlogdatapb.BinlogSource, pos, cell, tabletTypes string)

AddRow adds a row to the insert statement.

func (*InsertGenerator) String added in v1.0.1 

func (ig *InsertGenerator) String() string

String returns the generated statement.

type MySQLVStreamerClient added in v1.0.1 

type MySQLVStreamerClient struct {
	// contains filtered or unexported fields
}

MySQLVStreamerClient a vstream client backed by MySQL

func NewMySQLVStreamerClient added in v1.0.1 

func NewMySQLVStreamerClient() *MySQLVStreamerClient

NewMySQLVStreamerClient is a vstream client that allows you to stream directly from MySQL. In order to achieve this, the following creates a vstreamer Engine with a dummy in memorytopo.

func (*MySQLVStreamerClient) Close added in v1.0.1 

func (vsClient *MySQLVStreamerClient) Close(ctx context.Context) (err error)

Close part of the VStreamerClient interface

func (*MySQLVStreamerClient) Open added in v1.0.1 

func (vsClient *MySQLVStreamerClient) Open(ctx context.Context) (err error)

Open part of the VStreamerClient interface

func (*MySQLVStreamerClient) VStream added in v1.0.1 

func (vsClient *MySQLVStreamerClient) VStream(ctx context.Context, startPos string, filter *binlogdatapb.Filter, send func([]*binlogdatapb.VEvent) error) error

VStream part of the VStreamerClient interface

func (*MySQLVStreamerClient) VStreamRows added in v1.0.1 

func (vsClient *MySQLVStreamerClient) VStreamRows(ctx context.Context, query string, lastpk *querypb.QueryResult, send func(*binlogdatapb.VStreamRowsResponse) error) error

VStreamRows part of the VStreamerClient interface

type ReplicatorPlan added in v1.0.1 

type ReplicatorPlan struct {
	VStreamFilter *binlogdatapb.Filter
	TargetTables  map[string]*TablePlan
	TablePlans    map[string]*TablePlan
	// contains filtered or unexported fields
}

ReplicatorPlan is the execution plan for the replicator. It contains plans for all the tables it's replicating. Every phase of vreplication builds its own instance of the ReplicatorPlan. This is because the plan depends on copyState, which changes on every iteration. The table plans within ReplicatorPlan will not be fully populated because all the information is not available initially. For simplicity, the ReplicatorPlan is immutable. Once we get the field info for a table from the stream response, we'll have all the necessary info to build the final plan. At that time, buildExecutionPlan is invoked, which will make a copy of the TablePlan from ReplicatorPlan, and fill the rest of the members, leaving the original plan unchanged. The constructor is buildReplicatorPlan in table_plan_builder.go

func (*ReplicatorPlan) MarshalJSON added in v1.0.1 

func (rp *ReplicatorPlan) MarshalJSON() ([]byte, error)

MarshalJSON performs a custom JSON Marshalling.

type TablePlan added in v1.0.1 

type TablePlan struct {
	// TargetName, SendRule will always be initialized.
	TargetName string
	SendRule   *binlogdatapb.Rule
	// Lastpk will be initialized if it was specified, and
	// will be used for building the final plan after field info
	// is received.
	Lastpk *sqltypes.Result
	// BulkInsertFront, BulkInsertValues and BulkInsertOnDup are used
	// by vcopier. These three parts are combined to build bulk insert
	// statements. This is functionally equivalent to generating
	// multiple statements using the "Insert" construct, but much more
	// efficient for the copy phase.
	BulkInsertFront  *sqlparser.ParsedQuery
	BulkInsertValues *sqlparser.ParsedQuery
	BulkInsertOnDup  *sqlparser.ParsedQuery
	// Insert, Update and Delete are used by vplayer.
	// If the plan is an insertIgnore type, then Insert
	// and Update contain 'insert ignore' statements and
	// Delete is nil.
	Insert *sqlparser.ParsedQuery
	Update *sqlparser.ParsedQuery
	Delete *sqlparser.ParsedQuery
	Fields []*querypb.Field
	// PKReferences is used to check if an event changed
	// a primary key column (row move).
	PKReferences []string
}

TablePlan is the execution plan for a table within a replicator. If the column names are not known at the time of plan building (like select *), then only TargetName, SendRule and Lastpk are initialized. When the stream returns the field info, those are used as column names to build the final plan. Lastpk comes from copyState. If it's set, then the generated plans are significantly different because any events that fall beyond Lastpk must be excluded. If column names were known upfront, then all fields of TablePlan are built except for Fields. This member is populated only after the field info is received from the stream. The ParsedQuery objects assume that a map of before and after values will be built based on the streaming rows. Before image values will be prefixed with a "b_", and after image values will be prefixed with a "a_". The TablePlan structure is used during all the phases of vreplication: catchup, copy, fastforward, or regular replication.

func (*TablePlan) MarshalJSON added in v1.0.1 

func (tp *TablePlan) MarshalJSON() ([]byte, error)

MarshalJSON performs a custom JSON Marshalling.

type TabletVStreamerClient added in v1.0.1 

type TabletVStreamerClient struct {
	// contains filtered or unexported fields
}

TabletVStreamerClient a vstream client backed by vttablet

func NewTabletVStreamerClient added in v1.0.1 

func NewTabletVStreamerClient(tablet *topodatapb.Tablet) *TabletVStreamerClient

NewTabletVStreamerClient creates a new TabletVStreamerClient

func (*TabletVStreamerClient) Close added in v1.0.1 

func (vsClient *TabletVStreamerClient) Close(ctx context.Context) (err error)

Close part of the VStreamerClient interface

func (*TabletVStreamerClient) Open added in v1.0.1 

func (vsClient *TabletVStreamerClient) Open(ctx context.Context) (err error)

Open part of the VStreamerClient interface

func (*TabletVStreamerClient) VStream added in v1.0.1 

func (vsClient *TabletVStreamerClient) VStream(ctx context.Context, startPos string, filter *binlogdatapb.Filter, send func([]*binlogdatapb.VEvent) error) error

VStream part of the VStreamerClient interface

func (*TabletVStreamerClient) VStreamRows added in v1.0.1 

func (vsClient *TabletVStreamerClient) VStreamRows(ctx context.Context, query string, lastpk *querypb.QueryResult, send func(*binlogdatapb.VStreamRowsResponse) error) error

VStreamRows part of the VStreamerClient interface

type VStreamerClient added in v1.0.1 

type VStreamerClient interface {
	// Open sets up all the environment for a vstream
	Open(ctx context.Context) error
	// Close closes a vstream
	Close(ctx context.Context) error

	// VStream streams VReplication events based on the specified filter.
	VStream(ctx context.Context, startPos string, filter *binlogdatapb.Filter, send func([]*binlogdatapb.VEvent) error) error

	// VStreamRows streams rows of a table from the specified starting point.
	VStreamRows(ctx context.Context, query string, lastpk *querypb.QueryResult, send func(*binlogdatapb.VStreamRowsResponse) error) error
}

VStreamerClient exposes the core interface of a vstreamer

type VrLogStats added in v1.0.1 

type VrLogStats struct {
	Type       string
	Detail     string
	StartTime  time.Time
	LogTime    string
	DurationNs int64
}

VrLogStats collects attributes of a vreplication event for logging

func NewVrLogStats added in v1.0.1 

func NewVrLogStats(eventType string) *VrLogStats

NewVrLogStats should be called at the start of the event to be logged

func (*VrLogStats) Send added in v1.0.1 

func (stats *VrLogStats) Send(detail string)

Send records the log event, should be called on a stats object constructed by NewVrLogStats()

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.