mysqlctl

package
v0.0.0-...-d4b8c80 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 24, 2014 License: BSD-3-Clause Imports: 46 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	MysqlWaitTime = 120 * time.Second // default number of seconds to wait
)
View Source 
const (
	SlaveStartDeadline = 30
)
View Source 
const (
	SnapshotManifestFile = "snapshot_manifest.json"
)
View Source 
const (
	SnapshotURLPath = "/snapshot"
)

Variables

View Source 
var (
	ErrNotSlave  = errors.New("no slave status")
	ErrNotMaster = errors.New("no master status")
)

Functions

func ConcurrentMap 

func ConcurrentMap(concurrency, n int, fun MapFunc) error

ConcurrentMap applies fun in a concurrent manner on integers from 0 to n-1 (they are assumed to be indexes of some slice containing items to be processed). The first error returned by a fun application will returned (subsequent errors will only be logged). It will use concurrency goroutines.

func GetSubprocessFlags 

func GetSubprocessFlags() []string

GetSubprocessFlags returns the flags to pass to a subprocess to have the exact same mycnf config as us.

RegisterCommandLineFlags and NewMycnfFromFlags should have been called before this.

func MakeSplitCreateTableSql 

func MakeSplitCreateTableSql(schema, databaseName, tableName string, strategy string) (string, string, error)

MakeSplitCreateTableSql returns a table creation statement that is modified to be faster, and the associated optional 'alter table' to modify the table at the end. - If the strategy contains the string 'skipAutoIncrement(NNN)' then we do not re-add the auto_increment on that table. - If the strategy contains the string 'delaySecondaryIndexes', then non-primary key indexes will be added afterwards. - If the strategy contains the string 'useMyIsam' we load the data into a myisam table and we then convert to innodb - If the strategy contains the string 'delayPrimaryKey', then the primary key index will be added afterwards (use with useMyIsam)

func MySQLReplicationLag 

func MySQLReplicationLag(mysqld *Mysqld, allowedLagInSeconds int) health.Reporter

MySQLReplication lag returns a reporter that reports the MySQL replication lag. It uses the key "replication_lag".

func RegisterFlags 

func RegisterFlags()

RegisterFlags registers the command line flags for specifying the values of a mycnf config file. See NewMycnfFromFlags to get the supported modes.

func SanityCheckManifests 

func SanityCheckManifests(ssms []*SplitSnapshotManifest) error

SanityCheckManifests checks if the ssms can be restored together.

func SnapshotDir 

func SnapshotDir(uid uint32) string

SnapshotDir returns the default directory for a tablet's snapshots

func StartReplicationCommands 

func StartReplicationCommands(mysqld *Mysqld, replState *proto.ReplicationState) ([]string, error)

func TabletDir 

func TabletDir(uid uint32) string

TabletDir returns the default directory for a tablet

func TopLevelDirs 

func TopLevelDirs() []string

TopLevelDirs returns the list of directories in the toplevel tablet directory that might be located in a different place.

Types

type FakeMysqlDaemon 

type FakeMysqlDaemon struct {
	// will be returned by GetMasterAddr(). Set to "" to return
	// ErrNotSlave, or to "ERROR" to return an error.
	MasterAddr string

	// will be returned by GetMysqlPort(). Set to -1 to return an error.
	MysqlPort int
}

FakeMysqlDaemon implements MysqlDaemon and allows the user to fake everything.

func (*FakeMysqlDaemon) GetMasterAddr 

func (fmd *FakeMysqlDaemon) GetMasterAddr() (string, error)

func (*FakeMysqlDaemon) GetMysqlPort 

func (fmd *FakeMysqlDaemon) GetMysqlPort() (int, error)

type MapFunc 

type MapFunc func(index int) error

type Mycnf 

type Mycnf struct {
	// ServerId is the unique id for this server.
	// Used to create a bunch of named directories.
	ServerId uint32

	// MysqlPort is the port for the MySQL server running on this machine.
	// It is mainly used to communicate with topology server.
	MysqlPort int

	// DataDir is where the table files are
	// (used by vt software for Clone)
	DataDir string

	// InnodbDataHomeDir is the data directory for innodb.
	// (used by vt software for Clone)
	InnodbDataHomeDir string

	// InnodbLogGroupHomeDir is the logs directory for innodb.
	// (used by vt software for Clone)
	InnodbLogGroupHomeDir string

	// SocketFile is the path to the local mysql.sock file.
	// (used by vt software to check server is running)
	SocketFile string

	// ErrorLogPath is the path to store error logs at.
	// (unused by vt software for now)
	ErrorLogPath string

	// SlowLogPath is the slow query log path
	// (unused by vt software for now)
	SlowLogPath string

	// RelayLogPath is the path of the relay logs
	// (unused by vt software for now)
	RelayLogPath string

	// RelayLogIndexPath is the file name for the relay log index
	// (unused by vt software for now)
	RelayLogIndexPath string

	// RelayLogInfoPath is the file name for the relay log info file
	// (unused by vt software for now)
	RelayLogInfoPath string

	// BinLogPath is the base path for binlogs
	// (used by vt software for binlog streaming and rowcache invalidation)
	BinLogPath string

	// MasterInfoFile is the master.info file location.
	// (unused by vt software for now)
	MasterInfoFile string

	// PidFile is the mysql.pid file location
	// (used by vt software to check server is running)
	PidFile string

	// TmpDir is where to create temporary tables
	// (unused by vt software for now)
	TmpDir string

	// SlaveLoadTmpDir is where to create tmp files for replication
	// (unused by vt software for now)
	SlaveLoadTmpDir string
	// contains filtered or unexported fields
}

Mycnf is a memory structure that contains a bunch of interesting parameters to start mysqld. It can be used to generate standard my.cnf files from a server id and mysql port. It can also be populated from an existing my.cnf, or by command line parameters. command line parameters.

func NewMycnf 

func NewMycnf(uid uint32, mysqlPort int) *Mycnf

NewMycnf fills the Mycnf structure with vt root paths and derived values. This is used to fill out the cnfTemplate values and generate my.cnf. uid is a unique id for a particular tablet - it must be unique within the tabletservers deployed within a keyspace, lest there be collisions on disk. mysqldPort needs to be unique per instance per machine.

func NewMycnfFromFlags 

func NewMycnfFromFlags(uid uint32) (mycnf *Mycnf, err error)

NewMycnfFromFlags creates a Mycnf object from the command line flags.

Multiple modes are supported:

  • at least mycnf_server_id is set on the command line --> then we read all parameters from the command line, and not from any my.cnf file.
  • mycnf_server_id is not passed in, but mycnf-file is passed in --> then we read that mycnf file
  • mycnf_server_id and mycnf-file are not passed in: --> then we use the default location of the my.cnf file for the provided uid and read that my.cnf file.

RegisterCommandLineFlags should have been called before calling this, otherwise we'll panic.

func ReadMycnf 

func ReadMycnf(cnfFile string) (mycnf *Mycnf, err error)

ReadMycnf will read an existing my.cnf from disk, and create a Mycnf object.

type MysqlDaemon 

type MysqlDaemon interface {
	// GetMasterAddr returns the mysql master address, as shown by
	// 'show slave status'.
	GetMasterAddr() (string, error)

	// GetMysqlPort returns the current port mysql is listening on.
	GetMysqlPort() (int, error)
}

MysqlDaemon is the interface we use for abstracting Mysqld.

type MysqlFlavor 

type MysqlFlavor interface {
	// MasterStatus fills in the ReplicationPosition structure.
	// It has two components: the result of a standard 'SHOW MASTER STATUS'
	// and the corresponding transaction group id.
	MasterStatus(mysqld *Mysqld) (*proto.ReplicationPosition, error)

	// PromoteSlaveCommands returns the commands to run to change
	// a slave into a master
	PromoteSlaveCommands() []string
}

MysqlFlavor is the abstract interface for a flavor.

type Mysqld 

type Mysqld struct {
	TabletDir   string
	SnapshotDir string
	// contains filtered or unexported fields
}

Mysqld is the object that represents a mysqld daemon running on this server.

func NewMysqld 

func NewMysqld(name string, config *Mycnf, dba, repl *mysql.ConnectionParams) *Mysqld

NewMysqld creates a Mysqld object based on the provided configuration and connection parameters. name is the base for stats exports, use 'Dba', except in tests

func (*Mysqld) Addr 

func (mysqld *Mysqld) Addr() string

Addr returns the fully qualified host name + port for this instance.

func (*Mysqld) ApplySchemaChange 

func (mysqld *Mysqld) ApplySchemaChange(dbName string, change *proto.SchemaChange) (*proto.SchemaChangeResult, error)

ApplySchemaChange will apply the schema change to the given database.

func (*Mysqld) BinlogInfo 

func (mysqld *Mysqld) BinlogInfo(groupId int64) (rp *proto.ReplicationPosition, err error)

mysql> show binlog info for 5\G *************************** 1. row *************************** Log_name: vt-0000041983-bin.000001 Pos: 1194 Server_ID: 41983

func (*Mysqld) BreakSlaves 

func (mysqld *Mysqld) BreakSlaves() error

Force all slaves to error and stop. This is extreme, but helpful for emergencies and tests. Insert a row, block the propagation of its subsequent delete and reinsert it. This forces a failure on slaves only.

func (*Mysqld) CheckReplication 

func (mysqld *Mysqld) CheckReplication(timeCheck int64) error

Check for the magic row inserted under controlled reparenting.

func (*Mysqld) Close 

func (mysqld *Mysqld) Close()

Close will close this instance of Mysqld. It will wait for all dba queries to be finished.

func (*Mysqld) Cnf 

func (mt *Mysqld) Cnf() *Mycnf

Cnf returns the mysql config for the daemon

func (*Mysqld) CreateMultiSnapshot 

func (mysqld *Mysqld) CreateMultiSnapshot(keyRanges []key.KeyRange, dbName, keyName string, keyType key.KeyspaceIdType, sourceAddr string, allowHierarchicalReplication bool, snapshotConcurrency int, tables, excludeTables []string, skipSlaveRestart bool, maximumFilesize uint64, hookExtraEnv map[string]string) (snapshotManifestFilenames []string, err error)

CreateMultiSnapshot create snapshots of the data.

  • for a resharding snapshot, keyRanges+keyName+keyType are set, and tables is empty. This action will create multiple snapshots, one per keyRange.
  • for a vertical split, tables is set, keyRanges = [KeyRange{}] and keyName+keyType are empty. It will create a single snapshot of the contents of the tables.

Note combinations of table subset and keyranges are not supported.

func (*Mysqld) CreateSnapshot 

func (mysqld *Mysqld) CreateSnapshot(dbName, sourceAddr string, allowHierarchicalReplication bool, concurrency int, serverMode bool, hookExtraEnv map[string]string) (snapshotManifestUrlPath string, slaveStartRequired, readOnly bool, err error)

This function runs on the machine acting as the source for the clone.

Check master/slave status and determine restore needs. If this instance is a slave, stop replication, otherwise place in read-only mode. Record replication position. Shutdown mysql Check paths for storing data

Depending on the serverMode flag, we do the following: serverMode = false: 

Compress /vt/vt_[0-9a-f]+/data/vt_.+
Compute hash (of compressed files, as we serve .gz files here)
Place in /vt/clone_src where they will be served by http server (not rpc)
Restart mysql

serverMode = true: 

Make symlinks for /vt/vt_[0-9a-f]+/data/vt_.+ to innodb files
Compute hash (of uncompressed files, as we serve uncompressed files)
Place symlinks in /vt/clone_src where they will be served by http server
Leave mysql stopped, return slaveStartRequired, readOnly

func (*Mysqld) DemoteMaster 

func (mysqld *Mysqld) DemoteMaster() (*proto.ReplicationPosition, error)

if the master is still alive, then we need to demote it gracefully make it read-only, flush the writes and get the position

func (*Mysqld) ExecuteMysqlCommand 

func (mysqld *Mysqld) ExecuteMysqlCommand(sql string) error

executes some SQL commands using a mysql command line interface process

func (*Mysqld) ExecuteSuperQuery 

func (mysqld *Mysqld) ExecuteSuperQuery(query string) error

func (*Mysqld) ExecuteSuperQueryList 

func (mysqld *Mysqld) ExecuteSuperQueryList(queryList []string) error

ExecuteSuperQueryList alows the user to execute queries at a super user.

func (*Mysqld) FindSlaves 

func (mysqld *Mysqld) FindSlaves() ([]string, error)

Get IP addresses for all currently connected slaves.

func (*Mysqld) FindVtDatabases 

func (mysqld *Mysqld) FindVtDatabases() ([]string, error)

func (*Mysqld) GetColumns 

func (mysqld *Mysqld) GetColumns(dbName, table string) ([]string, error)

GetColumns returns the columns of table.

func (*Mysqld) GetDbaConnection 

func (mysqld *Mysqld) GetDbaConnection() (dbconnpool.PoolConnection, error)

GetDbaConnection returns a connection from the dba pool. Recycle needs to be called on the result.

func (*Mysqld) GetMasterAddr 

func (mysqld *Mysqld) GetMasterAddr() (string, error)

func (*Mysqld) GetMysqlPort 

func (mysqld *Mysqld) GetMysqlPort() (int, error)

func (*Mysqld) GetPermissions 

func (mysqld *Mysqld) GetPermissions() (*proto.Permissions, error)

func (*Mysqld) GetPrimaryKeyColumns 

func (mysqld *Mysqld) GetPrimaryKeyColumns(dbName, table string) ([]string, error)

GetPrimaryKeyColumns returns the primary key columns of table.

func (*Mysqld) GetSchema 

func (mysqld *Mysqld) GetSchema(dbName string, tables, excludeTables []string, includeViews bool) (*proto.SchemaDefinition, error)

GetSchema returns the schema for database for tables listed in tables. If tables is empty, return the schema for all tables.

func (*Mysqld) Init 

func (mt *Mysqld) Init(mysqlWaitTime time.Duration) error

Init will create the default directory structure for the mysqld process, generate / configure a my.cnf file, unpack a skeleton database, and create some management tables.

func (*Mysqld) IpAddr 

func (mysqld *Mysqld) IpAddr() string

IpAddr returns the IP address for this instance

func (*Mysqld) IsReadOnly 

func (mysqld *Mysqld) IsReadOnly() (bool, error)

func (*Mysqld) MasterStatus 

func (mysqld *Mysqld) MasterStatus() (rp *proto.ReplicationPosition, err error)

func (*Mysqld) MultiRestore 

func (mysqld *Mysqld) MultiRestore(destinationDbName string, keyRanges []key.KeyRange, sourceAddrs []*url.URL, fromStoragePaths []string, snapshotConcurrency, fetchConcurrency, insertTableConcurrency, fetchRetryCount int, strategy string) (err error)

MultiRestore is the main entry point for multi restore.

We will either: - read from the network if sourceAddrs != nil - read from a disk snapshot if fromStoragePaths != nil

The strategy is used as follows:

  • If it contains the string 'writeBinLogs' then we will also write to the binary logs.
  • If it contains the command 'populateBlpCheckpoint' then we will populate the blp_checkpoint table with master positions to start from
  • If is also contains the command 'dontStartBinlogPlayer' we won't start binlog replication on the destination (but it will be configured)

func (*Mysqld) PreflightSchemaChange 

func (mysqld *Mysqld) PreflightSchemaChange(dbName string, change string) (*proto.SchemaChangeResult, error)

PreflightSchemaChange will apply the schema change to a fake database that has the same schema as the target database, see if it works.

func (*Mysqld) PromoteSlave 

func (mysqld *Mysqld) PromoteSlave(setReadWrite bool, hookExtraEnv map[string]string) (replicationState *proto.ReplicationState, waitPosition *proto.ReplicationPosition, timePromoted int64, err error)

setReadWrite: set the new master in read-write mode.

replicationState: info slaves need to reparent themselves waitPosition: slaves can wait for this position when restarting replication timePromoted: this timestamp (unix nanoseconds) is inserted into _vt.replication_log to verify the replication config

func (*Mysqld) ReparentPosition 

func (mysqld *Mysqld) ReparentPosition(slavePosition *proto.ReplicationPosition) (rs *proto.ReplicationState, waitPosition *proto.ReplicationPosition, reparentTime int64, err error)

Return a replication state that will reparent a slave to the correct master for a specified position.

func (*Mysqld) ResolveTables 

func (mysqld *Mysqld) ResolveTables(dbName string, tables []string) ([]string, error)

ResolveTables returns a list of actual tables+views matching a list of regexps

func (*Mysqld) RestartSlave 

func (mysqld *Mysqld) RestartSlave(replicationState *proto.ReplicationState, waitPosition *proto.ReplicationPosition, timeCheck int64) error

func (*Mysqld) RestoreFromSnapshot 

func (mysqld *Mysqld) RestoreFromSnapshot(snapshotManifest *SnapshotManifest, fetchConcurrency, fetchRetryCount int, dontWaitForSlaveStart bool, hookExtraEnv map[string]string) error

This piece runs on the presumably empty machine acting as the target in the create replica action.

validate target (self) shutdown_mysql() create temp data directory /vt/target/vt_<keyspace> copy compressed data files via HTTP verify hash of compressed files uncompress into /vt/vt_<target-uid>/data/vt_<keyspace> start_mysql() clean up compressed files

func (*Mysqld) SetReadOnly 

func (mysqld *Mysqld) SetReadOnly(on bool) error

func (*Mysqld) Shutdown 

func (mt *Mysqld) Shutdown(waitForMysqld bool, mysqlWaitTime time.Duration) error

Shutdown will stop the mysqld daemon that is running in the background.

waitForMysqld: should the function block until mysqld has stopped? This can actually take a *long* time if the buffer cache needs to be fully flushed - on the order of 20-30 minutes.

func (*Mysqld) SlaveStatus 

func (mysqld *Mysqld) SlaveStatus() (*proto.ReplicationPosition, error)

func (*Mysqld) SnapshotSourceEnd 

func (mysqld *Mysqld) SnapshotSourceEnd(slaveStartRequired, readOnly, deleteSnapshot bool, hookExtraEnv map[string]string) error

func (*Mysqld) Start 

func (mt *Mysqld) Start(mysqlWaitTime time.Duration) error

Start will start the mysql daemon, either by running the 'mysqld_start' hook, or by running mysqld_safe in the background.

func (*Mysqld) StartSlave 

func (mysqld *Mysqld) StartSlave(hookExtraEnv map[string]string) error

func (*Mysqld) StopSlave 

func (mysqld *Mysqld) StopSlave(hookExtraEnv map[string]string) error

func (*Mysqld) Teardown 

func (mt *Mysqld) Teardown(force bool) error

Teardown will shutdown the running daemon, and delete the root directory.

func (*Mysqld) ValidateCloneTarget 

func (mysqld *Mysqld) ValidateCloneTarget(hookExtraEnv map[string]string) error

func (*Mysqld) ValidateSnapshotPath 

func (mysqld *Mysqld) ValidateSnapshotPath() error

Helper function to make sure we can write to the local snapshot area, before we actually do any action (can be used for both partial and full snapshots)

func (*Mysqld) WaitBlpPos 

func (mysqld *Mysqld) WaitBlpPos(bp *blproto.BlpPosition, waitTimeout time.Duration) error

WaitBlpPos will wait for the filtered replication to reach at least the provided position.

func (*Mysqld) WaitForMinimumReplicationPosition 

func (mysqld *Mysqld) WaitForMinimumReplicationPosition(groupId int64, waitTimeout time.Duration) (err error)

func (*Mysqld) WaitForSlave 

func (mysqld *Mysqld) WaitForSlave(maxLag int) (err error)

func (*Mysqld) WaitForSlaveStart 

func (mysqld *Mysqld) WaitForSlaveStart(slaveStartDeadline int) (err error)

func (*Mysqld) WaitMasterPos 

func (mysqld *Mysqld) WaitMasterPos(rp *proto.ReplicationPosition, waitTimeout time.Duration) error

type SnapshotFile 

type SnapshotFile struct {
	Path      string
	Size      int64
	Hash      string
	TableName string
}

SnapshotFile describes a file to serve. 'Path' is the path component of the URL. SnapshotManifest.Addr is the host+port component of the URL. If path ends in '.gz', it is compressed. Size and Hash are computed on the Path itself if TableName is set, this file belongs to that table

type SnapshotFiles 

type SnapshotFiles []SnapshotFile

func (SnapshotFiles) Len 

func (s SnapshotFiles) Len() int

sort.Interface we sort by descending file size

func (SnapshotFiles) Less 

func (s SnapshotFiles) Less(i, j int) bool

func (SnapshotFiles) Swap 

func (s SnapshotFiles) Swap(i, j int)

type SnapshotManifest 

type SnapshotManifest struct {
	Addr string // this is the address of the tabletserver, not mysql

	DbName string
	Files  SnapshotFiles

	ReplicationState *proto.ReplicationState
	MasterState      *proto.ReplicationState
}

a SnapshotManifest describes multiple SnapshotFiles and where to get them from.

func ReadSnapshotManifest 

func ReadSnapshotManifest(filename string) (*SnapshotManifest, error)

type SplitSnapshotManifest 

type SplitSnapshotManifest struct {
	// Source describes the files and our tablet
	Source *SnapshotManifest

	// KeyRange describes the data present in this snapshot
	// When splitting 40-80 into 40-60 and 60-80, this would
	// have 40-60 for instance.
	KeyRange key.KeyRange

	// The schema for this server
	SchemaDefinition *proto.SchemaDefinition
}

func NewSplitSnapshotManifest 

func NewSplitSnapshotManifest(myAddr, myMysqlAddr, masterAddr, dbName string, files []SnapshotFile, pos, myMasterPos *proto.ReplicationPosition, keyRange key.KeyRange, sd *proto.SchemaDefinition) (*SplitSnapshotManifest, error)

NewSplitSnapshotManifest creates a new SplitSnapshotManifest. myAddr and myMysqlAddr are the local server addresses. masterAddr is the address of the server to use as master. pos is the replication position to use on that master. myMasterPos is the local server master position

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.