automation

package
v1.0.10 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 21, 2020 License: Apache-2.0 Imports: 16 Imported by: 0

Documentation

Overview

Package automation contains code to execute high-level cluster operations (e.g. resharding) as a series of low-level operations (e.g. vtctl, shell commands, ...).

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func AddMissingTaskID 

func AddMissingTaskID(tc []*automationpb.TaskContainer, taskIDGenerator *IDGenerator)

AddMissingTaskID assigns a task id to each task in "tc".

func AddTask 

func AddTask(t *automationpb.TaskContainer, taskName string, parameters map[string]string)

AddTask adds a new task to an existing task container.

func ExecuteVtctl 

func ExecuteVtctl(ctx context.Context, server string, args []string) (string, error)

ExecuteVtctl runs vtctl using vtctlclient. The stream of Event messages is concatenated into one output string. Additionally, the start and the end of the command will be logged to make it easier to debug which command was executed and how long it took.

func ExecuteVtworker 

func ExecuteVtworker(ctx context.Context, server string, args []string) (string, error)

ExecuteVtworker executes the vtworker command in "args" via an RPC to "server". The output of the RPC, a stream of LoggerEvent messages, is concatenated into one output string. If a retryable error is encountered (e.g. the vtworker process is already executing another command), this function will keep retrying infinitely until "ctx" is cancelled.

func MarkTaskFailed 

func MarkTaskFailed(t *automationpb.Task, output string, err error)

MarkTaskFailed marks the task as failed.

func MarkTaskSucceeded 

func MarkTaskSucceeded(t *automationpb.Task, output string)

MarkTaskSucceeded marks the task as done.

func NewTask 

func NewTask(taskName string, parameters map[string]string) *automationpb.Task

NewTask creates a new task protobuf message for "taskName" with "parameters".

func NewTaskContainer 

func NewTaskContainer() *automationpb.TaskContainer

NewTaskContainer creates an empty task container. Use AddTask() to add tasks to it.

func NewTaskContainerWithSingleTask 

func NewTaskContainerWithSingleTask(taskName string, parameters map[string]string) *automationpb.TaskContainer

NewTaskContainerWithSingleTask creates a new task container with exactly one task.

Types

type ClusterOperationInstance 

type ClusterOperationInstance struct {
	automationpb.ClusterOperation
	// contains filtered or unexported fields
}

ClusterOperationInstance is a runtime type which enhances the protobuf message "ClusterOperation" with runtime specific data. Unlike the protobuf message, the additional runtime data will not be part of a checkpoint. Methods of this struct are not thread-safe.

func NewClusterOperationInstance 

func NewClusterOperationInstance(clusterOpID string, initialTask *automationpb.TaskContainer, taskIDGenerator *IDGenerator) ClusterOperationInstance

NewClusterOperationInstance creates a new cluster operation instance with one initial task.

func (ClusterOperationInstance) Clone 

func (c ClusterOperationInstance) Clone() ClusterOperationInstance

Clone creates a deep copy of the inner protobuf. Other elements e.g. taskIDGenerator are not deep-copied.

func (*ClusterOperationInstance) InsertTaskContainers 

func (c *ClusterOperationInstance) InsertTaskContainers(newTaskContainers []*automationpb.TaskContainer, pos int)

InsertTaskContainers inserts "newTaskContainers" at pos in the current list of task containers. Existing task containers will be moved after the new task containers.

type CopySchemaShardTask 

type CopySchemaShardTask struct {
}

CopySchemaShardTask runs vtctl CopySchemaShard to copy the schema from one shard to another.

func (*CopySchemaShardTask) OptionalParameters 

func (t *CopySchemaShardTask) OptionalParameters() []string

OptionalParameters is part of the Task interface.

func (*CopySchemaShardTask) RequiredParameters 

func (t *CopySchemaShardTask) RequiredParameters() []string

RequiredParameters is part of the Task interface.

func (*CopySchemaShardTask) Run 

func (t *CopySchemaShardTask) Run(parameters map[string]string) ([]*automationpb.TaskContainer, string, error)

Run is part of the Task interface.

type HorizontalReshardingTask 

type HorizontalReshardingTask struct {
}

HorizontalReshardingTask is a cluster operation which allows to increase the number of shards.

func (*HorizontalReshardingTask) OptionalParameters 

func (t *HorizontalReshardingTask) OptionalParameters() []string

OptionalParameters is part of the Task interface.

func (*HorizontalReshardingTask) RequiredParameters 

func (t *HorizontalReshardingTask) RequiredParameters() []string

RequiredParameters is part of the Task interface.

func (*HorizontalReshardingTask) Run 

func (t *HorizontalReshardingTask) Run(parameters map[string]string) ([]*automationpb.TaskContainer, string, error)

Run is part of the Task interface.

type IDGenerator 

type IDGenerator struct {
	// contains filtered or unexported fields
}

IDGenerator generates unique task and cluster operation IDs.

func (*IDGenerator) GetNextID 

func (ig *IDGenerator) GetNextID() string

GetNextID returns an ID which wasn't returned before.

type MigrateServedFromTask 

type MigrateServedFromTask struct {
}

MigrateServedFromTask runs vtctl MigrateServedFrom to let vertically split out tables get served from the new destination keyspace.

func (*MigrateServedFromTask) OptionalParameters 

func (t *MigrateServedFromTask) OptionalParameters() []string

OptionalParameters is part of the Task interface.

func (*MigrateServedFromTask) RequiredParameters 

func (t *MigrateServedFromTask) RequiredParameters() []string

RequiredParameters is part of the Task interface.

func (*MigrateServedFromTask) Run 

func (t *MigrateServedFromTask) Run(parameters map[string]string) ([]*automationpb.TaskContainer, string, error)

Run is part of the Task interface.

type MigrateServedTypesTask 

type MigrateServedTypesTask struct {
}

MigrateServedTypesTask runs vtctl MigrateServedTypes to migrate a serving type from the source shard to the shards that it replicates to.

func (*MigrateServedTypesTask) OptionalParameters 

func (t *MigrateServedTypesTask) OptionalParameters() []string

OptionalParameters is part of the Task interface.

func (*MigrateServedTypesTask) RequiredParameters 

func (t *MigrateServedTypesTask) RequiredParameters() []string

RequiredParameters is part of the Task interface.

func (*MigrateServedTypesTask) Run 

func (t *MigrateServedTypesTask) Run(parameters map[string]string) ([]*automationpb.TaskContainer, string, error)

Run is part of the Task interface.

type RebuildKeyspaceGraphTask 

type RebuildKeyspaceGraphTask struct {
}

RebuildKeyspaceGraphTask runs vtctl RebuildKeyspaceGraph to migrate a serving type from the source shard to the shards that it replicates to.

func (*RebuildKeyspaceGraphTask) OptionalParameters 

func (t *RebuildKeyspaceGraphTask) OptionalParameters() []string

OptionalParameters is part of the Task interface.

func (*RebuildKeyspaceGraphTask) RequiredParameters 

func (t *RebuildKeyspaceGraphTask) RequiredParameters() []string

RequiredParameters is part of the Task interface.

func (*RebuildKeyspaceGraphTask) Run 

func (t *RebuildKeyspaceGraphTask) Run(parameters map[string]string) ([]*automationpb.TaskContainer, string, error)

Run is part of the Task interface.

type Scheduler 

type Scheduler struct {
	// contains filtered or unexported fields
}

Scheduler executes automation tasks and maintains the execution state.

func NewScheduler 

func NewScheduler() (*Scheduler, error)

NewScheduler creates a new instance.

func (*Scheduler) Checkpoint 

func (s *Scheduler) Checkpoint(clusterOp ClusterOperationInstance)

Checkpoint should be called every time the state of the cluster op changes. It is used to update the copy of the state in activeClusterOperations.

func (*Scheduler) EnqueueClusterOperation 

func (s *Scheduler) EnqueueClusterOperation(ctx context.Context, req *automationpb.EnqueueClusterOperationRequest) (*automationpb.EnqueueClusterOperationResponse, error)

EnqueueClusterOperation can be used to start a new cluster operation.

func (*Scheduler) GetClusterOperationDetails 

func (s *Scheduler) GetClusterOperationDetails(ctx context.Context, req *automationpb.GetClusterOperationDetailsRequest) (*automationpb.GetClusterOperationDetailsResponse, error)

GetClusterOperationDetails can be used to query the full details of active or finished operations.

func (*Scheduler) Run 

func (s *Scheduler) Run()

Run processes queued cluster operations.

func (*Scheduler) ShutdownAndWait 

func (s *Scheduler) ShutdownAndWait()

ShutdownAndWait shuts down the scheduler and waits infinitely until all pending cluster operations have finished.

type SplitCloneTask 

type SplitCloneTask struct {
}

SplitCloneTask runs SplitClone on a remote vtworker to split an existing shard.

func (*SplitCloneTask) OptionalParameters 

func (t *SplitCloneTask) OptionalParameters() []string

OptionalParameters is part of the Task interface.

func (*SplitCloneTask) RequiredParameters 

func (t *SplitCloneTask) RequiredParameters() []string

RequiredParameters is part of the Task interface.

func (*SplitCloneTask) Run 

func (t *SplitCloneTask) Run(parameters map[string]string) ([]*automationpb.TaskContainer, string, error)

Run is part of the Task interface.

type SplitDiffTask 

type SplitDiffTask struct {
}

SplitDiffTask runs SplitDiff on a remote vtworker to compare the old shard against its new split shards.

func (*SplitDiffTask) OptionalParameters 

func (t *SplitDiffTask) OptionalParameters() []string

OptionalParameters is part of the Task interface.

func (*SplitDiffTask) RequiredParameters 

func (t *SplitDiffTask) RequiredParameters() []string

RequiredParameters is part of the Task interface.

func (*SplitDiffTask) Run 

func (t *SplitDiffTask) Run(parameters map[string]string) ([]*automationpb.TaskContainer, string, error)

Run is part of the Task interface.

type Task 

type Task interface {
	// Run executes the task using the key/values from parameters.
	// "newTaskContainers" contains new tasks which the task can emit. They'll be inserted in the cluster operation directly after this task. It may be "nil".
	// "output" may be empty. It contains any text which maybe must e.g. to debug the task or show it in the UI.
	Run(parameters map[string]string) (newTaskContainers []*automationpb.TaskContainer, output string, err error)

	// RequiredParameters() returns a list of parameter keys which must be provided as input for Run().
	RequiredParameters() []string

	// OptionalParameters() returns a list of parameter keys which are optional input for Run().
	OptionalParameters() []string
}

Task implementations can be executed by the scheduler.

type VerticalSplitCloneTask 

type VerticalSplitCloneTask struct {
}

VerticalSplitCloneTask runs VerticalSplitClone on a remote vtworker to split out tables from an existing keyspace to a different keyspace.

func (*VerticalSplitCloneTask) OptionalParameters 

func (t *VerticalSplitCloneTask) OptionalParameters() []string

OptionalParameters is part of the Task interface.

func (*VerticalSplitCloneTask) RequiredParameters 

func (t *VerticalSplitCloneTask) RequiredParameters() []string

RequiredParameters is part of the Task interface.

func (*VerticalSplitCloneTask) Run 

func (t *VerticalSplitCloneTask) Run(parameters map[string]string) ([]*automationpb.TaskContainer, string, error)

Run is part of the Task interface.

type VerticalSplitDiffTask 

type VerticalSplitDiffTask struct {
}

VerticalSplitDiffTask runs VerticalSplitDiff on a remote vtworker to compare the split out tables against the source keyspace.

func (*VerticalSplitDiffTask) OptionalParameters 

func (t *VerticalSplitDiffTask) OptionalParameters() []string

OptionalParameters is part of the Task interface.

func (*VerticalSplitDiffTask) RequiredParameters 

func (t *VerticalSplitDiffTask) RequiredParameters() []string

RequiredParameters is part of the Task interface.

func (*VerticalSplitDiffTask) Run 

func (t *VerticalSplitDiffTask) Run(parameters map[string]string) ([]*automationpb.TaskContainer, string, error)

Run is part of the Task interface.

type VerticalSplitTask 

type VerticalSplitTask struct {
}

VerticalSplitTask is a cluster operation to split out specific tables of one keyspace to a different keyspace.

func (*VerticalSplitTask) OptionalParameters 

func (t *VerticalSplitTask) OptionalParameters() []string

OptionalParameters is part of the Task interface.

func (*VerticalSplitTask) RequiredParameters 

func (t *VerticalSplitTask) RequiredParameters() []string

RequiredParameters is part of the Task interface.

func (*VerticalSplitTask) Run 

func (t *VerticalSplitTask) Run(parameters map[string]string) ([]*automationpb.TaskContainer, string, error)

Run is part of the Task interface.

type WaitForFilteredReplicationTask 

type WaitForFilteredReplicationTask struct {
}

WaitForFilteredReplicationTask runs vtctl WaitForFilteredReplication to block until the destination master (i.e. the receiving side of the filtered replication) has caught up up to max_delay with the source shard.

func (*WaitForFilteredReplicationTask) OptionalParameters 

func (t *WaitForFilteredReplicationTask) OptionalParameters() []string

OptionalParameters is part of the Task interface.

func (*WaitForFilteredReplicationTask) RequiredParameters 

func (t *WaitForFilteredReplicationTask) RequiredParameters() []string

RequiredParameters is part of the Task interface.

func (*WaitForFilteredReplicationTask) Run 

func (t *WaitForFilteredReplicationTask) Run(parameters map[string]string) ([]*automationpb.TaskContainer, string, error)

Run is part of the Task interface.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.