master

package
v0.0.0-...-690b2a9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 17, 2024 License: Apache-2.0 Imports: 23 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type BaseHandle 

type BaseHandle interface {
	Status() *frameModel.WorkerStatus
	ID() frameModel.WorkerID
}

BaseHandle provides some common api of a worker, no matter it is running or dead.

type Callback 

type Callback = func(ctx context.Context, handle WorkerHandle) error

Callback alias to worker callback function when there is no error along with.

type CallbackWithError 

type CallbackWithError = func(ctx context.Context, handle WorkerHandle, err error) error

CallbackWithError alias to worker callback function when there could be an error along with.

type CreateWorkerOpt 

type CreateWorkerOpt func(opts *createWorkerOpts)

CreateWorkerOpt represents an option of creating a worker.

func CreateWorkerWithResourceRequirements 

func CreateWorkerWithResourceRequirements(resources ...resModel.ResourceID) CreateWorkerOpt

CreateWorkerWithResourceRequirements specifies the resource requirement of a worker.

func CreateWorkerWithSelectors 

func CreateWorkerWithSelectors(selectors ...*label.Selector) CreateWorkerOpt

CreateWorkerWithSelectors specifies the selectors used to dispatch the worker.

type MockHandle 

type MockHandle struct {
	WorkerID      frameModel.WorkerID
	WorkerStatus  *frameModel.WorkerStatus
	ExecutorID    model.ExecutorID
	MessageSender *p2p.MockMessageSender
	IsTombstone   bool
	// contains filtered or unexported fields
}

MockHandle implements WorkerHandle, it can work as either a RunningHandle or a TombstoneHandle.

func (*MockHandle) CleanTombstone 

func (h *MockHandle) CleanTombstone(ctx context.Context) error

CleanTombstone implements TombstoneHandle.CleanTombstone

func (*MockHandle) GetTombstone 

func (h *MockHandle) GetTombstone() TombstoneHandle

GetTombstone implements WorkerHandle.GetTombstone

func (*MockHandle) ID 

func (h *MockHandle) ID() frameModel.WorkerID

ID implements WorkerHandle.ID

func (*MockHandle) SendMessageCount 

func (h *MockHandle) SendMessageCount() int

SendMessageCount returns the send message count, used in unit test only.

func (*MockHandle) Status 

func (h *MockHandle) Status() *frameModel.WorkerStatus

Status implements WorkerHandle.Status

func (*MockHandle) Unwrap 

func (h *MockHandle) Unwrap() RunningHandle

Unwrap implements WorkerHandle.Unwrap

type RunningHandle 

type RunningHandle interface {
	BaseHandle

	SendMessage(
		ctx context.Context,
		topic p2p.Topic,
		message interface{},
		nonblocking bool,
	) error
}

RunningHandle represents a running worker

type StartWorkerCallbackType 

type StartWorkerCallbackType = func(
	workerID frameModel.WorkerID,
	executorID model.ExecutorID,
	epoch frameModel.Epoch,
)

StartWorkerCallbackType is the type for a callback that's called before the executor launches the worker. It is useful in the 2-phase task submission process.

type TombstoneHandle 

type TombstoneHandle interface {
	BaseHandle

	// CleanTombstone cleans the metadata from the metastore,
	// and cleans the state managed by the framework.
	// Do not call any other methods on this handle after
	// CleanTombstone is called.
	CleanTombstone(ctx context.Context) error
}

TombstoneHandle represents a dead worker.

type WorkerCreationHooks 

type WorkerCreationHooks struct {
	// BeforeStartingWorker will be called AFTER the executor has received the task
	// but BEFORE the worker is actually started.
	BeforeStartingWorker StartWorkerCallbackType
}

WorkerCreationHooks contains hooks to be called at specifc points of the worker creation process.

type WorkerCreator 

type WorkerCreator struct {
	// contains filtered or unexported fields
}

WorkerCreator implements the worker creation logic.

func (*WorkerCreator) CreateWorker 

func (c *WorkerCreator) CreateWorker(
	ctx context.Context,
	projectInfo tenant.ProjectInfo,
	workerType frameModel.WorkerType,
	workerID frameModel.WorkerID,
	rawConfig []byte,
	opts ...CreateWorkerOpt,
) error

CreateWorker creates a worker synchronously.

type WorkerCreatorBuilder 

type WorkerCreatorBuilder struct {
	// contains filtered or unexported fields
}

WorkerCreatorBuilder is a helper for building a WorkerCreator.

func NewWorkerCreatorBuilder 

func NewWorkerCreatorBuilder() *WorkerCreatorBuilder

NewWorkerCreatorBuilder returns a new WorkerCreatorBuilder.

func (*WorkerCreatorBuilder) Build 

func (b *WorkerCreatorBuilder) Build() *WorkerCreator

Build returns the resulted WorkerCreator.

func (*WorkerCreatorBuilder) WithExecutorGroup 

func (b *WorkerCreatorBuilder) WithExecutorGroup(executorGroup client.ExecutorGroup) *WorkerCreatorBuilder

WithExecutorGroup passes an ExecutorGroup.

func (*WorkerCreatorBuilder) WithFrameMetaClient 

func (b *WorkerCreatorBuilder) WithFrameMetaClient(frameMetaClient pkgOrm.Client) *WorkerCreatorBuilder

WithFrameMetaClient passes a frameMetaClient.

func (*WorkerCreatorBuilder) WithHooks 

func (b *WorkerCreatorBuilder) WithHooks(hooks *WorkerCreationHooks) *WorkerCreatorBuilder

WithHooks passes a WorkerCreationHooks.

func (*WorkerCreatorBuilder) WithInheritedSelectors 

func (b *WorkerCreatorBuilder) WithInheritedSelectors(selectors ...*label.Selector) *WorkerCreatorBuilder

WithInheritedSelectors passes the selectors that will be inherited from the master.

func (*WorkerCreatorBuilder) WithLogger 

func (b *WorkerCreatorBuilder) WithLogger(logger *zap.Logger) *WorkerCreatorBuilder

WithLogger passes a logger.

func (*WorkerCreatorBuilder) WithMasterID 

func (b *WorkerCreatorBuilder) WithMasterID(masterID frameModel.MasterID) *WorkerCreatorBuilder

WithMasterID specifies the ID of the master that the WorkerCreator belongs to.

func (*WorkerCreatorBuilder) WithServerMasterClient 

func (b *WorkerCreatorBuilder) WithServerMasterClient(serverMasterClient client.ServerMasterClient) *WorkerCreatorBuilder

WithServerMasterClient passes a ServerMasterClient.

type WorkerHandle 

type WorkerHandle interface {
	BaseHandle

	GetTombstone() TombstoneHandle
	Unwrap() RunningHandle
}

WorkerHandle defines the interface of a worker, businiss logic can use this handler to get RunningHandle or TombstoneHandle

type WorkerManager 

type WorkerManager struct {
	// contains filtered or unexported fields
}

WorkerManager manages all workers belonging to a job master

func NewWorkerManager 

func NewWorkerManager(
	masterID frameModel.MasterID,
	epoch frameModel.Epoch,
	meta pkgOrm.Client,
	messageSender p2p.MessageSender,
	onWorkerOnline Callback,
	onWorkerOffline CallbackWithError,
	onWorkerStatusUpdated Callback,
	onWorkerDispatched CallbackWithError,
	isInit bool,
	timeoutConfig config.TimeoutConfig,
	clock clock.Clock,
) *WorkerManager

NewWorkerManager creates a new WorkerManager instance

func (*WorkerManager) AbortCreatingWorker 

func (m *WorkerManager) AbortCreatingWorker(workerID frameModel.WorkerID, errIn error)

AbortCreatingWorker is called by BaseMaster if starting the worker has failed for sure. NOTE: If the RPC used to start the worker returns errors such as Canceled or DeadlineExceeded, it has NOT failed FOR SURE.

func (*WorkerManager) BeforeStartingWorker 

func (m *WorkerManager) BeforeStartingWorker(
	workerID frameModel.WorkerID, executorID model.ExecutorID, epoch frameModel.Epoch,
)

BeforeStartingWorker is called by the BaseMaster BEFORE the executor runs the worker, but after the executor records the time at which the worker is submitted.

func (*WorkerManager) Close 

func (m *WorkerManager) Close()

Close closes the WorkerManager and waits all resource released.

func (*WorkerManager) GetWorkers 

func (m *WorkerManager) GetWorkers() map[frameModel.WorkerID]WorkerHandle

GetWorkers gets all workers maintained by WorkerManager, including both running workers and dead workers.

func (*WorkerManager) HandleHeartbeat 

func (m *WorkerManager) HandleHeartbeat(msg *frameModel.HeartbeatPingMessage, fromNode p2p.NodeID)

HandleHeartbeat handles heartbeat ping message from a worker

func (*WorkerManager) InitAfterRecover 

func (m *WorkerManager) InitAfterRecover(ctx context.Context) (retErr error)

InitAfterRecover should be called after the master has failed over. This method will block until a timeout period for heartbeats has passed.

func (*WorkerManager) IsInitialized 

func (m *WorkerManager) IsInitialized() bool

IsInitialized returns true after the worker manager has checked all tombstone workers are online or dead.

func (*WorkerManager) OnWorkerStatusUpdateMessage 

func (m *WorkerManager) OnWorkerStatusUpdateMessage(msg *statusutil.WorkerStatusMessage)

OnWorkerStatusUpdateMessage should be called in the message handler for WorkerStatusMessage.

func (*WorkerManager) Tick 

func (m *WorkerManager) Tick(ctx context.Context) error

Tick should be called by the BaseMaster so that the callbacks can be run in the main goroutine.

func (*WorkerManager) WithLogger 

func (m *WorkerManager) WithLogger(logger *zap.Logger) *WorkerManager

WithLogger passes a logger.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.