taskexchanger

Documentation

Index

• type E
  • func NewE(infProcessor *infprocessor.P, k8sClient k8sclient.Client, gRPCPort int, ...) *E
  • func (e *E) AddOrUpdateServerStatus(taskSenderSrv taskSenderSrv, status *v1.ServerStatus)
  • func (e *E) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)
  • func (e *E) RemoveServer(serverPodName string)
  • func (e *E) SetupWithManager(mgr ctrl.Manager) error

Types

type E

type E struct {
	// contains filtered or unexported fields
}

E is a task exchanger that allows multiple inference servers to exchange tasks.

It has two main components: task senders and task receivers. A task receiver is a gRPC client that connects to other server instances. It behaves like a gRPC client that an engine has. It receives tasks from other servers and scheduels them to the inference processor so that they are forwared to engines.

A task sender is a gRPC server that receives connections from other server instances. It registers engines reported by other services to the inference processor and sends tasks to a server if locally connected engines don't have a specified model.

The following shows the data flow of the task exchanger:

Engine registration:

local server's task receiver --> remote server's internal gRPC server --> remote server's task sender --> remote server's infprocessor.P

New task scheduling:

local server's HTTP handler --> local server's infprocessor.P --> local server's task sender --> remote server's task receiver --> remote server's infprocessor.P --> engine

Task result processing:

engine --> remote server's infprocessor.P --> remote server's task receiver --> local server's task sender --> local server's infprocessor.P --> local server's HTTP handler

func NewE

func NewE(
	infProcessor *infprocessor.P,
	k8sClient k8sclient.Client,
	gRPCPort int,
	localPodName string,
	podLabelKey string,
	podLabelValue string,
	logger logr.Logger,
) *E

NewE creates a new E.

func (*E) AddOrUpdateServerStatus

func (e *E) AddOrUpdateServerStatus(taskSenderSrv taskSenderSrv, status *v1.ServerStatus)

AddOrUpdateServerStatus adds or udpates the server status. A new task sender is created if needed.

func (*E) Reconcile

func (e *E) Reconcile(
	ctx context.Context,
	req ctrl.Request,
) (ctrl.Result, error)

Reconcile reconciles the pod by creating/deleting a corresponding task receiver.

func (*E) RemoveServer

func (e *E) RemoveServer(serverPodName string)

RemoveServer removes the server.

func (*E) SetupWithManager

func (e *E) SetupWithManager(mgr ctrl.Manager) error

SetupWithManager sets up the E with the manager.