scheduler

package
v0.0.0-...-2e51320 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 11, 2024 License: MIT Imports: 35 Imported by: 10

Documentation

Overview

Code generated by mockery v1.0.0. DO NOT EDIT.

Code generated by mockery v1.0.0. DO NOT EDIT.

Package scheduler contains Funnel's builtin compute scheduler and node.

Package scheduler is a reverse proxy.

It translates gRPC into RESTful JSON APIs.

Index

Constants

View Source 
const (
	SchedulerService_PutNode_FullMethodName    = "/scheduler.SchedulerService/PutNode"
	SchedulerService_DeleteNode_FullMethodName = "/scheduler.SchedulerService/DeleteNode"
	SchedulerService_ListNodes_FullMethodName  = "/scheduler.SchedulerService/ListNodes"
	SchedulerService_GetNode_FullMethodName    = "/scheduler.SchedulerService/GetNode"
)
View Source 
const (
	CPU = "cpu"
	RAM = "ram"
)

Scores keys

Variables

View Source 
var (
	NodeState_name = map[int32]string{
		0: "UNINITIALIZED",
		1: "ALIVE",
		2: "DEAD",
		3: "GONE",
		4: "INITIALIZING",
		5: "DRAIN",
	}
	NodeState_value = map[string]int32{
		"UNINITIALIZED": 0,
		"ALIVE":         1,
		"DEAD":          2,
		"GONE":          3,
		"INITIALIZING":  4,
		"DRAIN":         5,
	}
)

Enum value maps for NodeState.

View Source 
var DefaultPredicates = []Predicate{
	ResourcesFit,
	ZonesFit,
	NotDead,
	Alive,
}

DefaultPredicates is a list of Predicate functions that check the whether a task fits a node.

View Source 
var File_scheduler_proto protoreflect.FileDescriptor
View Source 
var SchedulerService_ServiceDesc = grpc.ServiceDesc{
	ServiceName: "scheduler.SchedulerService",
	HandlerType: (*SchedulerServiceServer)(nil),
	Methods: []grpc.MethodDesc{
		{
			MethodName: "PutNode",
			Handler:    _SchedulerService_PutNode_Handler,
		},
		{
			MethodName: "DeleteNode",
			Handler:    _SchedulerService_DeleteNode_Handler,
		},
		{
			MethodName: "ListNodes",
			Handler:    _SchedulerService_ListNodes_Handler,
		},
		{
			MethodName: "GetNode",
			Handler:    _SchedulerService_GetNode_Handler,
		},
	},
	Streams:  []grpc.StreamDesc{},
	Metadata: "scheduler.proto",
}

SchedulerService_ServiceDesc is the grpc.ServiceDesc for SchedulerService service. It's only intended for direct use with grpc.RegisterService, and not to be introspected or modified (even as a copy)

Functions

func Alive 

func Alive(j *tes.Task, n *Node) error

Alive returns true if the node state is not Dead or Gone.

func GenNodeID 

func GenNodeID() string

GenNodeID returns a UUID string.

func Match 

func Match(node *Node, task *tes.Task, predicates []Predicate) bool

Match checks whether a task fits a node using the given Predicate list.

func NoopWorker 

func NoopWorker(ctx context.Context, taskID string) error

NoopWorker does nothing.

func NotDead 

func NotDead(j *tes.Task, n *Node) error

NotDead returns true if the node state is not Dead or Gone.

func RegisterSchedulerServiceHandler 

func RegisterSchedulerServiceHandler(ctx context.Context, mux *runtime.ServeMux, conn *grpc.ClientConn) error

RegisterSchedulerServiceHandler registers the http handlers for service SchedulerService to "mux". The handlers forward requests to the grpc endpoint over "conn".

func RegisterSchedulerServiceHandlerClient 

func RegisterSchedulerServiceHandlerClient(ctx context.Context, mux *runtime.ServeMux, client SchedulerServiceClient) error

RegisterSchedulerServiceHandlerClient registers the http handlers for service SchedulerService to "mux". The handlers forward requests to the grpc endpoint over the given implementation of "SchedulerServiceClient". Note: the gRPC framework executes interceptors within the gRPC handler. If the passed in "SchedulerServiceClient" doesn't go through the normal gRPC flow (creating a gRPC client etc.) then it will be up to the passed in "SchedulerServiceClient" to call the correct interceptors.

func RegisterSchedulerServiceHandlerFromEndpoint 

func RegisterSchedulerServiceHandlerFromEndpoint(ctx context.Context, mux *runtime.ServeMux, endpoint string, opts []grpc.DialOption) (err error)

RegisterSchedulerServiceHandlerFromEndpoint is same as RegisterSchedulerServiceHandler but automatically dials to "endpoint" and closes the connection when "ctx" gets done.

func RegisterSchedulerServiceHandlerServer 

func RegisterSchedulerServiceHandlerServer(ctx context.Context, mux *runtime.ServeMux, server SchedulerServiceServer) error

RegisterSchedulerServiceHandlerServer registers the http handlers for service SchedulerService to "mux". UnaryRPC :call SchedulerServiceServer directly. StreamingRPC :currently unsupported pending https://github.com/grpc/grpc-go/issues/906. Note that using this registration option will cause many gRPC library features to stop working. Consider using RegisterSchedulerServiceHandlerFromEndpoint instead.

func RegisterSchedulerServiceServer 

func RegisterSchedulerServiceServer(s grpc.ServiceRegistrar, srv SchedulerServiceServer)

func ResourcesFit 

func ResourcesFit(t *tes.Task, n *Node) error

ResourcesFit determines whether a task fits a node's resources.

func SortByAverageScore 

func SortByAverageScore(offers []*Offer)

SortByAverageScore sorts the given offers by their average score. This modifies the offers list in place.

func UpdateNode 

func UpdateNode(ctx context.Context, cli tes.ReadOnlyServer, node, existing *Node) error

UpdateNode helps scheduler database backend update a node when PutNode() is called.

func ZonesFit 

func ZonesFit(t *tes.Task, n *Node) error

ZonesFit determines whether a task's zones fit a node.

Types

type Client 

type Client interface {
	events.EventServiceClient
	SchedulerServiceClient
	Close()
}

Client is a client for the scheduler and event gRPC services.

func NewClient 

func NewClient(ctx context.Context, conf config.RPCClient) (Client, error)

NewClient returns a new Client instance connected to the scheduler and task logger services at a given address (e.g. "localhost:9090")

type DeleteNodeResponse 

type DeleteNodeResponse struct {
	// contains filtered or unexported fields
}

func (*DeleteNodeResponse) Descriptor deprecated 

func (*DeleteNodeResponse) Descriptor() ([]byte, []int)

Deprecated: Use DeleteNodeResponse.ProtoReflect.Descriptor instead.

func (*DeleteNodeResponse) ProtoMessage 

func (*DeleteNodeResponse) ProtoMessage()

func (*DeleteNodeResponse) ProtoReflect 

func (x *DeleteNodeResponse) ProtoReflect() protoreflect.Message

func (*DeleteNodeResponse) Reset 

func (x *DeleteNodeResponse) Reset()

func (*DeleteNodeResponse) String 

func (x *DeleteNodeResponse) String() string

type GetNodeRequest 

type GetNodeRequest struct {
	Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"`
	// contains filtered or unexported fields
}

func (*GetNodeRequest) Descriptor deprecated 

func (*GetNodeRequest) Descriptor() ([]byte, []int)

Deprecated: Use GetNodeRequest.ProtoReflect.Descriptor instead.

func (*GetNodeRequest) GetId 

func (x *GetNodeRequest) GetId() string

func (*GetNodeRequest) ProtoMessage 

func (*GetNodeRequest) ProtoMessage()

func (*GetNodeRequest) ProtoReflect 

func (x *GetNodeRequest) ProtoReflect() protoreflect.Message

func (*GetNodeRequest) Reset 

func (x *GetNodeRequest) Reset()

func (*GetNodeRequest) String 

func (x *GetNodeRequest) String() string

type ListNodesRequest 

type ListNodesRequest struct {
	// contains filtered or unexported fields
}

func (*ListNodesRequest) Descriptor deprecated 

func (*ListNodesRequest) Descriptor() ([]byte, []int)

Deprecated: Use ListNodesRequest.ProtoReflect.Descriptor instead.

func (*ListNodesRequest) ProtoMessage 

func (*ListNodesRequest) ProtoMessage()

func (*ListNodesRequest) ProtoReflect 

func (x *ListNodesRequest) ProtoReflect() protoreflect.Message

func (*ListNodesRequest) Reset 

func (x *ListNodesRequest) Reset()

func (*ListNodesRequest) String 

func (x *ListNodesRequest) String() string

type ListNodesResponse 

type ListNodesResponse struct {
	Nodes []*Node `protobuf:"bytes,1,rep,name=nodes,proto3" json:"nodes,omitempty"`
	// contains filtered or unexported fields
}

func (*ListNodesResponse) Descriptor deprecated 

func (*ListNodesResponse) Descriptor() ([]byte, []int)

Deprecated: Use ListNodesResponse.ProtoReflect.Descriptor instead.

func (*ListNodesResponse) GetNodes 

func (x *ListNodesResponse) GetNodes() []*Node

func (*ListNodesResponse) ProtoMessage 

func (*ListNodesResponse) ProtoMessage()

func (*ListNodesResponse) ProtoReflect 

func (x *ListNodesResponse) ProtoReflect() protoreflect.Message

func (*ListNodesResponse) Reset 

func (x *ListNodesResponse) Reset()

func (*ListNodesResponse) String 

func (x *ListNodesResponse) String() string

type MockClient 

type MockClient struct {
	mock.Mock
}

MockClient is an autogenerated mock type for the Client type

func (*MockClient) Close 

func (_m *MockClient) Close()

Close provides a mock function with given fields:

func (*MockClient) DeleteNode 

func (_m *MockClient) DeleteNode(ctx context.Context, in *Node, opts ...grpc.CallOption) (*DeleteNodeResponse, error)

DeleteNode provides a mock function with given fields: ctx, in, opts

func (*MockClient) GetNode 

func (_m *MockClient) GetNode(ctx context.Context, in *GetNodeRequest, opts ...grpc.CallOption) (*Node, error)

GetNode provides a mock function with given fields: ctx, in, opts

func (*MockClient) ListNodes 

func (_m *MockClient) ListNodes(ctx context.Context, in *ListNodesRequest, opts ...grpc.CallOption) (*ListNodesResponse, error)

ListNodes provides a mock function with given fields: ctx, in, opts

func (*MockClient) PutNode 

func (_m *MockClient) PutNode(ctx context.Context, in *Node, opts ...grpc.CallOption) (*PutNodeResponse, error)

PutNode provides a mock function with given fields: ctx, in, opts

func (*MockClient) WriteEvent 

func (_m *MockClient) WriteEvent(ctx context.Context, in *events.Event, opts ...grpc.CallOption) (*events.WriteEventResponse, error)

WriteEvent provides a mock function with given fields: ctx, in, opts

type MockSchedulerServiceServer 

type MockSchedulerServiceServer struct {
	UnimplementedSchedulerServiceServer
	mock.Mock
}

MockSchedulerServiceServer is an autogenerated mock type for the SchedulerServiceServer type

func (*MockSchedulerServiceServer) DeleteNode 

func (_m *MockSchedulerServiceServer) DeleteNode(_a0 context.Context, _a1 *Node) (*DeleteNodeResponse, error)

DeleteNode provides a mock function with given fields: _a0, _a1

func (*MockSchedulerServiceServer) GetNode 

func (_m *MockSchedulerServiceServer) GetNode(_a0 context.Context, _a1 *GetNodeRequest) (*Node, error)

GetNode provides a mock function with given fields: _a0, _a1

func (*MockSchedulerServiceServer) ListNodes 

func (_m *MockSchedulerServiceServer) ListNodes(_a0 context.Context, _a1 *ListNodesRequest) (*ListNodesResponse, error)

ListNodes provides a mock function with given fields: _a0, _a1

func (*MockSchedulerServiceServer) PutNode 

func (_m *MockSchedulerServiceServer) PutNode(_a0 context.Context, _a1 *Node) (*PutNodeResponse, error)

PutNode provides a mock function with given fields: _a0, _a1

type Node 

type Node struct {
	Id          string     `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"`
	Resources   *Resources `protobuf:"bytes,5,opt,name=resources,proto3" json:"resources,omitempty"`
	Available   *Resources `protobuf:"bytes,6,opt,name=available,proto3" json:"available,omitempty"`
	State       NodeState  `protobuf:"varint,8,opt,name=state,proto3,enum=scheduler.NodeState" json:"state,omitempty"`
	Preemptible bool       `protobuf:"varint,9,opt,name=preemptible,proto3" json:"preemptible,omitempty"`
	Zone        string     `protobuf:"bytes,11,opt,name=zone,proto3" json:"zone,omitempty"`
	Hostname    string     `protobuf:"bytes,13,opt,name=hostname,proto3" json:"hostname,omitempty"`
	// Timestamp version of the record in the database. Used to prevent write conflicts and as the last ping time.
	Version  int64             `protobuf:"varint,14,opt,name=version,proto3" json:"version,omitempty"`
	Metadata map[string]string `` /* 158-byte string literal not displayed */
	TaskIds  []string          `protobuf:"bytes,16,rep,name=task_ids,json=taskIds,proto3" json:"task_ids,omitempty"`
	LastPing int64             `protobuf:"varint,17,opt,name=last_ping,json=lastPing,proto3" json:"last_ping,omitempty"`
	// contains filtered or unexported fields
}

func UpdateNodeState 

func UpdateNodeState(nodes []*Node, conf config.Scheduler) []*Node

UpdateNodeState checks whether a node is dead/gone based on the last time it pinged.

func (*Node) Descriptor deprecated 

func (*Node) Descriptor() ([]byte, []int)

Deprecated: Use Node.ProtoReflect.Descriptor instead.

func (*Node) GetAvailable 

func (x *Node) GetAvailable() *Resources

func (*Node) GetHostname 

func (x *Node) GetHostname() string

func (*Node) GetId 

func (x *Node) GetId() string

func (*Node) GetLastPing 

func (x *Node) GetLastPing() int64

func (*Node) GetMetadata 

func (x *Node) GetMetadata() map[string]string

func (*Node) GetPreemptible 

func (x *Node) GetPreemptible() bool

func (*Node) GetResources 

func (x *Node) GetResources() *Resources

func (*Node) GetState 

func (x *Node) GetState() NodeState

func (*Node) GetTaskIds 

func (x *Node) GetTaskIds() []string

func (*Node) GetVersion 

func (x *Node) GetVersion() int64

func (*Node) GetZone 

func (x *Node) GetZone() string

func (*Node) ProtoMessage 

func (*Node) ProtoMessage()

func (*Node) ProtoReflect 

func (x *Node) ProtoReflect() protoreflect.Message

func (*Node) Reset 

func (x *Node) Reset()

func (*Node) String 

func (x *Node) String() string

type NodeProcess 

type NodeProcess struct {
	// contains filtered or unexported fields
}

NodeProcess is a structure used for tracking available resources on a compute resource.

func NewNodeProcess 

func NewNodeProcess(ctx context.Context, conf config.Config, factory Worker, log *logger.Logger) (*NodeProcess, error)

NewNodeProcess returns a new Node instance

func (*NodeProcess) Drain 

func (n *NodeProcess) Drain()

Drain sets the node state to DRAIN.

func (*NodeProcess) Run 

func (n *NodeProcess) Run(ctx context.Context) error

Run runs a node with the given config. This is responsible for communication with the server and starting task workers

type NodeState 

type NodeState int32
const (
	NodeState_UNINITIALIZED NodeState = 0
	NodeState_ALIVE         NodeState = 1
	NodeState_DEAD          NodeState = 2
	NodeState_GONE          NodeState = 3
	NodeState_INITIALIZING  NodeState = 4
	NodeState_DRAIN         NodeState = 5
)

func (NodeState) Descriptor 

func (NodeState) Descriptor() protoreflect.EnumDescriptor

func (NodeState) Enum 

func (x NodeState) Enum() *NodeState

func (NodeState) EnumDescriptor deprecated 

func (NodeState) EnumDescriptor() ([]byte, []int)

Deprecated: Use NodeState.Descriptor instead.

func (NodeState) Number 

func (x NodeState) Number() protoreflect.EnumNumber

func (NodeState) String 

func (x NodeState) String() string

func (NodeState) Type 

func (NodeState) Type() protoreflect.EnumType

type Offer 

type Offer struct {
	TaskID string
	Node   *Node
	Scores Scores
}

Offer describes a node offered by a scheduler for a task. The Scores describe how well the task fits this node, which could be used by other a scheduler to pick the best offer.

func DefaultScheduleAlgorithm 

func DefaultScheduleAlgorithm(j *tes.Task, nodes []*Node, weights map[string]float32) *Offer

DefaultScheduleAlgorithm implements a simple scheduling algorithm that is (currently) common across a few scheduler backends. Given a task, list of nodes, and weights, it returns the best Offer or nil.

func NewOffer 

func NewOffer(n *Node, t *tes.Task, s Scores) *Offer

NewOffer returns a new Offer instance.

type Predicate 

type Predicate func(*tes.Task, *Node) error

Predicate is a function that checks whether a task fits a node.

func NodeHasTag 

func NodeHasTag(tag string) Predicate

NodeHasTag returns a predicate function which returns true if the node has the given tag (key in Metadata field).

type PutNodeResponse 

type PutNodeResponse struct {
	// contains filtered or unexported fields
}

func (*PutNodeResponse) Descriptor deprecated 

func (*PutNodeResponse) Descriptor() ([]byte, []int)

Deprecated: Use PutNodeResponse.ProtoReflect.Descriptor instead.

func (*PutNodeResponse) ProtoMessage 

func (*PutNodeResponse) ProtoMessage()

func (*PutNodeResponse) ProtoReflect 

func (x *PutNodeResponse) ProtoReflect() protoreflect.Message

func (*PutNodeResponse) Reset 

func (x *PutNodeResponse) Reset()

func (*PutNodeResponse) String 

func (x *PutNodeResponse) String() string

type Resources 

type Resources struct {
	Cpus uint32 `protobuf:"varint,1,opt,name=cpus,proto3" json:"cpus,omitempty"`
	// In GB
	RamGb float64 `protobuf:"fixed64,2,opt,name=ram_gb,json=ramGb,proto3" json:"ram_gb,omitempty"`
	// In GB
	DiskGb float64 `protobuf:"fixed64,3,opt,name=disk_gb,json=diskGb,proto3" json:"disk_gb,omitempty"`
	// contains filtered or unexported fields
}

func AvailableResources 

func AvailableResources(tasks []*tes.Task, res *Resources) *Resources

AvailableResources calculates available resources given a list of tasks and base resources.

func SubtractResources 

func SubtractResources(t *tes.Task, in *Resources) *Resources

SubtractResources subtracts the resources requested by "task" from the node resources "in".

func (*Resources) Descriptor deprecated 

func (*Resources) Descriptor() ([]byte, []int)

Deprecated: Use Resources.ProtoReflect.Descriptor instead.

func (*Resources) GetCpus 

func (x *Resources) GetCpus() uint32

func (*Resources) GetDiskGb 

func (x *Resources) GetDiskGb() float64

func (*Resources) GetRamGb 

func (x *Resources) GetRamGb() float64

func (*Resources) ProtoMessage 

func (*Resources) ProtoMessage()

func (*Resources) ProtoReflect 

func (x *Resources) ProtoReflect() protoreflect.Message

func (*Resources) Reset 

func (x *Resources) Reset()

func (*Resources) String 

func (x *Resources) String() string

type Scheduler 

type Scheduler struct {
	Conf  config.Scheduler
	Log   *logger.Logger
	Nodes SchedulerServiceServer
	Queue TaskQueue
	Event events.Writer
}

Scheduler handles scheduling tasks to nodes and support many backends.

func (*Scheduler) CheckNodes 

func (s *Scheduler) CheckNodes() error

CheckNodes is used by the scheduler to check for dead/gone nodes. This is not an RPC endpoint

func (*Scheduler) GetOffer 

func (s *Scheduler) GetOffer(j *tes.Task) *Offer

GetOffer returns an offer based on available funnel nodes.

func (*Scheduler) Run 

func (s *Scheduler) Run(ctx context.Context) error

Run starts the scheduling loop. This blocks.

The scheduler will take a chunk of tasks from the queue, request the the configured backend schedule them, and act on offers made by the backend.

func (*Scheduler) Schedule 

func (s *Scheduler) Schedule(ctx context.Context) error

Schedule does a scheduling iteration. It checks the health of nodes in the database, gets a chunk of tasks from the queue (configurable by config.ScheduleChunk), and calls the given scheduler backend. If the backend returns a valid offer, the task is assigned to the offered node.

type SchedulerServiceClient 

type SchedulerServiceClient interface {
	PutNode(ctx context.Context, in *Node, opts ...grpc.CallOption) (*PutNodeResponse, error)
	DeleteNode(ctx context.Context, in *Node, opts ...grpc.CallOption) (*DeleteNodeResponse, error)
	ListNodes(ctx context.Context, in *ListNodesRequest, opts ...grpc.CallOption) (*ListNodesResponse, error)
	GetNode(ctx context.Context, in *GetNodeRequest, opts ...grpc.CallOption) (*Node, error)
}

SchedulerServiceClient is the client API for SchedulerService service.

For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.

func NewSchedulerServiceClient 

func NewSchedulerServiceClient(cc grpc.ClientConnInterface) SchedulerServiceClient

type SchedulerServiceServer 

type SchedulerServiceServer interface {
	PutNode(context.Context, *Node) (*PutNodeResponse, error)
	DeleteNode(context.Context, *Node) (*DeleteNodeResponse, error)
	ListNodes(context.Context, *ListNodesRequest) (*ListNodesResponse, error)
	GetNode(context.Context, *GetNodeRequest) (*Node, error)
	// contains filtered or unexported methods
}

SchedulerServiceServer is the server API for SchedulerService service. All implementations must embed UnimplementedSchedulerServiceServer for forward compatibility

type Scores 

type Scores map[string]float32

Scores describe how well a task fits a node.

func DefaultScores 

func DefaultScores(w *Node, t *tes.Task) Scores

DefaultScores returns a default set of scores.

func (Scores) Average 

func (s Scores) Average() float32

Average returns the average of the scores.

func (Scores) Weighted 

func (s Scores) Weighted(w map[string]float32) Scores

Weighted returns a new Scores instance with each score multiplied by the given weights. Weights default to 0.0

type TaskQueue 

type TaskQueue interface {
	ReadQueue(count int) []*tes.Task
}

TaskQueue describes the interface the scheduler uses to find tasks that need scheduling.

type UnimplementedSchedulerServiceServer 

type UnimplementedSchedulerServiceServer struct {
}

UnimplementedSchedulerServiceServer must be embedded to have forward compatible implementations.

func (UnimplementedSchedulerServiceServer) DeleteNode 

func (UnimplementedSchedulerServiceServer) DeleteNode(context.Context, *Node) (*DeleteNodeResponse, error)

func (UnimplementedSchedulerServiceServer) GetNode 

func (UnimplementedSchedulerServiceServer) GetNode(context.Context, *GetNodeRequest) (*Node, error)

func (UnimplementedSchedulerServiceServer) ListNodes 

func (UnimplementedSchedulerServiceServer) ListNodes(context.Context, *ListNodesRequest) (*ListNodesResponse, error)

func (UnimplementedSchedulerServiceServer) PutNode 

func (UnimplementedSchedulerServiceServer) PutNode(context.Context, *Node) (*PutNodeResponse, error)

type UnsafeSchedulerServiceServer 

type UnsafeSchedulerServiceServer interface {
	// contains filtered or unexported methods
}

UnsafeSchedulerServiceServer may be embedded to opt out of forward compatibility for this service. Use of this interface is not recommended, as added methods to SchedulerServiceServer will result in compilation errors.

type Worker 

type Worker func(ctx context.Context, taskID string) error

Worker is a function which creates a new worker instance.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.