scheduler

Documentation

Overview

Package scheduler uses a resizable worker pool to execute containers in priority order.

Index

• type ContainerQueue
• type QueueEnt
• type Scheduler
  • func New(ctx context.Context, client *arvados.Client, queue ContainerQueue, ...) *Scheduler
  • func (sch *Scheduler) Queue() []QueueEnt
  • func (sch *Scheduler) Start()
  • func (sch *Scheduler) Stop()
• type WorkerPool

Types

type ContainerQueue

type ContainerQueue interface {
	Entries() (entries map[string]container.QueueEnt, updated time.Time)
	Lock(uuid string) error
	Unlock(uuid string) error
	Cancel(uuid string) error
	Forget(uuid string)
	Get(uuid string) (arvados.Container, bool)
	Subscribe() <-chan struct{}
	Unsubscribe(<-chan struct{})
	Update() error
}

A ContainerQueue is a set of containers that need to be started or stopped. Implemented by container.Queue and test stubs. See container.Queue method documentation for details.

type QueueEnt

type QueueEnt struct {
	container.QueueEnt

	// Human-readable scheduling status as of the last scheduling
	// iteration.
	SchedulingStatus string `json:"scheduling_status"`
}

type Scheduler

type Scheduler struct {
	// contains filtered or unexported fields
}

A Scheduler maps queued containers onto unallocated workers in priority order, creating new workers if needed. It locks containers that can be mapped onto existing/pending workers, and starts them if possible.

A Scheduler unlocks any containers that are locked but can't be mapped. (For example, this happens when the cloud provider reaches quota/capacity and a previously mappable container's priority is surpassed by a newer container.)

If it encounters errors while creating new workers, a Scheduler shuts down idle workers, in case they are consuming quota.

func New

func New(ctx context.Context, client *arvados.Client, queue ContainerQueue, pool WorkerPool, reg *prometheus.Registry, staleLockTimeout, queueUpdateInterval time.Duration, minQuota, maxInstances int, supervisorFraction float64) *Scheduler

New returns a new unstarted Scheduler.

Any given queue and pool should not be used by more than one scheduler at a time.

func (*Scheduler) Queue

func (sch *Scheduler) Queue() []QueueEnt

Queue returns the sorted queue from the last scheduling iteration.

func (*Scheduler) Start

func (sch *Scheduler) Start()

Start starts the scheduler.

func (*Scheduler) Stop

func (sch *Scheduler) Stop()

Stop stops the scheduler. No other method should be called after Stop.

type WorkerPool

type WorkerPool interface {
	Running() map[string]time.Time
	Unallocated() map[arvados.InstanceType]int
	CountWorkers() map[worker.State]int
	AtCapacity(arvados.InstanceType) bool
	AtQuota() bool
	Create(arvados.InstanceType) bool
	Shutdown(arvados.InstanceType) bool
	StartContainer(arvados.InstanceType, arvados.Container) bool
	KillContainer(uuid, reason string) bool
	ForgetContainer(uuid string)
	Subscribe() <-chan struct{}
	Unsubscribe(<-chan struct{})
}

A WorkerPool asynchronously starts and stops worker VMs, and starts and stops containers on them. Implemented by worker.Pool and test stubs. See worker.Pool method documentation for details.