Documentation
¶
Overview ¶
Package scheduler lets the jobqueue server interact with the local job scheduler (if any) to submit jobqueue clients and have them run on a compute cluster (or local machine). Examples of job schedulers are things like LSF and SGE.
It's a pseudo plug-in system in that it is designed so that you can easily add a go file that implements the methods of the scheduleri interface, to support a new job scheduler. On the other hand, there is no dynamic loading of these go files; they are all imported (they all belong to the scheduler package), and the correct one used at run time. To "register" a new scheduleri implementation you must add a case for it to New() and rebuild.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( ErrBadScheduler = "unknown scheduler name" ErrImpossible = "scheduler cannot accept the job, since its resource requirements are too high" )
Err* constants are found in the returned Errors under err.Err, so you can cast and check if it's a certain type of error.
Functions ¶
This section is empty.
Types ¶
type CmdStatus ¶
type CmdStatus struct {
Count int
Running [][2]int // a slice of [id, index] tuples
Pending [][2]int // ditto
Other [][2]int // ditto, for jobs in some strange state
}
CmdStatus lets you describe how many of a given cmd are already in the job scheduler, and gives the details of those jobs.
type ConfigLSF ¶
type ConfigLSF struct {
// deployment is one of "development" or "production".
Deployment string
// shell is the shell to use to run the commands to interact with your job
// scheduler; 'bash' is recommended.
Shell string
}
ConfigLSF represents the configuration options required by the LSF scheduler. All are required with no usable defaults.
type ConfigLocal ¶
type ConfigLocal struct {
// Shell is the shell to use to run your commands with; 'bash' is
// recommended.
Shell string
}
ConfigLocal represents the configuration options required by the local scheduler. All are required with no usable defaults.
type ConfigOpenStack ¶
type ConfigOpenStack struct {
// ResourceName is the resource name prefix used to name any resources (such
// as keys, security groups and servers) that need to be created.
ResourceName string
// OSPrefix is the prefix or full name of the Operating System image you
// wish spawned servers to run.
OSPrefix string
// OSUser is the login username of your chosen Operating System.
OSUser string
// OSRAM is the minimum RAM in MB needed to bring up a server instance that runs
// your Operating System image. It defaults to 2048.
OSRAM int
// ServerPorts are the TCP port numbers you need to be open for
// communication with any spawned servers. At a minimum you will need to
// specify []int{22}.
ServerPorts []int
// SavePath is an absolute path to a file on disk where details of any
// created resources can be read from and written to.
SavePath string
// ServerKeepTime is the time to wait before an idle server is destroyed.
// Zero duration means "never destroy due to being idle".
ServerKeepTime time.Duration
// MaxInstances is the maximum number of instances we are allowed to spawn.
// A 0 value (the default) means we will be limited by your quota, if any.
MaxInstances int
// Shell is the shell to use to run your commands with; 'bash' is
// recommended.
Shell string
}
ConfigOpenStack represents the configuration options required by the OpenStack scheduler. All are required with no usable defaults, unless otherwise noted.
type Error ¶
type Error struct {
Scheduler string // the scheduler's Name
Op string // name of the method
Err string // one of our Err* vars
}
Error records an error and the operation and scheduler that caused it.
type Requirements ¶
type Requirements struct {
RAM int // the expected peak RAM in MB Cmd will use while running
Time time.Duration // the expected time Cmd will take to run
Cores int // how many processor cores the Cmd will use
Disk int // the required local disk space in GB the Cmd needs to run
Other string // an arbitrary string that will be passed through to the job scheduler, defining further resource requirements
}
Requirements describes the resource requirements of the commands you want to run, so that when provided to a scheduler it will be able to schedule things appropriately.
type Scheduler ¶
Scheduler gives you access to all of the methods you'll need to interact with a job scheduler.
func New ¶
New creates a new Scheduler to interact with the given job scheduler. Possible names so far are "lsf", "local" and "openstack". You must also provide a config struct appropriate for your chosen scheduler, eg. for the local scheduler you will provide a ConfigLocal.
func (*Scheduler) Busy ¶
Busy reports true if there are any Schedule()d cmds still in the job scheduler's system. This is useful when testing and other situations where you want to avoid shutting down the server while there are still clients running/ about to run.
func (*Scheduler) Cleanup ¶
func (s *Scheduler) Cleanup()
Cleanup means you've finished using a scheduler and it can delete any remaining jobs in its system and clean up any other used resources.
func (*Scheduler) MaxQueueTime ¶
func (s *Scheduler) MaxQueueTime(req *Requirements) time.Duration
MaxQueueTime returns the maximum amount of time that jobs with the given resource requirements are allowed to run for in the job scheduler's queue. If the job scheduler doesn't have a queue system, or if the queue allows jobs to run forever, then this returns a 0 length duration, which should be regarded as "infinite" queue time.
func (*Scheduler) ReserveTimeout ¶
ReserveTimeout returns the number of seconds that runners spawned in this scheduler should wait for new jobs to appear in the manager's queue.
func (*Scheduler) Schedule ¶
func (s *Scheduler) Schedule(cmd string, req *Requirements, count int) error
Schedule gets your cmd scheduled in the job scheduler. You give it a command that you would like `count` identical instances of running via your job scheduler. If you already had `count` many scheduled, it will do nothing. If you had less than `count`, it will schedule more to run. If you have more than `count`, it will remove the appropriate number of scheduled (but not yet running) jobs that were previously scheduled for this same cmd (counts of 0 are legitimate - it will get rid of all non-running jobs for the cmd). If no error is returned, you know all `count` of your jobs are now scheduled and will eventually run unless you call Schedule() again with the same command and a lower count. NB: there is no guarantee that the jobs run successfully, and no feedback on their success or failure is given.
Source Files