configs

Documentation

Index

• Constants
• Variables
• func GetConfigurationString(requestBytes []byte) string
• func MockSchedulerConfigByData(data []byte)
• func SetChecksum(content []byte, conf *SchedulerConfig)
• func Validate(newConfig *SchedulerConfig) error
• type ChildTemplate
• type ConfigReloader
• type ConfigWatcher
  • func CreateConfigWatcher(rmID string, policyGroup string, expiration time.Duration) *ConfigWatcher
  • func GetInstance() *ConfigWatcher
  • func (cw *ConfigWatcher) RegisterCallback(reloader ConfigReloader)
  • func (cw *ConfigWatcher) Run()
• type Filter
• type Limit
• type Limits
• type LoadSchedulerConfigFunc
• type NodeSortingPolicy
• type PartitionConfig
• type PartitionPreemptionConfig
• type PlacementRule
• type QueueConfig
• type Resources
• type SchedulerConfig
  • func LoadSchedulerConfigFromByteArray(content []byte) (*SchedulerConfig, error)
  • func ParseAndValidateConfig(content []byte) (*SchedulerConfig, error)
• type SchedulerConfigContext
  • func (ctx *SchedulerConfigContext) Get(policyGroup string) *SchedulerConfig
  • func (ctx *SchedulerConfigContext) Set(policyGroup string, config *SchedulerConfig)

Constants

const (
	SchedulerConfigPath        = "scheduler-config-path"
	DefaultSchedulerConfigPath = "/etc/yunikorn"
)

const (
	RootQueue        = "root"
	DOT              = "."
	DotReplace       = "_dot_"
	DefaultPartition = "default"
	// How to sort applications in leaf queues, valid options are defined in the scheduler.policies
	ApplicationSortPolicy = "application.sort.policy"
)

const (
	DefaultConfigWatcherDuration = 2 * time.Minute
)

Variables

var ConfigMap map[string]string

var GroupRegExp = regexp.MustCompile(`^[_a-zA-Z][a-zA-Z0-9_-]*$`)

Groups should have a slightly more restrictive regexp (no @ . or $ at the end)

var QueueNameRegExp = regexp.MustCompile(`^[a-zA-Z0-9_-]{1,64}$`)

A queue can be a username with the dot replaced. Most systems allow a 32 character user name. The queue name must thus allow for at least that length with the replacement of dots.

var RuleNameRegExp = regexp.MustCompile(`^[_a-zA-Z][a-zA-Z0-9_]*$`)

The rule maps to a go identifier check that regexp only

var SpecialRegExp = regexp.MustCompile(`[\^$*+?()\[{}|]`)

all characters that make a name different from a regexp

var UserRegExp = regexp.MustCompile(`^[_a-zA-Z][a-zA-Z0-9_.@-]*[$]?$`)

User and group name check: systems allow different things POSIX is the base but we need to be lenient and allow more. allow upper and lower case, add the @ and . (dot) and officially no length.

Functions

func GetConfigurationString

func GetConfigurationString(requestBytes []byte) string

func MockSchedulerConfigByData

func MockSchedulerConfigByData(data []byte)

func SetChecksum

func SetChecksum(content []byte, conf *SchedulerConfig)

func Validate

func Validate(newConfig *SchedulerConfig) error

Check the partition configuration. Any parsing issues will return an error which means that the configuration is invalid. This *must* be called before the configuration is activated. Any configuration that does not pass must be rejected. Check performed: - at least 1 partition must be defined - no more than 1 partition called "default" For the sub components: - The queue config is syntax checked - The placement rules are syntax checked - The user objects are syntax checked

Types

type ChildTemplate

type ChildTemplate struct {
	MaxApplications uint64            `yaml:",omitempty" json:",omitempty"`
	Properties      map[string]string `yaml:",omitempty" json:",omitempty"`
	Resources       Resources         `yaml:",omitempty" json:",omitempty"`
}

type ConfigReloader

type ConfigReloader interface {
	DoReloadConfiguration() error
}

interface for the actual reload function

type ConfigWatcher

type ConfigWatcher struct {
	// contains filtered or unexported fields
}

Config watcher watches on a configuration file, it is initiated with a certain expiration time, it will stop running either it detects configuration changes, or the expiration time reaches

func CreateConfigWatcher

func CreateConfigWatcher(rmID string, policyGroup string, expiration time.Duration) *ConfigWatcher

func GetInstance

func GetInstance() *ConfigWatcher

func (*ConfigWatcher) RegisterCallback

func (cw *ConfigWatcher) RegisterCallback(reloader ConfigReloader)

func (*ConfigWatcher) Run

func (cw *ConfigWatcher) Run()

if configWatcher is not running, kick-off running it if configWatcher is already running, this is a noop

type Filter

type Filter struct {
	Type   string
	Users  []string `yaml:",omitempty" json:",omitempty"`
	Groups []string `yaml:",omitempty" json:",omitempty"`
}

The user and group filter for a rule. - type of filter (allow or deny filter, empty means allow) - list of users to filter (maybe empty) - list of groups to filter (maybe empty) if the list of users or groups is exactly 1 long it is interpreted as a regular expression

type Limit

type Limit struct {
	Limit           string
	Users           []string          `yaml:",omitempty" json:",omitempty"`
	Groups          []string          `yaml:",omitempty" json:",omitempty"`
	MaxResources    map[string]string `yaml:",omitempty" json:",omitempty"`
	MaxApplications uint64            `yaml:",omitempty" json:",omitempty"`
}

The limit object to specify user and or group limits at different levels in the partition or queues Different limits for the same user or group may be defined at different levels in the hierarchy - limit description (optional) - list of users (maybe empty) - list of groups (maybe empty) - maximum resources as a resource object to allow for the user or group - maximum number of applications the user or group can have running

type Limits

type Limits struct {
	Limit []Limit
}

A list of limit objects to define limits for a partition or queue

type LoadSchedulerConfigFunc

type LoadSchedulerConfigFunc func(policyGroup string) (*SchedulerConfig, error)

var SchedulerConfigLoader LoadSchedulerConfigFunc = loadSchedulerConfigFromFile

Default loader, can be updated by tests

type NodeSortingPolicy

type NodeSortingPolicy struct {
	Type            string
	ResourceWeights map[string]float64 `yaml:",omitempty" json:",omitempty"`
}

Global Node Sorting Policy section - type: different type of policies supported (binpacking, fair etc)

type PartitionConfig

type PartitionConfig struct {
	Name              string
	Queues            []QueueConfig
	PlacementRules    []PlacementRule           `yaml:",omitempty" json:",omitempty"`
	Limits            []Limit                   `yaml:",omitempty" json:",omitempty"`
	Preemption        PartitionPreemptionConfig `yaml:",omitempty" json:",omitempty"`
	NodeSortPolicy    NodeSortingPolicy         `yaml:",omitempty" json:",omitempty"`
	StateDumpFilePath string                    `yaml:",omitempty" json:",omitempty"`
}

The partition object for each partition: - the name of the partition - a list of sub or child queues - a list of placement rule definition objects - a list of users specifying limits on the partition - the preemption configuration for the partition

type PartitionPreemptionConfig

type PartitionPreemptionConfig struct {
	Enabled bool
}

type PlacementRule

type PlacementRule struct {
	Name   string
	Create bool           `yaml:",omitempty" json:",omitempty"`
	Filter Filter         `yaml:",omitempty" json:",omitempty"`
	Parent *PlacementRule `yaml:",omitempty" json:",omitempty"`
	Value  string         `yaml:",omitempty" json:",omitempty"`
}

The queue placement rule definition - the name of the rule - create flag: can the rule create a queue - user and group filter to be applied on the callers - rule link to allow setting a rule to generate the parent - value a generic value interpreted depending on the rule type (i.e queue name for the "fixed" rule or the application label name for the "tag" rule)

type QueueConfig

type QueueConfig struct {
	Name            string
	Parent          bool              `yaml:",omitempty" json:",omitempty"`
	Resources       Resources         `yaml:",omitempty" json:",omitempty"`
	MaxApplications uint64            `yaml:",omitempty" json:",omitempty"`
	Properties      map[string]string `yaml:",omitempty" json:",omitempty"`
	AdminACL        string            `yaml:",omitempty" json:",omitempty"`
	SubmitACL       string            `yaml:",omitempty" json:",omitempty"`
	ChildTemplate   ChildTemplate     `yaml:",omitempty" json:",omitempty"`
	Queues          []QueueConfig     `yaml:",omitempty" json:",omitempty"`
	Limits          []Limit           `yaml:",omitempty" json:",omitempty"`
}

The queue object for each queue: - the name of the queue - a resources object to specify resource limits on the queue - the maximum number of applications that can run in the queue - a set of properties, exact definition of what can be set is not part of the yaml - ACL for submit and or admin access - a list of sub or child queues - a list of users specifying limits on a queue

type Resources

type Resources struct {
	Guaranteed map[string]string `yaml:",omitempty" json:",omitempty"`
	Max        map[string]string `yaml:",omitempty" json:",omitempty"`
}

The resource limits to set on the queue. The definition allows for an unlimited number of types to be used. The mapping to "known" resources is not handled here. - guaranteed resources - max resources

type SchedulerConfig

type SchedulerConfig struct {
	Partitions []PartitionConfig
	Checksum   string `yaml:",omitempty" json:",omitempty"`
}

The configuration can contain multiple partitions. Each partition contains the queue definition for a logical set of scheduler resources.

func LoadSchedulerConfigFromByteArray

func LoadSchedulerConfigFromByteArray(content []byte) (*SchedulerConfig, error)

Visible by tests

func ParseAndValidateConfig

func ParseAndValidateConfig(content []byte) (*SchedulerConfig, error)

type SchedulerConfigContext

type SchedulerConfigContext struct {
	// contains filtered or unexported fields
}

scheduler config context provides thread-safe access for scheduler configurations

var ConfigContext *SchedulerConfigContext

func (*SchedulerConfigContext) Get

func (ctx *SchedulerConfigContext) Get(policyGroup string) *SchedulerConfig

func (*SchedulerConfigContext) Set

func (ctx *SchedulerConfigContext) Set(policyGroup string, config *SchedulerConfig)