objects

package
v1.6.0-1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 10, 2024 License: Apache-2.0 Imports: 31 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	Soft string = "Soft"
	Hard string = "Hard"

	NotEnoughUserQuota  = "Not enough user quota"
	NotEnoughQueueQuota = "Not enough queue quota"
)
View Source 
const (
	RunApplication applicationEvent = iota
	RejectApplication
	CompleteApplication
	FailApplication
	ExpireApplication
	ResumeApplication
)
View Source 
const (
	New applicationState = iota
	Accepted
	Running
	Rejected
	Completing
	Completed
	Failing
	Failed
	Expired
	Resuming
)

Application states are used for filtering in the webservice handlers. Please check&update the logic as needed if the state machine is modified

View Source 
const (
	EventNotNeeded = -1
)
View Source 
const (
	UnknownInstanceType = "UNKNOWN"
)

Variables

This section is empty.

Functions

func CloneAllocationTags added in v1.1.0 

func CloneAllocationTags(tags map[string]string) map[string]string

CloneAllocationTags clones a tag map for safe copying.

func NewAppState 

func NewAppState() *fsm.FSM

func NewObjectState 

func NewObjectState() *fsm.FSM

func NewTreeIterator added in v1.4.0 

func NewTreeIterator(accept func(*Node) bool, getTree func() *btree.BTree) *treeIterator

func SetCompletingTimeout 

func SetCompletingTimeout(duration time.Duration)

SetCompletingTimeout should be used for testing only.

func SetReservationDelay 

func SetReservationDelay(delay time.Duration)

Set the reservation delay. Set when the cluster context is created to disable reservation.

Types

type Allocation 

type Allocation struct {
	locking.RWMutex
	// contains filtered or unexported fields
}

func NewAllocationFromSI 

func NewAllocationFromSI(alloc *si.Allocation) *Allocation

NewAllocationFromSI Create a new Allocation which has already been placed on a node, populating it with info from the SI Allocation object. If the input object is invalid, nil is returned.

func (*Allocation) ClearRelease added in v1.6.0 

func (a *Allocation) ClearRelease()

ClearRelease removes any release from this allocation.

func (*Allocation) GetAllocatedResource added in v1.1.0 

func (a *Allocation) GetAllocatedResource() *resources.Resource

GetAllocatedResource returns a reference to the allocated resources for this allocation. This must be treated as read-only.

func (*Allocation) GetAllocationKey added in v1.1.0 

func (a *Allocation) GetAllocationKey() string

GetAllocationKey returns the allocation key for this allocation.

func (*Allocation) GetAllocationLog added in v1.6.0 

func (a *Allocation) GetAllocationLog() []*AllocationLogEntry

GetAllocationLog returns a list of log entries corresponding to allocation preconditions not being met.

func (*Allocation) GetApplicationID added in v1.1.0 

func (a *Allocation) GetApplicationID() string

GetApplicationID returns the application ID for this allocation.

func (*Allocation) GetBindTime added in v1.3.0 

func (a *Allocation) GetBindTime() time.Time

GetBindTime returns the time this allocation was bound.

func (*Allocation) GetCreateTime 

func (a *Allocation) GetCreateTime() time.Time

GetCreateTime returns the time this allocation was created.

func (*Allocation) GetInstanceType added in v1.3.0 

func (a *Allocation) GetInstanceType() string

GetInstanceType return the type of the instance used by this allocation.

func (*Allocation) GetNodeID added in v1.1.0 

func (a *Allocation) GetNodeID() string

GetNodeID gets the node this allocation is assigned to.

func (*Allocation) GetPlaceholderCreateTime 

func (a *Allocation) GetPlaceholderCreateTime() time.Time

GetPlaceholderCreateTime returns the placeholder's create time for this allocation, if applicable.

func (*Allocation) GetPreemptCheckTime added in v1.6.0 

func (a *Allocation) GetPreemptCheckTime() time.Time

GetPreemptCheckTime returns the time this allocation was last evaluated for preemption.

func (*Allocation) GetPriority added in v1.1.0 

func (a *Allocation) GetPriority() int32

GetPriority returns the priority of this allocation.

func (*Allocation) GetRelease added in v1.6.0 

func (a *Allocation) GetRelease() *Allocation

GetRelease returns the associated release for this allocation.

func (*Allocation) GetRequiredNode added in v1.6.0 

func (a *Allocation) GetRequiredNode() string

GetRequiredNode gets the node (if any) required by this allocation.

func (*Allocation) GetTag added in v1.6.0 

func (a *Allocation) GetTag(tagName string) string

GetTag returns the value of a named tag or an empty string if not present.

func (*Allocation) GetTagsClone added in v1.1.0 

func (a *Allocation) GetTagsClone() map[string]string

GetTagsClone returns the copy of the tags for this allocation.

func (*Allocation) GetTaskGroup added in v1.1.0 

func (a *Allocation) GetTaskGroup() string

GetTaskGroup returns the task group name for this allocation.

func (*Allocation) HasRelease added in v1.6.0 

func (a *Allocation) HasRelease() bool

HasRelease determines if this allocation has an associated release.

func (*Allocation) HasTriggeredPreemption added in v1.6.0 

func (a *Allocation) HasTriggeredPreemption() bool

HasTriggeredPreemption returns whether this allocation has triggered preemption.

func (*Allocation) HasTriggeredScaleUp added in v1.6.0 

func (a *Allocation) HasTriggeredScaleUp() bool

HasTriggeredScaleUp determines if this allocation has triggered auto-scaling.

func (*Allocation) IsAllocated added in v1.6.0 

func (a *Allocation) IsAllocated() bool

IsAllocated determines if this request has been allocated yet.

func (*Allocation) IsAllowPreemptOther added in v1.6.0 

func (a *Allocation) IsAllowPreemptOther() bool

IsAllowPreemptOther returns whether this allocation is allowed to preempt others.

func (*Allocation) IsAllowPreemptSelf added in v1.6.0 

func (a *Allocation) IsAllowPreemptSelf() bool

IsAllowPreemptSelf returns whether preemption is allowed for this allocation.

func (*Allocation) IsOriginator added in v1.6.0 

func (a *Allocation) IsOriginator() bool

IsOriginator returns whether this alloocation is the originator for the application.

func (*Allocation) IsPlaceholder 

func (a *Allocation) IsPlaceholder() bool

IsPlaceholder returns whether this allocation represents a placeholder.

func (*Allocation) IsPlaceholderUsed added in v1.1.0 

func (a *Allocation) IsPlaceholderUsed() bool

IsPlaceholderUsed returns whether this allocation is replacing a placeholder.

func (*Allocation) IsPreempted added in v1.2.0 

func (a *Allocation) IsPreempted() bool

IsPreempted returns whether the allocation has been marked for preemption or not.

func (*Allocation) IsReleased 

func (a *Allocation) IsReleased() bool

IsReleased returns the release status of the allocation.

func (*Allocation) IsSchedulingAttempted added in v1.6.0 

func (a *Allocation) IsSchedulingAttempted() bool

IsSchedulingAttempted determines whether scheduling has been attempted at least once for this allocation.

func (*Allocation) LessThan added in v1.6.0 

func (a *Allocation) LessThan(other *Allocation) bool

LessThan compares two allocations by priority and then creation time.

func (*Allocation) LogAllocationFailure added in v1.6.0 

func (a *Allocation) LogAllocationFailure(message string, allocate bool)

LogAllocationFailure keeps track of preconditions not being met for an allocation.

func (*Allocation) MarkPreempted added in v1.2.0 

func (a *Allocation) MarkPreempted()

MarkPreempted marks the allocation as preempted.

func (*Allocation) MarkTriggeredPreemption added in v1.6.0 

func (a *Allocation) MarkTriggeredPreemption()

MarkTriggeredPreemption marks the current allocation because it triggered preemption during scheduling.

func (*Allocation) NewSIFromAllocation 

func (a *Allocation) NewSIFromAllocation() *si.Allocation

NewSIFromAllocation converts the Allocation into a SI object. This is a limited set of values that gets copied into the SI. This is only used to communicate *back* to the RM. All other fields are considered incoming fields from the RM into the core. The limited set of fields link the Allocation to an Application and Node.

func (*Allocation) SendPredicatesFailedEvent added in v1.6.0 

func (a *Allocation) SendPredicatesFailedEvent(predicateErrors map[string]int)

SendPredicatesFailedEvent updates the event system with the reason for predicate failures. The map predicateErrors contains how many times certain predicates failed in the scheduling cycle for this ask.

func (*Allocation) SetAllocatedResource added in v1.6.0 

func (a *Allocation) SetAllocatedResource(allocatedResource *resources.Resource)

SetAllocatedResource updates the allocated resources for this allocation.

func (*Allocation) SetBindTime added in v1.3.0 

func (a *Allocation) SetBindTime(bindTime time.Time)

SetBindTime sets the time this allocation was bound.

func (*Allocation) SetInstanceType added in v1.4.0 

func (a *Allocation) SetInstanceType(instType string)

SetInstanceType sets node instance type for this allocation.

func (*Allocation) SetNodeID added in v1.6.0 

func (a *Allocation) SetNodeID(nodeID string)

SetNodeID sets the node this allocation is assigned to.

func (*Allocation) SetPlaceholderCreateTime added in v1.1.0 

func (a *Allocation) SetPlaceholderCreateTime(placeholderCreateTime time.Time)

SetPlaceholderCreateTime updates the placeholder's creation time.

func (*Allocation) SetPlaceholderUsed added in v1.1.0 

func (a *Allocation) SetPlaceholderUsed(placeholderUsed bool)

SetPlaceholderUsed sets whether this allocation is replacing a placeholder.

func (*Allocation) SetRelease added in v1.1.0 

func (a *Allocation) SetRelease(release *Allocation)

SetRelease sets the release for this allocation.

func (*Allocation) SetReleased added in v1.1.0 

func (a *Allocation) SetReleased(released bool)

SetReleased updates the release status of the allocation.

func (*Allocation) SetRequiredNode added in v1.6.0 

func (a *Allocation) SetRequiredNode(node string)

SetRequiredNode sets the required node (used only by testing so lock is not taken)

func (*Allocation) SetScaleUpTriggered added in v1.6.0 

func (a *Allocation) SetScaleUpTriggered(triggered bool)

SetScaleUpTriggered marks this allocation as having triggered the autoscaler.

func (*Allocation) SetSchedulingAttempted added in v1.6.0 

func (a *Allocation) SetSchedulingAttempted(attempted bool)

SetSchedulingAttempted marks whether scheduling has been attempted at least once for this allocation.

func (*Allocation) String 

func (a *Allocation) String() string

func (*Allocation) UpdatePreemptCheckTime added in v1.6.0 

func (a *Allocation) UpdatePreemptCheckTime()

UpdatePreemptCheckTime is used to mark when this allocation is evaluated for preemption.

type AllocationLogEntry added in v1.1.0 

type AllocationLogEntry struct {
	Message        string
	LastOccurrence time.Time
	Count          int32
}

type AllocationResult added in v1.1.0 

type AllocationResult struct {
	ResultType     AllocationResultType
	Request        *Allocation
	NodeID         string
	ReservedNodeID string
}

func (*AllocationResult) String added in v1.1.0 

func (ar *AllocationResult) String() string

type AllocationResultType added in v1.6.0 

type AllocationResultType int
const (
	None AllocationResultType = iota
	Allocated
	AllocatedReserved
	Reserved
	Unreserved
	Replaced
)

func (AllocationResultType) String added in v1.6.0 

func (art AllocationResultType) String() string

type Application 

type Application struct {
	ApplicationID  string    // application ID
	Partition      string    // partition Name
	SubmissionTime time.Time // time application was submitted

	locking.RWMutex
	// contains filtered or unexported fields
}

func NewApplication 

func NewApplication(siApp *si.AddApplicationRequest, ugi security.UserGroup, eventHandler handler.EventHandler, rmID string) *Application

func (*Application) AddAllocation 

func (sa *Application) AddAllocation(alloc *Allocation)

Add a new Allocation to the application

func (*Application) AddAllocationAsk 

func (sa *Application) AddAllocationAsk(ask *Allocation) error

Add an allocation ask to this application If the ask already exist update the existing info

func (*Application) AllocateAsk added in v1.6.0 

func (sa *Application) AllocateAsk(allocKey string) (*resources.Resource, error)

func (*Application) CleanupTrackedResource added in v1.4.0 

func (sa *Application) CleanupTrackedResource()

func (*Application) CurrentState 

func (sa *Application) CurrentState() string

Return the current state or a checked specific state for the application. The state machine handles the locking.

func (*Application) DeallocateAsk added in v1.6.0 

func (sa *Application) DeallocateAsk(allocKey string) (*resources.Resource, error)

func (*Application) FailApplication 

func (sa *Application) FailApplication(failureMessage string) error

FailApplication fails this application.

func (*Application) FinishedTime 

func (sa *Application) FinishedTime() time.Time

func (*Application) GetAllAllocations 

func (sa *Application) GetAllAllocations() []*Allocation

get a copy of all allocations of the application

func (*Application) GetAllPlaceholderData 

func (sa *Application) GetAllPlaceholderData() []*PlaceholderData

func (*Application) GetAllRequests added in v1.1.0 

func (sa *Application) GetAllRequests() []*Allocation

GetAllRequests returns a copy of all requests of the application

func (*Application) GetAllocatedResource 

func (sa *Application) GetAllocatedResource() *resources.Resource

Return the allocated resources for this application

func (*Application) GetAllocationAsk 

func (sa *Application) GetAllocationAsk(allocationKey string) *Allocation

Return the allocation ask for the key, nil if not found

func (*Application) GetApplicationSummary added in v1.3.0 

func (sa *Application) GetApplicationSummary(rmID string) *ApplicationSummary

func (*Application) GetAskMaxPriority added in v1.2.0 

func (sa *Application) GetAskMaxPriority() int32

func (*Application) GetAskReservations 

func (sa *Application) GetAskReservations(allocKey string) []string

Return the allocation reservations on any node. The returned array is 0 or more keys into the reservations map. No locking must be called while holding the lock

func (*Application) GetGuaranteedResource added in v1.6.0 

func (sa *Application) GetGuaranteedResource() *resources.Resource

GetGuaranteedResource returns the guaranteed resource that is set in the application tags

func (*Application) GetMaxAllocatedResource 

func (sa *Application) GetMaxAllocatedResource() *resources.Resource

func (*Application) GetMaxApps added in v1.6.0 

func (sa *Application) GetMaxApps() uint64

GetMaxApps returns the max apps that is set in the application tags

func (*Application) GetMaxResource added in v1.6.0 

func (sa *Application) GetMaxResource() *resources.Resource

GetMaxResource returns the max resource that is set in the application tags

func (*Application) GetPendingResource 

func (sa *Application) GetPendingResource() *resources.Resource

Return the pending resources for this application

func (*Application) GetPlaceholderAsk 

func (sa *Application) GetPlaceholderAsk() *resources.Resource

Return the total placeholder ask for this application Is only set on app creation and used when app is added to a queue

func (*Application) GetPlaceholderResource 

func (sa *Application) GetPlaceholderResource() *resources.Resource

Return the allocated placeholder resources for this application

func (*Application) GetQueue 

func (sa *Application) GetQueue() *Queue

func (*Application) GetQueuePath 

func (sa *Application) GetQueuePath() string

func (*Application) GetRejectedMessage 

func (sa *Application) GetRejectedMessage() string

func (*Application) GetReservations 

func (sa *Application) GetReservations() []string

GetReservations returns an array of all reservation keys for the application. This will return an empty array if there are no reservations. Visible for tests

func (*Application) GetStateLog 

func (sa *Application) GetStateLog() []*StateLogEntry

func (*Application) GetTag 

func (sa *Application) GetTag(tag string) string

Get a tag from the application Note: tags are not case sensitive

func (*Application) GetUser 

func (sa *Application) GetUser() security.UserGroup

get a copy of the user details for the application

func (*Application) HandleApplicationEvent 

func (sa *Application) HandleApplicationEvent(event applicationEvent) error

HandleApplicationEvent handles the state event for the application. The application lock is expected to be held.

func (*Application) HandleApplicationEventWithInfo 

func (sa *Application) HandleApplicationEventWithInfo(event applicationEvent, eventInfo string) error

HandleApplicationEventWithInfo handles the state event for the application with associated info object. The application lock is expected to be held.

func (*Application) HasPlaceholderAllocation added in v1.4.0 

func (sa *Application) HasPlaceholderAllocation() bool

func (*Application) HasReserved added in v1.2.0 

func (sa *Application) HasReserved() bool

HasReserved returns true if the application has any reservations.

func (*Application) IsAccepted 

func (sa *Application) IsAccepted() bool

func (*Application) IsAllocationAssignedToApp 

func (sa *Application) IsAllocationAssignedToApp(alloc *Allocation) bool

func (*Application) IsCompleted 

func (sa *Application) IsCompleted() bool

func (*Application) IsCompleting 

func (sa *Application) IsCompleting() bool

func (*Application) IsCreateForced added in v1.4.0 

func (sa *Application) IsCreateForced() bool

func (*Application) IsExpired 

func (sa *Application) IsExpired() bool

func (*Application) IsFailed 

func (sa *Application) IsFailed() bool

func (*Application) IsFailing 

func (sa *Application) IsFailing() bool

func (*Application) IsNew 

func (sa *Application) IsNew() bool

func (*Application) IsRejected 

func (sa *Application) IsRejected() bool

func (*Application) IsReservedOnNode 

func (sa *Application) IsReservedOnNode(nodeID string) bool

IsReservedOnNode returns true if and only if the node has been reserved by the application An empty nodeID is never reserved.

func (*Application) IsResuming 

func (sa *Application) IsResuming() bool

func (*Application) IsRunning 

func (sa *Application) IsRunning() bool

func (*Application) LogAppSummary added in v1.3.0 

func (sa *Application) LogAppSummary(rmID string)

func (*Application) OnStateChange 

func (sa *Application) OnStateChange(event *fsm.Event, eventInfo string)

OnStatChange every time the application enters a new state. It sends an event about the state change to the shim as an application update. The only state that does not generate an event is Rejected.

func (*Application) RecoverAllocationAsk 

func (sa *Application) RecoverAllocationAsk(alloc *Allocation)

Add the ask when a node allocation is recovered. Safeguarded against a nil but the recovery generates the ask and should never be nil.

func (*Application) RejectApplication 

func (sa *Application) RejectApplication(rejectedMessage string) error

RejectApplication rejects this application.

func (*Application) RemoveAllAllocations 

func (sa *Application) RemoveAllAllocations() []*Allocation

Remove all allocations from the application. All allocations that have been removed are returned.

func (*Application) RemoveAllocation 

func (sa *Application) RemoveAllocation(allocationKey string, releaseType si.TerminationType) *Allocation

Remove the Allocation from the application. Return the allocation that was removed or nil if not found.

func (*Application) RemoveAllocationAsk 

func (sa *Application) RemoveAllocationAsk(allocKey string) int

Remove one or more allocation asks from this application. This also removes any reservations that are linked to the ask. The return value is the number of reservations released

func (*Application) ReplaceAllocation 

func (sa *Application) ReplaceAllocation(allocationKey string) *Allocation

func (*Application) Reserve 

func (sa *Application) Reserve(node *Node, ask *Allocation) error

Reserve the application for this node and ask combination. If the reservation fails the function returns false, if the reservation is made it returns true. If the node and ask combination was already reserved for the application this is a noop and returns true.

func (*Application) SetQueue 

func (sa *Application) SetQueue(queue *Queue)

Set the leaf queue the application runs in.

func (*Application) SetQueuePath 

func (sa *Application) SetQueuePath(queuePath string)

Set the leaf queue the application runs in. The queue will be created when the app is added to the partition. The queue name is set to what the placement rule returned.

func (*Application) SetState 

func (sa *Application) SetState(state string)

func (*Application) SetTerminatedCallback 

func (sa *Application) SetTerminatedCallback(callback func(appID string))

func (*Application) SetTimedOutPlaceholder added in v1.1.0 

func (sa *Application) SetTimedOutPlaceholder(taskGroupName string, timedOut int64)

SetTimedOutPlaceholder should be used for testing only.

func (*Application) StartTime added in v1.3.0 

func (sa *Application) StartTime() time.Time

func (*Application) String 

func (sa *Application) String() string

func (*Application) UnReserve 

func (sa *Application) UnReserve(node *Node, ask *Allocation) (int, error)

UnReserve the application for this node and ask combination. This first removes the reservation from the node. If the reservation does not exist it returns 0 for reservations removed, if the reservation is removed it returns 1. The error is set if the reservation key cannot be removed from the app or node.

func (*Application) UnSetQueue 

func (sa *Application) UnSetQueue()

remove the leaf queue the application runs in, used when completing the app

func (*Application) UpdateAllocationResources added in v1.6.0 

func (sa *Application) UpdateAllocationResources(alloc *Allocation) error

UpdateAllocationResources updates the app, queue, and user tracker with deltas for an allocation. If an existing allocation cannot be found or alloc is invalid, an error is returned.

type ApplicationSummary added in v1.3.0 

type ApplicationSummary struct {
	ApplicationID       string
	SubmissionTime      time.Time
	StartTime           time.Time
	FinishTime          time.Time
	User                string
	Queue               string
	State               string
	RmID                string
	ResourceUsage       *resources.TrackedResource
	PreemptedResource   *resources.TrackedResource
	PlaceholderResource *resources.TrackedResource
}

func (*ApplicationSummary) DoLogging added in v1.3.0 

func (as *ApplicationSummary) DoLogging()

func (*ApplicationSummary) String added in v1.5.0 

func (as *ApplicationSummary) String() string

type Node 

type Node struct {
	// Fields for fast access These fields are considered read only.
	// Values should only be set when creating a new node and never changed.
	NodeID    string
	Hostname  string
	Rackname  string
	Partition string

	locking.RWMutex
	// contains filtered or unexported fields
}

func NewNode 

func NewNode(proto *si.NodeInfo) *Node

func (*Node) AddAllocation 

func (sn *Node) AddAllocation(alloc *Allocation)

AddAllocation adds the allocation to the node. Used resources will increase available will decrease. A nil Allocation makes no changes. Preempted resources must have been released already. Do a sanity check to make sure it still fits in the node and nothing has changed

func (*Node) AddListener 

func (sn *Node) AddListener(listener NodeListener)

func (*Node) CanAllocate 

func (sn *Node) CanAllocate(res *resources.Resource) bool

CanAllocate checks if the proposed allocation fits in the available resources. If the proposed allocation does not fit false is returned.

func (*Node) FitInNode 

func (sn *Node) FitInNode(resRequest *resources.Resource) bool

FitInNode checks if the request fits in the node. All resources types requested must match the resource types provided by the nodes. A request may ask for only a subset of the types, but the node must provide at least the resource types requested in a larger or equal quantity as requested.

func (*Node) GetAllAllocations 

func (sn *Node) GetAllAllocations() []*Allocation

Get a copy of the allocations on this node

func (*Node) GetAllocatedResource 

func (sn *Node) GetAllocatedResource() *resources.Resource

Get the allocated resource on this node.

func (*Node) GetAllocation 

func (sn *Node) GetAllocation(allocationKey string) *Allocation

Return the allocation based on the allocationKey of the allocation. returns nil if the allocation is not found

func (*Node) GetAttribute 

func (sn *Node) GetAttribute(key string) string

Get an attribute by name. The most used attributes can be directly accessed via the fields: HostName, RackName and Partition. This is a lock free call. All attributes are considered read only

func (*Node) GetAttributes added in v1.3.0 

func (sn *Node) GetAttributes() map[string]string

func (*Node) GetAvailableResource 

func (sn *Node) GetAvailableResource() *resources.Resource

Get the available resource on this node.

func (*Node) GetCapacity 

func (sn *Node) GetCapacity() *resources.Resource

func (*Node) GetFitInScoreForAvailableResource added in v1.3.0 

func (sn *Node) GetFitInScoreForAvailableResource(res *resources.Resource) float64

GetFitInScoreForAvailableResource calculates a fit in score for "res" based on the current available resources, avoiding cloning. The caller must ensure that "res" cannot change while this method is running.

func (*Node) GetInstanceType added in v1.3.0 

func (sn *Node) GetInstanceType() string

Get InstanceType of this node. This is a lock free call because all attributes are considered read only

func (*Node) GetOccupiedResource 

func (sn *Node) GetOccupiedResource() *resources.Resource

func (*Node) GetReservationKeys added in v1.2.0 

func (sn *Node) GetReservationKeys() []string

GetReservationKeys Return an array of all reservation keys for the node. This will return an empty array if there are no reservations. Visible for tests

func (*Node) GetReservations 

func (sn *Node) GetReservations() []*reservation

GetReservations returns all reservation made on this node

func (*Node) GetResourceUsageShares 

func (sn *Node) GetResourceUsageShares() map[string]float64

GetResourceUsageShares gets a map of name -> resource usages per type in shares (0 to 1). Can return NaN.

func (*Node) GetUtilizedResource 

func (sn *Node) GetUtilizedResource() *resources.Resource

Get the utilized resource on this node.

func (*Node) IsReserved 

func (sn *Node) IsReserved() bool

Return if the node has been reserved by any application

func (*Node) IsSchedulable 

func (sn *Node) IsSchedulable() bool

Can this node be used in scheduling.

func (*Node) RemoveAllocation 

func (sn *Node) RemoveAllocation(allocationKey string) *Allocation

Remove the allocation to the node. Returns nil if the allocation was not found and no changes are made. If the allocation is found the Allocation removed is returned. Used resources will decrease available will increase as per the allocation removed.

func (*Node) RemoveListener 

func (sn *Node) RemoveListener(listener NodeListener)

func (*Node) ReplaceAllocation 

func (sn *Node) ReplaceAllocation(allocationKey string, replace *Allocation, delta *resources.Resource)

ReplaceAllocation replaces the placeholder with the real allocation on the node. The delta passed in is the difference in resource usage between placeholder and real allocation. It should always be a negative value or zero: it is a decrease in usage or no change

func (*Node) Reserve 

func (sn *Node) Reserve(app *Application, ask *Allocation) error

Reserve the node for this application and ask combination, if not reserved yet. The reservation is checked against the node resources. If the reservation fails the function returns false, if the reservation is made it returns true.

func (*Node) SendNodeAddedEvent added in v1.4.0 

func (sn *Node) SendNodeAddedEvent()

func (*Node) SendNodeRemovedEvent added in v1.4.0 

func (sn *Node) SendNodeRemovedEvent()

func (*Node) SetCapacity 

func (sn *Node) SetCapacity(newCapacity *resources.Resource) *resources.Resource

SetCapacity changes the node resource capacity and returns the resource delta. The delta is positive for an increased capacity and negative for a decrease.

func (*Node) SetOccupiedResource 

func (sn *Node) SetOccupiedResource(occupiedResource *resources.Resource)

func (*Node) SetSchedulable 

func (sn *Node) SetSchedulable(schedulable bool)

Set the node to unschedulable. This will cause the node to be skipped during the scheduling cycle. Visible for testing only

func (*Node) String 

func (sn *Node) String() string

func (*Node) TryAddAllocation added in v1.6.0 

func (sn *Node) TryAddAllocation(alloc *Allocation) bool

TryAddAllocation attempts to add the allocation to the node. Used resources will increase available will decrease. A nil Allocation makes no changes. Preempted resources must have been released already. Do a sanity check to make sure it still fits in the node and nothing has changed

func (*Node) UpdateAllocatedResource added in v1.6.0 

func (sn *Node) UpdateAllocatedResource(delta *resources.Resource)

type NodeCollection 

type NodeCollection interface {
	AddNode(node *Node) error
	RemoveNode(nodeID string) *Node
	GetNode(nodeID string) *Node
	GetNodeCount() int
	GetNodes() []*Node
	GetNodeIterator() NodeIterator
	GetFullNodeIterator() NodeIterator
	SetNodeSortingPolicy(policy NodeSortingPolicy)
	GetNodeSortingPolicy() NodeSortingPolicy
}

NodeCollection represents a collection of nodes for a partition. Implementations of this interface must be internally synchronized to avoid data races.

func NewNodeCollection 

func NewNodeCollection(partition string) NodeCollection

Create a new collection for the given partition.

type NodeIterator 

type NodeIterator interface {
	// ForEachNode Calls the provided function on the sorted Node object until it returns false
	ForEachNode(func(*Node) bool)
}

NodeIterator iterates over a list of nodes based on the defined policy

type NodeListener 

type NodeListener interface {
	NodeUpdated(sn *Node)
}

type NodeSortingPolicy 

type NodeSortingPolicy interface {
	PolicyType() policies.SortingPolicy
	ScoreNode(node *Node) float64
	ResourceWeights() map[string]float64
}

func NewNodeSortingPolicy 

func NewNodeSortingPolicy(policyType string, resourceWeights map[string]float64) NodeSortingPolicy

type ObjectEvent 

type ObjectEvent int

---------------------------------- object events these events are used for: partitions and managed queues ---------------------------------- 

const (
	Remove ObjectEvent = iota
	Start
	Stop
)

func (ObjectEvent) String 

func (oe ObjectEvent) String() string

type ObjectState 

type ObjectState int

---------------------------------- object states these states are used by: partitions and managed queues ---------------------------------- 

const (
	Active ObjectState = iota
	Draining
	Stopped
)

func (ObjectState) String 

func (os ObjectState) String() string

type PlaceholderData 

type PlaceholderData struct {
	TaskGroupName string
	Count         int64
	MinResource   *resources.Resource
	Replaced      int64
	TimedOut      int64
}

type PreemptionContext added in v1.1.0 

type PreemptionContext struct {
	locking.RWMutex
	// contains filtered or unexported fields
}

func NewRequiredNodePreemptor added in v1.2.0 

func NewRequiredNodePreemptor(node *Node, requiredAsk *Allocation) *PreemptionContext

func (*PreemptionContext) GetVictims added in v1.1.0 

func (p *PreemptionContext) GetVictims() []*Allocation

type Preemptor added in v1.3.0 

type Preemptor struct {
	// contains filtered or unexported fields
}

Preemptor encapsulates the functionality required for preemption victim selection

func NewPreemptor added in v1.3.0 

func NewPreemptor(application *Application, headRoom *resources.Resource, preemptionDelay time.Duration, ask *Allocation, iterator NodeIterator, nodesTried bool) *Preemptor

NewPreemptor creates a new preemptor. The preemptor itself is not thread safe, and assumes the application lock is held.

func (*Preemptor) CheckPreconditions added in v1.3.0 

func (p *Preemptor) CheckPreconditions() bool

CheckPreconditions performs simple sanity checks designed to determine if preemption should be attempted for an ask. If checks succeed, updates the ask preemption check time.

func (*Preemptor) TryPreemption added in v1.3.0 

func (p *Preemptor) TryPreemption() (*AllocationResult, bool)

type Queue 

type Queue struct {
	QueuePath string // Fully qualified path for the queue
	Name      string // Queue name as in the config etc.

	locking.RWMutex
	// contains filtered or unexported fields
}

Queue structure inside Scheduler

func NewConfiguredQueue 

func NewConfiguredQueue(conf configs.QueueConfig, parent *Queue) (*Queue, error)

NewConfiguredQueue creates a new queue from scratch based on the configuration lock free as it cannot be referenced yet

func NewDynamicQueue 

func NewDynamicQueue(name string, leaf bool, parent *Queue) (*Queue, error)

NewDynamicQueue creates a new queue to be added to the system based on the placement rules A dynamically added queue can never be the root queue so parent must be set lock free as it cannot be referenced yet

func NewRecoveryQueue added in v1.4.0 

func NewRecoveryQueue(parent *Queue) (*Queue, error)

NewRecoveryQueue creates a recovery queue if it does not exist. The recovery queue is a dynamic queue, but has an invalid name so that it cannot be directly referenced.

func (*Queue) AddApplication 

func (sq *Queue) AddApplication(app *Application)

AddApplication adds the application to the queue. All checks are assumed to have passed before we get here. No update of pending resource is needed as it should not have any requests yet. Replaces the existing application without further checks.

func (*Queue) ApplyConf 

func (sq *Queue) ApplyConf(conf configs.QueueConfig) error

ApplyConf is the locked version of applyConf

func (*Queue) CheckAdminAccess 

func (sq *Queue) CheckAdminAccess(user security.UserGroup) bool

CheckAdminAccess checks if the user has access to the queue to perform administrative actions. The check is performed recursively: i.e. access to the parent allows access to this queue.

func (*Queue) CheckSubmitAccess 

func (sq *Queue) CheckSubmitAccess(user security.UserGroup) bool

CheckSubmitAccess checks if the user has access to the queue to submit an application. The check is performed recursively: i.e. access to the parent allows access to this queue. This will check both submitACL and adminACL.

func (*Queue) CurrentState 

func (sq *Queue) CurrentState() string

CurrentState returns the current state of the queue in string form.

func (*Queue) DecAllocatedResource 

func (sq *Queue) DecAllocatedResource(alloc *resources.Resource) error

DecAllocatedResource decrement the allocated resources for this queue (recursively) Guard against going below zero resources.

func (*Queue) DecPreemptingResource added in v1.3.0 

func (sq *Queue) DecPreemptingResource(alloc *resources.Resource)

DecPreemptingResource decrements the preempting resources for this queue (recursively).

func (*Queue) FindEligiblePreemptionVictims added in v1.3.0 

func (sq *Queue) FindEligiblePreemptionVictims(queuePath string, ask *Allocation) map[string]*QueuePreemptionSnapshot

FindEligiblePreemptionVictims is used to locate tasks which may be preempted for the given ask. queuePath is the fully-qualified path of the queue where ask resides ask is the ask we are attempting to preempt for return is a map of potential victims keyed by queue path

func (*Queue) FindQueueByAppID added in v1.3.0 

func (sq *Queue) FindQueueByAppID(appID string) *Queue

FindQueueByAppID searches the queue hierarchy for an application with the given appID and returns the queue it belongs to

func (*Queue) GetActualGuaranteedResource added in v1.3.0 

func (sq *Queue) GetActualGuaranteedResource() *resources.Resource

GetActualGuaranteedResources returns the actual (including parent) guaranteed resources for the queue.

func (*Queue) GetAllocatedResource 

func (sq *Queue) GetAllocatedResource() *resources.Resource

GetAllocatedResource returns a clone of the allocated resources for this queue.

func (*Queue) GetApplication added in v1.1.0 

func (sq *Queue) GetApplication(appID string) *Application

getApplication return the Application based on the ID.

func (*Queue) GetChildQueue 

func (sq *Queue) GetChildQueue(name string) *Queue

GetChildQueue returns a queue if the name exists in the child map as a key.

func (*Queue) GetCopyOfApps 

func (sq *Queue) GetCopyOfApps() map[string]*Application

GetCopyOfApps gets a shallow copy of all non-completed apps holding the lock

func (*Queue) GetCopyOfChildren 

func (sq *Queue) GetCopyOfChildren() map[string]*Queue

GetCopyOfChildren return a shallow copy of the child queue map. This is used by the partition manager to find all queues to clean however we can not guarantee that there is no new child added while we clean up since there is no overall lock on the scheduler. We'll need to test just before to make sure the parent is empty

func (*Queue) GetCurrentPriority added in v1.2.0 

func (sq *Queue) GetCurrentPriority() int32

func (*Queue) GetFairMaxResource added in v1.6.0 

func (sq *Queue) GetFairMaxResource() *resources.Resource

GetFairMaxResource computes the fair max resources for a given queue. Starting with the root, descend down to the target queue allowing children to override Resource values . If the root includes an explicit 0 value for a Resource, do not include it in the accumulator and treat it as missing. If no children provide a maximum capacity override, the resulting value will be the value found on the Root. It is useful for fair-scheduling to allow a ratio to be produced representing the rough utilization % of a given queue.

func (*Queue) GetGuaranteedResource 

func (sq *Queue) GetGuaranteedResource() *resources.Resource

GetGuaranteedResource returns a clone of the guaranteed resource for the queue.

func (*Queue) GetMaxApps added in v1.6.0 

func (sq *Queue) GetMaxApps() uint64

GetMaxApps returns the maximum number of applications that can run in this queue.

func (*Queue) GetMaxQueueSet 

func (sq *Queue) GetMaxQueueSet() *resources.Resource

GetMaxQueueSet returns the max resource for the queue. The max resource should never be larger than the max resource of the parent. The cluster size, which defines the root limit, is not relevant for this call. Contrary to the GetMaxResource call. This will return nil unless a limit is set. Used during scheduling in an auto-scaling cluster. NOTE: if a resource quantity is missing and a limit is defined the missing quantity will be seen as a limit of 0. When defining a limit you therefore should define all resource quantities.

func (*Queue) GetMaxResource 

func (sq *Queue) GetMaxResource() *resources.Resource

GetMaxResource returns the max resource for the queue. The max resource should never be larger than the max resource of the parent. The root queue always has its limit set to the total cluster size (dynamic based on node registration) In case there are no nodes in a newly started cluster and no queues have a limit configured this call will return nil. NOTE: if a resource quantity is missing and a limit is defined the missing quantity will be seen as a limit of 0. When defining a limit you therefore should define all resource quantities.

func (*Queue) GetPartitionQueueDAOInfo 

func (sq *Queue) GetPartitionQueueDAOInfo(include bool) dao.PartitionQueueDAOInfo

GetPartitionQueueDAOInfo returns the queue hierarchy as an object for a REST call. Include is false, which means that returns the specified queue object, but does not return the children of the specified queue.

func (*Queue) GetPendingResource 

func (sq *Queue) GetPendingResource() *resources.Resource

GetPendingResource returns the pending resources for this queue.

func (*Queue) GetPreemptingResource 

func (sq *Queue) GetPreemptingResource() *resources.Resource

GetPreemptingResource returns a clone of the preempting resources for this queue.

func (*Queue) GetPreemptionDelay added in v1.3.0 

func (sq *Queue) GetPreemptionDelay() time.Duration

func (*Queue) GetPreemptionPolicy added in v1.3.0 

func (sq *Queue) GetPreemptionPolicy() policies.PreemptionPolicy

func (*Queue) GetPriorityPolicyAndOffset added in v1.3.0 

func (sq *Queue) GetPriorityPolicyAndOffset() (policies.PriorityPolicy, int32)

func (*Queue) GetQueueOutstandingRequests 

func (sq *Queue) GetQueueOutstandingRequests(total *[]*Allocation)

GetQueueOutstandingRequests builds a slice of pending allocation asks that fits into the queue's headroom.

func (*Queue) GetQueuePath 

func (sq *Queue) GetQueuePath() string

GetQueuePath returns the fully qualified path of this queue.

func (*Queue) GetReservedApps added in v1.2.0 

func (sq *Queue) GetReservedApps() map[string]int

GetReservedApps returns a shallow copy of the reserved app list locked to prevent race conditions from event updates

func (*Queue) IncAllocatedResource 

func (sq *Queue) IncAllocatedResource(alloc *resources.Resource)

IncAllocatedResource increments the allocated resources for this queue (recursively). No queue limits are checked.

func (*Queue) IncPreemptingResource 

func (sq *Queue) IncPreemptingResource(alloc *resources.Resource)

IncPreemptingResource increments the preempting resources for this queue (recursively).

func (*Queue) IsDraining 

func (sq *Queue) IsDraining() bool

IsDraining returns true if the queue in Draining state. Existing applications will still be scheduled No new applications will be accepted.

func (*Queue) IsEmpty 

func (sq *Queue) IsEmpty() bool

IsEmpty returns true if a queue is empty based on the following definition: A parent queue is empty when it has no children left A leaf queue is empty when there are no applications left

func (*Queue) IsLeafQueue 

func (sq *Queue) IsLeafQueue() bool

IsLeafQueue returns true is the queue a leaf. Returns false for a parent queue.

func (*Queue) IsManaged 

func (sq *Queue) IsManaged() bool

IsManaged returns true for a managed queue. Returns false for a dynamic queue.

func (*Queue) IsPrioritySortEnabled added in v1.2.0 

func (sq *Queue) IsPrioritySortEnabled() bool

func (*Queue) IsRunning 

func (sq *Queue) IsRunning() bool

IsRunning returns true if the queue in Active state.

func (*Queue) IsStopped 

func (sq *Queue) IsStopped() bool

IsStopped returns true if the queue in Stopped state. The queue is skipped for scheduling in this state.

func (*Queue) MarkQueueForRemoval 

func (sq *Queue) MarkQueueForRemoval()

MarkQueueForRemoval marks the managed queue for removal from the system. This can be executed multiple times and is only effective the first time. This is a noop on an unmanaged queue.

func (*Queue) RemoveApplication 

func (sq *Queue) RemoveApplication(app *Application)

RemoveApplication removes the app from the list of tracked applications. Make sure that the app is assigned to this queue and not removed yet. If not found this call is a noop

func (*Queue) RemoveQueue 

func (sq *Queue) RemoveQueue() bool

RemoveQueue remove the queue from the structure. Since nothing is allocated there shouldn't be anything referencing this queue anymore. The real removal is the removal of the queue from the parent's child list. Use a read lock on this queue to prevent other changes but allow status checks etc.

func (*Queue) Reserve 

func (sq *Queue) Reserve(appID string)

Reserve increments the number of reservations for the application adding it to the map if needed. No checks this is only called when a reservation is processed using the app stored in the queue.

func (*Queue) SetMaxResource 

func (sq *Queue) SetMaxResource(max *resources.Resource)

SetMaxResource sets the max resource for the root queue. Called as part of adding or removing a node. Should only happen on the root, all other queues get it from the config via properties.

func (*Queue) SetMaxRunningApps added in v1.2.0 

func (sq *Queue) SetMaxRunningApps(maxApps uint64)

SetMaxRunningApps allows setting the maximum running apps on a queue

func (*Queue) SetResources added in v1.6.0 

func (sq *Queue) SetResources(guaranteedResource, maxResource *resources.Resource)

func (*Queue) String 

func (sq *Queue) String() string

func (*Queue) SupportTaskGroup 

func (sq *Queue) SupportTaskGroup() bool

SupportTaskGroup returns true if the queue supports task groups. FIFO policy is required to support this. NOTE: this call does not make sense for a parent queue, and always returns false

func (*Queue) TryAllocate 

func (sq *Queue) TryAllocate(iterator func() NodeIterator, fullIterator func() NodeIterator, getnode func(string) *Node, allowPreemption bool) *AllocationResult

TryAllocate tries to allocate a pending requests. This only gets called if there is a pending request on this queue or its children. This is a depth first algorithm: descend into the depth of the queue tree first. Child queues are sorted based on the configured queue sortPolicy. Queues without pending resources are skipped. Applications are sorted based on the application sortPolicy. Applications without pending resources are skipped. Lock free call this all locks are taken when needed in called functions

func (*Queue) TryIncAllocatedResource added in v1.6.0 

func (sq *Queue) TryIncAllocatedResource(alloc *resources.Resource) error

TryIncAllocatedResource increments the allocated resources for this queue (recursively). Guard against going over max resources if set

func (*Queue) TryPlaceholderAllocate 

func (sq *Queue) TryPlaceholderAllocate(iterator func() NodeIterator, getnode func(string) *Node) *AllocationResult

TryPlaceholderAllocate tries to replace a placeholders with a real allocation. This only gets called if there is a pending request on this queue or its children. This is a depth first algorithm: descend into the depth of the queue tree first. Child queues are sorted based on the configured queue sortPolicy. Queues without pending resources are skipped. Applications are sorted based on the application sortPolicy. Applications without pending resources are skipped. Lock free call this all locks are taken when needed in called functions

func (*Queue) TryReservedAllocate 

func (sq *Queue) TryReservedAllocate(iterator func() NodeIterator) *AllocationResult

TryReservedAllocate tries to allocate a reservation. This only gets called if there is a pending request on this queue or its children. This is a depth first algorithm: descend into the depth of the queue tree first. Child queues are sorted based on the configured queue sortPolicy. Queues without pending resources are skipped. Applications are currently NOT sorted and are iterated over in a random order. Lock free call this all locks are taken when needed in called functions

func (*Queue) UnReserve 

func (sq *Queue) UnReserve(appID string, releases int)

UnReserve decrements the number of reservations for the application removing it to the map if all reservations are removed. No checks this is only called when a reservation is processed using the app stored in the queue.

func (*Queue) UpdateApplicationPriority added in v1.2.0 

func (sq *Queue) UpdateApplicationPriority(applicationID string, priority int32)

func (*Queue) UpdateQueuePriority added in v1.2.0 

func (sq *Queue) UpdateQueuePriority(queueName string, priority int32)

func (*Queue) UpdateQueueProperties added in v1.2.0 

func (sq *Queue) UpdateQueueProperties()

UpdateQueueProperties updates the queue properties defined as text

type QueuePreemptionSnapshot added in v1.3.0 

type QueuePreemptionSnapshot struct {
	Parent             *QueuePreemptionSnapshot // snapshot of parent queue
	QueuePath          string                   // fully qualified path to queue
	Leaf               bool                     // true if queue is a leaf queue
	AllocatedResource  *resources.Resource      // allocated resources
	PreemptingResource *resources.Resource      // resources currently flagged for preemption
	MaxResource        *resources.Resource      // maximum resources for this queue
	GuaranteedResource *resources.Resource      // guaranteed resources for this queue
	PotentialVictims   []*Allocation            // list of allocations which could be preempted
	AskQueue           *QueuePreemptionSnapshot // snapshot of ask or preemptor queue
}

QueuePreemptionSnapshot is used to track a snapshot of a queue for preemption

func (*QueuePreemptionSnapshot) AddAllocation added in v1.3.0 

func (qps *QueuePreemptionSnapshot) AddAllocation(alloc *resources.Resource)

AddAllocation adds an allocation to this snapshot's resource usage

func (*QueuePreemptionSnapshot) Duplicate added in v1.3.0 

func (qps *QueuePreemptionSnapshot) Duplicate(copy map[string]*QueuePreemptionSnapshot) *QueuePreemptionSnapshot

Duplicate creates a copy of this snapshot into the given map by queue path

func (*QueuePreemptionSnapshot) GetGuaranteedResource added in v1.3.0 

func (qps *QueuePreemptionSnapshot) GetGuaranteedResource() *resources.Resource

GetGuaranteedResource computes the current guaranteed resources considering parent guaranteed

func (*QueuePreemptionSnapshot) GetMaxResource added in v1.3.0 

func (qps *QueuePreemptionSnapshot) GetMaxResource() *resources.Resource

GetMaxResource computes the current max resources considering parent max

func (*QueuePreemptionSnapshot) GetPreemptableResource added in v1.6.0 

func (qps *QueuePreemptionSnapshot) GetPreemptableResource() *resources.Resource

func (*QueuePreemptionSnapshot) GetRemainingGuaranteedResource added in v1.6.0 

func (qps *QueuePreemptionSnapshot) GetRemainingGuaranteedResource() *resources.Resource

func (*QueuePreemptionSnapshot) RemoveAllocation added in v1.3.0 

func (qps *QueuePreemptionSnapshot) RemoveAllocation(alloc *resources.Resource)

RemoveAllocation removes an allocation from this snapshot's resource usage

type StateLogEntry 

type StateLogEntry struct {
	Time             time.Time
	ApplicationState string
}

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.