objects

package
v0.0.0-...-bd24502 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 26, 2024 License: Apache-2.0 Imports: 38 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	Soft string = "Soft"
	Hard string = "Hard"

	NotEnoughUserQuota  = "Not enough user quota"
	NotEnoughQueueQuota = "Not enough queue quota"
)
View Source 
const (
	RunApplication applicationEvent = iota
	RejectApplication
	CompleteApplication
	FailApplication
	ExpireApplication
	ResumeApplication
)
View Source 
const (
	New applicationState = iota
	Accepted
	Running
	Rejected
	Completing
	Completed
	Failing
	Failed
	Expired
	Resuming
)

Application states are used for filtering in the webservice handlers. Please check&update the logic as needed if the state machine is modified

View Source 
const (
	EventNotNeeded = -1
)
View Source 
const (
	UnknownInstanceType = "UNKNOWN"
)

Variables

View Source 
var (
	Entropy *rand.Rand
	Ms      uint64
)

Functions

func CloneAllocationTags 

func CloneAllocationTags(tags map[string]string) map[string]string

CloneAllocationTags clones a tag map for safe copying.

func NewAppState 

func NewAppState() *fsm.FSM

func NewObjectState 

func NewObjectState() *fsm.FSM

func NewTreeIterator 

func NewTreeIterator(accept func(*Node) bool, getTree func() *btree.BTree) *treeIterator

func SetCompletingTimeout 

func SetCompletingTimeout(duration time.Duration)

SetCompletingTimeout should be used for testing only.

func SetReservationDelay 

func SetReservationDelay(delay time.Duration)

Set the reservation delay. Set when the cluster context is created to disable reservation.

Types

type Allocation 

type Allocation struct {
	locking.RWMutex
	// contains filtered or unexported fields
}

func NewAllocationFromSI 

func NewAllocationFromSI(alloc *si.Allocation) *Allocation

NewAllocationFromSI Create a new Allocation which has already been placed on a node, populating it with info from the SI Allocation object. If the input object is invalid, nil is returned.

func (*Allocation) AskDAO 

func (a *Allocation) AskDAO() *dao.AllocationAskDAOInfo

func (*Allocation) ClearRelease 

func (a *Allocation) ClearRelease()

ClearRelease removes any release from this allocation.

func (*Allocation) DAO 

func (alloc *Allocation) DAO() *dao.AllocationDAOInfo

func (*Allocation) ForeignAllocationDAO 

func (alloc *Allocation) ForeignAllocationDAO() *dao.ForeignAllocationDAOInfo

func (*Allocation) GetAllocatedResource 

func (a *Allocation) GetAllocatedResource() *resources.Resource

GetAllocatedResource returns a reference to the allocated resources for this allocation. This must be treated as read-only.

func (*Allocation) GetAllocationKey 

func (a *Allocation) GetAllocationKey() string

GetAllocationKey returns the allocation key for this allocation.

func (*Allocation) GetAllocationLog 

func (a *Allocation) GetAllocationLog() []*AllocationLogEntry

GetAllocationLog returns a list of log entries corresponding to allocation preconditions not being met.

func (*Allocation) GetApplicationID 

func (a *Allocation) GetApplicationID() string

GetApplicationID returns the application ID for this allocation.

func (*Allocation) GetBindTime 

func (a *Allocation) GetBindTime() time.Time

GetBindTime returns the time this allocation was bound.

func (*Allocation) GetCreateTime 

func (a *Allocation) GetCreateTime() time.Time

GetCreateTime returns the time this allocation was created.

func (*Allocation) GetInstanceType 

func (a *Allocation) GetInstanceType() string

GetInstanceType return the type of the instance used by this allocation.

func (*Allocation) GetNodeID 

func (a *Allocation) GetNodeID() string

GetNodeID gets the node this allocation is assigned to.

func (*Allocation) GetPlaceholderCreateTime 

func (a *Allocation) GetPlaceholderCreateTime() time.Time

GetPlaceholderCreateTime returns the placeholder's create time for this allocation, if applicable.

func (*Allocation) GetPreemptCheckTime 

func (a *Allocation) GetPreemptCheckTime() time.Time

GetPreemptCheckTime returns the time this allocation was last evaluated for preemption.

func (*Allocation) GetPriority 

func (a *Allocation) GetPriority() int32

GetPriority returns the priority of this allocation.

func (*Allocation) GetRelease 

func (a *Allocation) GetRelease() *Allocation

GetRelease returns the associated release for this allocation.

func (*Allocation) GetRequiredNode 

func (a *Allocation) GetRequiredNode() string

GetRequiredNode gets the node (if any) required by this allocation.

func (*Allocation) GetTag 

func (a *Allocation) GetTag(tagName string) string

GetTag returns the value of a named tag or an empty string if not present.

func (*Allocation) GetTagsClone 

func (a *Allocation) GetTagsClone() map[string]string

GetTagsClone returns the copy of the tags for this allocation.

func (*Allocation) GetTaskGroup 

func (a *Allocation) GetTaskGroup() string

GetTaskGroup returns the task group name for this allocation.

func (*Allocation) HasRelease 

func (a *Allocation) HasRelease() bool

HasRelease determines if this allocation has an associated release.

func (*Allocation) HasTriggeredPreemption 

func (a *Allocation) HasTriggeredPreemption() bool

HasTriggeredPreemption returns whether this allocation has triggered preemption.

func (*Allocation) HasTriggeredScaleUp 

func (a *Allocation) HasTriggeredScaleUp() bool

HasTriggeredScaleUp determines if this allocation has triggered auto-scaling.

func (*Allocation) IsAllocated 

func (a *Allocation) IsAllocated() bool

IsAllocated determines if this request has been allocated yet.

func (*Allocation) IsAllowPreemptOther 

func (a *Allocation) IsAllowPreemptOther() bool

IsAllowPreemptOther returns whether this allocation is allowed to preempt others.

func (*Allocation) IsAllowPreemptSelf 

func (a *Allocation) IsAllowPreemptSelf() bool

IsAllowPreemptSelf returns whether preemption is allowed for this allocation.

func (*Allocation) IsForeign 

func (a *Allocation) IsForeign() bool

func (*Allocation) IsOriginator 

func (a *Allocation) IsOriginator() bool

IsOriginator returns whether this alloocation is the originator for the application.

func (*Allocation) IsPlaceholder 

func (a *Allocation) IsPlaceholder() bool

IsPlaceholder returns whether this allocation represents a placeholder.

func (*Allocation) IsPlaceholderUsed 

func (a *Allocation) IsPlaceholderUsed() bool

IsPlaceholderUsed returns whether this allocation is replacing a placeholder.

func (*Allocation) IsPreemptable 

func (a *Allocation) IsPreemptable() bool

func (*Allocation) IsPreempted 

func (a *Allocation) IsPreempted() bool

IsPreempted returns whether the allocation has been marked for preemption or not.

func (*Allocation) IsReleased 

func (a *Allocation) IsReleased() bool

IsReleased returns the release status of the allocation.

func (*Allocation) IsSchedulingAttempted 

func (a *Allocation) IsSchedulingAttempted() bool

IsSchedulingAttempted determines whether scheduling has been attempted at least once for this allocation.

func (*Allocation) LessThan 

func (a *Allocation) LessThan(other *Allocation) bool

LessThan compares two allocations by priority and then creation time.

func (*Allocation) LogAllocationFailure 

func (a *Allocation) LogAllocationFailure(message string, allocate bool)

LogAllocationFailure keeps track of preconditions not being met for an allocation.

func (*Allocation) MarkPreempted 

func (a *Allocation) MarkPreempted()

MarkPreempted marks the allocation as preempted.

func (*Allocation) MarkTriggeredPreemption 

func (a *Allocation) MarkTriggeredPreemption()

MarkTriggeredPreemption marks the current allocation because it triggered preemption during scheduling.

func (*Allocation) NewSIFromAllocation 

func (a *Allocation) NewSIFromAllocation() *si.Allocation

NewSIFromAllocation converts the Allocation into a SI object. This is a limited set of values that gets copied into the SI. This is only used to communicate *back* to the RM. All other fields are considered incoming fields from the RM into the core. The limited set of fields link the Allocation to an Application and Node.

func (*Allocation) SendPredicatesFailedEvent 

func (a *Allocation) SendPredicatesFailedEvent(predicateErrors map[string]int)

SendPredicatesFailedEvent updates the event system with the reason for predicate failures. The map predicateErrors contains how many times certain predicates failed in the scheduling cycle for this ask.

func (*Allocation) SetAllocatedResource 

func (a *Allocation) SetAllocatedResource(allocatedResource *resources.Resource)

SetAllocatedResource updates the allocated resources for this allocation.

func (*Allocation) SetBindTime 

func (a *Allocation) SetBindTime(bindTime time.Time)

SetBindTime sets the time this allocation was bound.

func (*Allocation) SetInstanceType 

func (a *Allocation) SetInstanceType(instType string)

SetInstanceType sets node instance type for this allocation.

func (*Allocation) SetNodeID 

func (a *Allocation) SetNodeID(nodeID string)

SetNodeID sets the node this allocation is assigned to.

func (*Allocation) SetPlaceholderCreateTime 

func (a *Allocation) SetPlaceholderCreateTime(placeholderCreateTime time.Time)

SetPlaceholderCreateTime updates the placeholder's creation time.

func (*Allocation) SetPlaceholderUsed 

func (a *Allocation) SetPlaceholderUsed(placeholderUsed bool)

SetPlaceholderUsed sets whether this allocation is replacing a placeholder.

func (*Allocation) SetRelease 

func (a *Allocation) SetRelease(release *Allocation)

SetRelease sets the release for this allocation.

func (*Allocation) SetReleased 

func (a *Allocation) SetReleased(released bool)

SetReleased updates the release status of the allocation.

func (*Allocation) SetRequiredNode 

func (a *Allocation) SetRequiredNode(node string)

SetRequiredNode sets the required node (used only by testing so lock is not taken)

func (*Allocation) SetScaleUpTriggered 

func (a *Allocation) SetScaleUpTriggered(triggered bool)

SetScaleUpTriggered marks this allocation as having triggered the autoscaler.

func (*Allocation) SetSchedulingAttempted 

func (a *Allocation) SetSchedulingAttempted(attempted bool)

SetSchedulingAttempted marks whether scheduling has been attempted at least once for this allocation.

func (*Allocation) String 

func (a *Allocation) String() string

func (*Allocation) UpdatePreemptCheckTime 

func (a *Allocation) UpdatePreemptCheckTime()

UpdatePreemptCheckTime is used to mark when this allocation is evaluated for preemption.

type AllocationLogEntry 

type AllocationLogEntry struct {
	Message        string
	LastOccurrence time.Time
	Count          int32
}

type AllocationResult 

type AllocationResult struct {
	ResultType     AllocationResultType
	Request        *Allocation
	NodeID         string
	ReservedNodeID string
}

func (*AllocationResult) String 

func (ar *AllocationResult) String() string

type AllocationResultType 

type AllocationResultType int
const (
	None AllocationResultType = iota
	Allocated
	AllocatedReserved
	Reserved
	Unreserved
	Replaced
)

func (AllocationResultType) String 

func (art AllocationResultType) String() string

type Application 

type Application struct {
	ID             string    // a formatted ULID
	ApplicationID  string    // application ID
	Partition      string    // partition Name
	PartitionID    string    // partition ID (a formatted ULID)
	SubmissionTime time.Time // time application was submitted

	locking.RWMutex
	// contains filtered or unexported fields
}

func NewApplication 

func NewApplication(siApp *si.AddApplicationRequest, ugi security.UserGroup, eventHandler handler.EventHandler, rmID string) *Application

func NewTestApplication 

func NewTestApplication(t *testing.T) *Application

func (*Application) AddAllocation 

func (sa *Application) AddAllocation(alloc *Allocation)

Add a new Allocation to the application

func (*Application) AddAllocationAsk 

func (sa *Application) AddAllocationAsk(ask *Allocation) error

Add an allocation ask to this application If the ask already exist update the existing info

func (*Application) AllocateAsk 

func (sa *Application) AllocateAsk(allocKey string) (*resources.Resource, error)

func (*Application) CleanupTrackedResource 

func (sa *Application) CleanupTrackedResource()

func (*Application) CurrentState 

func (sa *Application) CurrentState() string

Return the current state or a checked specific state for the application. The state machine handles the locking.

func (*Application) DAO 

func (app *Application) DAO() *dao.ApplicationDAOInfo

func (*Application) DeallocateAsk 

func (sa *Application) DeallocateAsk(allocKey string) (*resources.Resource, error)

func (*Application) FailApplication 

func (sa *Application) FailApplication(failureMessage string) error

FailApplication fails this application.

func (*Application) FinishedTime 

func (sa *Application) FinishedTime() time.Time

func (*Application) GetAllAllocations 

func (sa *Application) GetAllAllocations() []*Allocation

get a copy of all allocations of the application

func (*Application) GetAllPlaceholderData 

func (sa *Application) GetAllPlaceholderData() []*PlaceholderData

func (*Application) GetAllRequests 

func (sa *Application) GetAllRequests() []*Allocation

GetAllRequests returns a copy of all requests of the application

func (*Application) GetAllocatedResource 

func (sa *Application) GetAllocatedResource() *resources.Resource

Return the allocated resources for this application

func (*Application) GetAllocationAsk 

func (sa *Application) GetAllocationAsk(allocationKey string) *Allocation

Return the allocation ask for the key, nil if not found

func (*Application) GetApplicationSummary 

func (sa *Application) GetApplicationSummary(rmID string) *ApplicationSummary

func (*Application) GetAskMaxPriority 

func (sa *Application) GetAskMaxPriority() int32

func (*Application) GetAskReservations 

func (sa *Application) GetAskReservations(allocKey string) []string

Return the allocation reservations on any node. The returned array is 0 or more keys into the reservations map. No locking must be called while holding the lock

func (*Application) GetGuaranteedResource 

func (sa *Application) GetGuaranteedResource() *resources.Resource

GetGuaranteedResource returns the guaranteed resource that is set in the application tags

func (*Application) GetMaxAllocatedResource 

func (sa *Application) GetMaxAllocatedResource() *resources.Resource

func (*Application) GetMaxApps 

func (sa *Application) GetMaxApps() uint64

GetMaxApps returns the max apps that is set in the application tags

func (*Application) GetMaxResource 

func (sa *Application) GetMaxResource() *resources.Resource

GetMaxResource returns the max resource that is set in the application tags

func (*Application) GetPendingResource 

func (sa *Application) GetPendingResource() *resources.Resource

Return the pending resources for this application

func (*Application) GetPlaceholderAsk 

func (sa *Application) GetPlaceholderAsk() *resources.Resource

Return the total placeholder ask for this application Is only set on app creation and used when app is added to a queue

func (*Application) GetPlaceholderResource 

func (sa *Application) GetPlaceholderResource() *resources.Resource

Return the allocated placeholder resources for this application

func (*Application) GetQueue 

func (sa *Application) GetQueue() *Queue

func (*Application) GetQueuePath 

func (sa *Application) GetQueuePath() string

func (*Application) GetRejectedMessage 

func (sa *Application) GetRejectedMessage() string

func (*Application) GetReservations 

func (sa *Application) GetReservations() []string

GetReservations returns an array of all reservation keys for the application. This will return an empty array if there are no reservations. Visible for tests

func (*Application) GetStateLog 

func (sa *Application) GetStateLog() []*StateLogEntry

func (*Application) GetTag 

func (sa *Application) GetTag(tag string) string

Get a tag from the application Note: tags are not case sensitive

func (*Application) GetUser 

func (sa *Application) GetUser() security.UserGroup

get a copy of the user details for the application

func (*Application) HandleApplicationEvent 

func (sa *Application) HandleApplicationEvent(event applicationEvent) error

HandleApplicationEvent handles the state event for the application. The application lock is expected to be held.

func (*Application) HandleApplicationEventWithInfo 

func (sa *Application) HandleApplicationEventWithInfo(event applicationEvent, eventInfo string) error

HandleApplicationEventWithInfo handles the state event for the application with associated info object. The application lock is expected to be held.

func (*Application) HasPlaceholderAllocation 

func (sa *Application) HasPlaceholderAllocation() bool

func (*Application) HasReserved 

func (sa *Application) HasReserved() bool

HasReserved returns true if the application has any reservations.

func (*Application) IsAccepted 

func (sa *Application) IsAccepted() bool

func (*Application) IsAllocationAssignedToApp 

func (sa *Application) IsAllocationAssignedToApp(alloc *Allocation) bool

func (*Application) IsCompleted 

func (sa *Application) IsCompleted() bool

func (*Application) IsCompleting 

func (sa *Application) IsCompleting() bool

func (*Application) IsCreateForced 

func (sa *Application) IsCreateForced() bool

func (*Application) IsExpired 

func (sa *Application) IsExpired() bool

func (*Application) IsFailed 

func (sa *Application) IsFailed() bool

func (*Application) IsFailing 

func (sa *Application) IsFailing() bool

func (*Application) IsNew 

func (sa *Application) IsNew() bool

func (*Application) IsRejected 

func (sa *Application) IsRejected() bool

func (*Application) IsReservedOnNode 

func (sa *Application) IsReservedOnNode(nodeID string) bool

IsReservedOnNode returns true if and only if the node has been reserved by the application An empty nodeID is never reserved.

func (*Application) IsResuming 

func (sa *Application) IsResuming() bool

func (*Application) IsRunning 

func (sa *Application) IsRunning() bool

func (*Application) LogAppSummary 

func (sa *Application) LogAppSummary(rmID string)

func (*Application) OnStateChange 

func (sa *Application) OnStateChange(event *fsm.Event, eventInfo string)

OnStatChange every time the application enters a new state. It sends an event about the state change to the shim as an application update. The only state that does not generate an event is Rejected.

func (*Application) RecoverAllocationAsk 

func (sa *Application) RecoverAllocationAsk(alloc *Allocation)

Add the ask when a node allocation is recovered. Safeguarded against a nil but the recovery generates the ask and should never be nil.

func (*Application) RejectApplication 

func (sa *Application) RejectApplication(rejectedMessage string) error

RejectApplication rejects this application.

func (*Application) RemoveAllAllocations 

func (sa *Application) RemoveAllAllocations() []*Allocation

Remove all allocations from the application. All allocations that have been removed are returned.

func (*Application) RemoveAllocation 

func (sa *Application) RemoveAllocation(allocationKey string, releaseType si.TerminationType) *Allocation

Remove the Allocation from the application. Return the allocation that was removed or nil if not found.

func (*Application) RemoveAllocationAsk 

func (sa *Application) RemoveAllocationAsk(allocKey string) int

Remove one or more allocation asks from this application. This also removes any reservations that are linked to the ask. The return value is the number of reservations released

func (*Application) ReplaceAllocation 

func (sa *Application) ReplaceAllocation(allocationKey string) *Allocation

func (*Application) Reserve 

func (sa *Application) Reserve(node *Node, ask *Allocation) error

Reserve the application for this node and ask combination. If the reservation fails the function returns false, if the reservation is made it returns true. If the node and ask combination was already reserved for the application this is a noop and returns true.

func (*Application) SetQueue 

func (sa *Application) SetQueue(queue *Queue)

Set the leaf queue the application runs in.

func (*Application) SetQueuePath 

func (sa *Application) SetQueuePath(queuePath string)

Set the leaf queue the application runs in. The queue will be created when the app is added to the partition. The queue name is set to what the placement rule returned.

func (*Application) SetState 

func (sa *Application) SetState(state string)

func (*Application) SetTerminatedCallback 

func (sa *Application) SetTerminatedCallback(callback func(appID string))

func (*Application) SetTimedOutPlaceholder 

func (sa *Application) SetTimedOutPlaceholder(taskGroupName string, timedOut int64)

SetTimedOutPlaceholder should be used for testing only.

func (*Application) StartTime 

func (sa *Application) StartTime() time.Time

func (*Application) String 

func (sa *Application) String() string

func (*Application) UnReserve 

func (sa *Application) UnReserve(node *Node, ask *Allocation) (int, error)

UnReserve the application for this node and ask combination. This first removes the reservation from the node. If the reservation does not exist it returns 0 for reservations removed, if the reservation is removed it returns 1. The error is set if the reservation key cannot be removed from the app or node.

func (*Application) UnSetQueue 

func (sa *Application) UnSetQueue()

remove the leaf queue the application runs in, used when completing the app

func (*Application) UpdateAllocationResources 

func (sa *Application) UpdateAllocationResources(alloc *Allocation) error

UpdateAllocationResources updates the app, queue, and user tracker with deltas for an allocation. If an existing allocation cannot be found or alloc is invalid, an error is returned.

type ApplicationSummary 

type ApplicationSummary struct {
	ApplicationID       string
	SubmissionTime      time.Time
	StartTime           time.Time
	FinishTime          time.Time
	User                string
	Queue               string
	State               string
	RmID                string
	ResourceUsage       *resources.TrackedResource
	PreemptedResource   *resources.TrackedResource
	PlaceholderResource *resources.TrackedResource
}

func (*ApplicationSummary) DoLogging 

func (as *ApplicationSummary) DoLogging()

func (*ApplicationSummary) String 

func (as *ApplicationSummary) String() string

type Node 

type Node struct {
	// Fields for fast access These fields are considered read only.
	// Values should only be set when creating a new node and never changed.
	ID          string
	NodeID      string
	Hostname    string
	Rackname    string
	Partition   string
	PartitionID string

	locking.RWMutex
	// contains filtered or unexported fields
}

func NewNode 

func NewNode(proto *si.NodeInfo) *Node

func (*Node) AddAllocation 

func (sn *Node) AddAllocation(alloc *Allocation)

AddAllocation adds the allocation to the node. Used resources will increase available will decrease. A nil Allocation makes no changes. Preempted resources must have been released already. Do a sanity check to make sure it still fits in the node and nothing has changed

func (*Node) AddListener 

func (sn *Node) AddListener(listener NodeListener)

func (*Node) CanAllocate 

func (sn *Node) CanAllocate(res *resources.Resource) bool

CanAllocate checks if the proposed allocation fits in the available resources. If the proposed allocation does not fit false is returned.

func (*Node) FitInNode 

func (sn *Node) FitInNode(resRequest *resources.Resource) bool

FitInNode checks if the request fits in the node. All resources types requested must match the resource types provided by the nodes. A request may ask for only a subset of the types, but the node must provide at least the resource types requested in a larger or equal quantity as requested.

func (*Node) GetAllocatedResource 

func (sn *Node) GetAllocatedResource() *resources.Resource

Get the allocated resource on this node.

func (*Node) GetAllocation 

func (sn *Node) GetAllocation(allocationKey string) *Allocation

Return the allocation based on the allocationKey of the allocation. returns nil if the allocation is not found

func (*Node) GetAttribute 

func (sn *Node) GetAttribute(key string) string

Get an attribute by name. The most used attributes can be directly accessed via the fields: HostName, RackName and Partition. This is a lock free call. All attributes are considered read only

func (*Node) GetAttributes 

func (sn *Node) GetAttributes() map[string]string

func (*Node) GetAvailableResource 

func (sn *Node) GetAvailableResource() *resources.Resource

Get the available resource on this node.

func (*Node) GetCapacity 

func (sn *Node) GetCapacity() *resources.Resource

func (*Node) GetFitInScoreForAvailableResource 

func (sn *Node) GetFitInScoreForAvailableResource(res *resources.Resource) float64

GetFitInScoreForAvailableResource calculates a fit in score for "res" based on the current available resources, avoiding cloning. The caller must ensure that "res" cannot change while this method is running.

func (*Node) GetForeignAllocations 

func (sn *Node) GetForeignAllocations() []*Allocation

GetForeignAllocations returns a copy of non-Yunikorn allocations on this node

func (*Node) GetInstanceType 

func (sn *Node) GetInstanceType() string

Get InstanceType of this node. This is a lock free call because all attributes are considered read only

func (*Node) GetOccupiedResource 

func (sn *Node) GetOccupiedResource() *resources.Resource

func (*Node) GetReservationKeys 

func (sn *Node) GetReservationKeys() []string

GetReservationKeys Return an array of all reservation keys for the node. This will return an empty array if there are no reservations. Visible for tests

func (*Node) GetReservations 

func (sn *Node) GetReservations() []*reservation

GetReservations returns all reservation made on this node

func (*Node) GetResourceUsageShares 

func (sn *Node) GetResourceUsageShares() map[string]float64

GetResourceUsageShares gets a map of name -> resource usages per type in shares (0 to 1). Can return NaN.

func (*Node) GetUtilizedResource 

func (sn *Node) GetUtilizedResource() *resources.Resource

Get the utilized resource on this node.

func (*Node) GetYunikornAllocations 

func (sn *Node) GetYunikornAllocations() []*Allocation

GetYunikornAllocations returns a copy of Yunikorn allocations on this node

func (*Node) IsReserved 

func (sn *Node) IsReserved() bool

Return if the node has been reserved by any application

func (*Node) IsSchedulable 

func (sn *Node) IsSchedulable() bool

Can this node be used in scheduling.

func (*Node) RemoveAllocation 

func (sn *Node) RemoveAllocation(allocationKey string) *Allocation

Remove the allocation to the node. Returns nil if the allocation was not found and no changes are made. If the allocation is found the Allocation removed is returned. Used resources will decrease available will increase as per the allocation removed.

func (*Node) RemoveListener 

func (sn *Node) RemoveListener(listener NodeListener)

func (*Node) ReplaceAllocation 

func (sn *Node) ReplaceAllocation(allocationKey string, replace *Allocation, delta *resources.Resource)

ReplaceAllocation replaces the placeholder with the real allocation on the node. The delta passed in is the difference in resource usage between placeholder and real allocation. It should always be a negative value or zero: it is a decrease in usage or no change

func (*Node) Reserve 

func (sn *Node) Reserve(app *Application, ask *Allocation) error

Reserve the node for this application and ask combination, if not reserved yet. The reservation is checked against the node resources. If the reservation fails the function returns false, if the reservation is made it returns true.

func (*Node) SendNodeAddedEvent 

func (sn *Node) SendNodeAddedEvent()

func (*Node) SendNodeRemovedEvent 

func (sn *Node) SendNodeRemovedEvent()

func (*Node) SetCapacity 

func (sn *Node) SetCapacity(newCapacity *resources.Resource) *resources.Resource

SetCapacity changes the node resource capacity and returns the resource delta. The delta is positive for an increased capacity and negative for a decrease.

func (*Node) SetOccupiedResource 

func (sn *Node) SetOccupiedResource(occupiedResource *resources.Resource)

func (*Node) SetSchedulable 

func (sn *Node) SetSchedulable(schedulable bool)

Set the node to unschedulable. This will cause the node to be skipped during the scheduling cycle. Visible for testing only

func (*Node) String 

func (sn *Node) String() string

func (*Node) TryAddAllocation 

func (sn *Node) TryAddAllocation(alloc *Allocation) bool

TryAddAllocation attempts to add the allocation to the node. Used resources will increase available will decrease. A nil Allocation makes no changes. Preempted resources must have been released already. Do a sanity check to make sure it still fits in the node and nothing has changed

func (*Node) UpdateAllocatedResource 

func (sn *Node) UpdateAllocatedResource(delta *resources.Resource)

type NodeCollection 

type NodeCollection interface {
	AddNode(node *Node) error
	RemoveNode(nodeID string) *Node
	GetNode(nodeID string) *Node
	GetNodeCount() int
	GetNodes() []*Node
	GetNodeIterator() NodeIterator
	GetFullNodeIterator() NodeIterator
	SetNodeSortingPolicy(policy NodeSortingPolicy)
	GetNodeSortingPolicy() NodeSortingPolicy
}

NodeCollection represents a collection of nodes for a partition. Implementations of this interface must be internally synchronized to avoid data races.

func NewNodeCollection 

func NewNodeCollection(partition string) NodeCollection

Create a new collection for the given partition.

type NodeIterator 

type NodeIterator interface {
	// ForEachNode Calls the provided function on the sorted Node object until it returns false
	ForEachNode(func(*Node) bool)
}

NodeIterator iterates over a list of nodes based on the defined policy

type NodeListener 

type NodeListener interface {
	NodeUpdated(sn *Node)
}

type NodeSortingPolicy 

type NodeSortingPolicy interface {
	PolicyType() policies.SortingPolicy
	ScoreNode(node *Node) float64
	ResourceWeights() map[string]float64
}

func NewNodeSortingPolicy 

func NewNodeSortingPolicy(policyType string, resourceWeights map[string]float64) NodeSortingPolicy

type ObjectEvent 

type ObjectEvent int

---------------------------------- object events these events are used for: partitions and managed queues ---------------------------------- 

const (
	Remove ObjectEvent = iota
	Start
	Stop
)

func (ObjectEvent) String 

func (oe ObjectEvent) String() string

type ObjectState 

type ObjectState int

---------------------------------- object states these states are used by: partitions and managed queues ---------------------------------- 

const (
	Active ObjectState = iota
	Draining
	Stopped
)

func (ObjectState) String 

func (os ObjectState) String() string

type PlaceholderData 

type PlaceholderData struct {
	TaskGroupName string
	Count         int64
	MinResource   *resources.Resource
	Replaced      int64
	TimedOut      int64
}

func (*PlaceholderData) DAO 

func (ph *PlaceholderData) DAO() *dao.PlaceholderDAOInfo

type PreemptionContext 

type PreemptionContext struct {
	locking.RWMutex
	// contains filtered or unexported fields
}

func NewRequiredNodePreemptor 

func NewRequiredNodePreemptor(node *Node, requiredAsk *Allocation) *PreemptionContext

func (*PreemptionContext) GetVictims 

func (p *PreemptionContext) GetVictims() []*Allocation

type Preemptor 

type Preemptor struct {
	// contains filtered or unexported fields
}

Preemptor encapsulates the functionality required for preemption victim selection

func NewPreemptor 

func NewPreemptor(application *Application, headRoom *resources.Resource, preemptionDelay time.Duration, ask *Allocation, iterator NodeIterator, nodesTried bool) *Preemptor

NewPreemptor creates a new preemptor. The preemptor itself is not thread safe, and assumes the application lock is held.

func (*Preemptor) CheckPreconditions 

func (p *Preemptor) CheckPreconditions() bool

CheckPreconditions performs simple sanity checks designed to determine if preemption should be attempted for an ask. If checks succeed, updates the ask preemption check time.

func (*Preemptor) TryPreemption 

func (p *Preemptor) TryPreemption() (*AllocationResult, bool)

type Queue 

type Queue struct {
	ID          string // A formatted ULID
	QueuePath   string // Fully qualified path for the queue
	Name        string // Queue name as in the config etc.
	PartitionID string // Partition ID (not name) in which this queue resides

	locking.RWMutex
	// contains filtered or unexported fields
}

Queue structure inside Scheduler

func NewConfiguredQueue 

func NewConfiguredQueue(conf configs.QueueConfig, parent *Queue, partitionID string) (*Queue, error)

NewConfiguredQueue creates a new queue from scratch based on the configuration lock free as it cannot be referenced yet

func NewDynamicQueue 

func NewDynamicQueue(name string, leaf bool, parent *Queue) (*Queue, error)

NewDynamicQueue creates a new queue to be added to the system based on the placement rules A dynamically added queue can never be the root queue so parent must be set lock free as it cannot be referenced yet

func NewRecoveryQueue 

func NewRecoveryQueue(parent *Queue) (*Queue, error)

NewRecoveryQueue creates a recovery queue if it does not exist. The recovery queue is a dynamic queue, but has an invalid name so that it cannot be directly referenced.

func NewTestQueue 

func NewTestQueue(t *testing.T) *Queue

func (*Queue) AddApplication 

func (sq *Queue) AddApplication(app *Application)

AddApplication adds the application to the queue. All checks are assumed to have passed before we get here. No update of pending resource is needed as it should not have any requests yet. Replaces the existing application without further checks.

func (*Queue) ApplyConf 

func (sq *Queue) ApplyConf(conf configs.QueueConfig) error

ApplyConf is the locked version of applyConf

func (*Queue) CheckAdminAccess 

func (sq *Queue) CheckAdminAccess(user security.UserGroup) bool

CheckAdminAccess checks if the user has access to the queue to perform administrative actions. The check is performed recursively: i.e. access to the parent allows access to this queue.

func (*Queue) CheckSubmitAccess 

func (sq *Queue) CheckSubmitAccess(user security.UserGroup) bool

CheckSubmitAccess checks if the user has access to the queue to submit an application. The check is performed recursively: i.e. access to the parent allows access to this queue. This will check both submitACL and adminACL.

func (*Queue) CurrentState 

func (sq *Queue) CurrentState() string

CurrentState returns the current state of the queue in string form.

func (*Queue) DAO 

func (sq *Queue) DAO(include bool) dao.PartitionQueueDAOInfo

func (*Queue) DecAllocatedResource 

func (sq *Queue) DecAllocatedResource(alloc *resources.Resource) error

DecAllocatedResource decrement the allocated resources for this queue (recursively) Guard against going below zero resources.

func (*Queue) DecPreemptingResource 

func (sq *Queue) DecPreemptingResource(alloc *resources.Resource)

DecPreemptingResource decrements the preempting resources for this queue (recursively).

func (*Queue) FindEligiblePreemptionVictims 

func (sq *Queue) FindEligiblePreemptionVictims(queuePath string, ask *Allocation) map[string]*QueuePreemptionSnapshot

FindEligiblePreemptionVictims is used to locate tasks which may be preempted for the given ask. queuePath is the fully-qualified path of the queue where ask resides ask is the ask we are attempting to preempt for return is a map of potential victims keyed by queue path

func (*Queue) FindQueueByAppID 

func (sq *Queue) FindQueueByAppID(appID string) *Queue

FindQueueByAppID searches the queue hierarchy for an application with the given appID and returns the queue it belongs to

func (*Queue) GetActualGuaranteedResource 

func (sq *Queue) GetActualGuaranteedResource() *resources.Resource

GetActualGuaranteedResources returns the actual (including parent) guaranteed resources for the queue.

func (*Queue) GetAllocatedResource 

func (sq *Queue) GetAllocatedResource() *resources.Resource

GetAllocatedResource returns a clone of the allocated resources for this queue.

func (*Queue) GetApplication 

func (sq *Queue) GetApplication(appID string) *Application

getApplication return the Application based on the ID.

func (*Queue) GetChildQueue 

func (sq *Queue) GetChildQueue(name string) *Queue

GetChildQueue returns a queue if the name exists in the child map as a key.

func (*Queue) GetCopyOfApps 

func (sq *Queue) GetCopyOfApps() map[string]*Application

GetCopyOfApps gets a shallow copy of all non-completed apps holding the lock

func (*Queue) GetCopyOfChildren 

func (sq *Queue) GetCopyOfChildren() map[string]*Queue

GetCopyOfChildren return a shallow copy of the child queue map. This is used by the partition manager to find all queues to clean however we can not guarantee that there is no new child added while we clean up since there is no overall lock on the scheduler. We'll need to test just before to make sure the parent is empty

func (*Queue) GetCurrentPriority 

func (sq *Queue) GetCurrentPriority() int32

func (*Queue) GetFairMaxResource 

func (sq *Queue) GetFairMaxResource() *resources.Resource

GetFairMaxResource computes the fair max resources for a given queue. Starting with the root, descend down to the target queue allowing children to override Resource values . If the root includes an explicit 0 value for a Resource, do not include it in the accumulator and treat it as missing. If no children provide a maximum capacity override, the resulting value will be the value found on the Root. It is useful for fair-scheduling to allow a ratio to be produced representing the rough utilization % of a given queue.

func (*Queue) GetGuaranteedResource 

func (sq *Queue) GetGuaranteedResource() *resources.Resource

GetGuaranteedResource returns a clone of the guaranteed resource for the queue.

func (*Queue) GetMaxApps 

func (sq *Queue) GetMaxApps() uint64

GetMaxApps returns the maximum number of applications that can run in this queue.

func (*Queue) GetMaxQueueSet 

func (sq *Queue) GetMaxQueueSet() *resources.Resource

GetMaxQueueSet returns the max resource for the queue. The max resource should never be larger than the max resource of the parent. The cluster size, which defines the root limit, is not relevant for this call. Contrary to the GetMaxResource call. This will return nil unless a limit is set. Used during scheduling in an auto-scaling cluster. NOTE: if a resource quantity is missing and a limit is defined the missing quantity will be seen as a limit of 0. When defining a limit you therefore should define all resource quantities.

func (*Queue) GetMaxResource 

func (sq *Queue) GetMaxResource() *resources.Resource

GetMaxResource returns the max resource for the queue. The max resource should never be larger than the max resource of the parent. The root queue always has its limit set to the total cluster size (dynamic based on node registration) In case there are no nodes in a newly started cluster and no queues have a limit configured this call will return nil. NOTE: if a resource quantity is missing and a limit is defined the missing quantity will be seen as a limit of 0. When defining a limit you therefore should define all resource quantities.

func (*Queue) GetPartitionQueueDAOInfo 

func (sq *Queue) GetPartitionQueueDAOInfo(include bool) dao.PartitionQueueDAOInfo

GetPartitionQueueDAOInfo returns the queue hierarchy as an object for a REST call. Include is false, which means that returns the specified queue object, but does not return the children of the specified queue.

func (*Queue) GetPendingResource 

func (sq *Queue) GetPendingResource() *resources.Resource

GetPendingResource returns the pending resources for this queue.

func (*Queue) GetPreemptingResource 

func (sq *Queue) GetPreemptingResource() *resources.Resource

GetPreemptingResource returns a clone of the preempting resources for this queue.

func (*Queue) GetPreemptionDelay 

func (sq *Queue) GetPreemptionDelay() time.Duration

func (*Queue) GetPreemptionPolicy 

func (sq *Queue) GetPreemptionPolicy() policies.PreemptionPolicy

func (*Queue) GetPriorityPolicyAndOffset 

func (sq *Queue) GetPriorityPolicyAndOffset() (policies.PriorityPolicy, int32)

func (*Queue) GetQueueOutstandingRequests 

func (sq *Queue) GetQueueOutstandingRequests(total *[]*Allocation)

GetQueueOutstandingRequests builds a slice of pending allocation asks that fits into the queue's headroom.

func (*Queue) GetQueuePath 

func (sq *Queue) GetQueuePath() string

GetQueuePath returns the fully qualified path of this queue.

func (*Queue) GetReservedApps 

func (sq *Queue) GetReservedApps() map[string]int

GetReservedApps returns a shallow copy of the reserved app list locked to prevent race conditions from event updates

func (*Queue) IncAllocatedResource 

func (sq *Queue) IncAllocatedResource(alloc *resources.Resource)

IncAllocatedResource increments the allocated resources for this queue (recursively). No queue limits are checked.

func (*Queue) IncPreemptingResource 

func (sq *Queue) IncPreemptingResource(alloc *resources.Resource)

IncPreemptingResource increments the preempting resources for this queue (recursively).

func (*Queue) IsDraining 

func (sq *Queue) IsDraining() bool

IsDraining returns true if the queue in Draining state. Existing applications will still be scheduled No new applications will be accepted.

func (*Queue) IsEmpty 

func (sq *Queue) IsEmpty() bool

IsEmpty returns true if a queue is empty based on the following definition: A parent queue is empty when it has no children left A leaf queue is empty when there are no applications left

func (*Queue) IsLeafQueue 

func (sq *Queue) IsLeafQueue() bool

IsLeafQueue returns true is the queue a leaf. Returns false for a parent queue.

func (*Queue) IsManaged 

func (sq *Queue) IsManaged() bool

IsManaged returns true for a managed queue. Returns false for a dynamic queue.

func (*Queue) IsPrioritySortEnabled 

func (sq *Queue) IsPrioritySortEnabled() bool

func (*Queue) IsRunning 

func (sq *Queue) IsRunning() bool

IsRunning returns true if the queue in Active state.

func (*Queue) IsStopped 

func (sq *Queue) IsStopped() bool

IsStopped returns true if the queue in Stopped state. The queue is skipped for scheduling in this state.

func (*Queue) MarkQueueForRemoval 

func (sq *Queue) MarkQueueForRemoval()

MarkQueueForRemoval marks the managed queue for removal from the system. This can be executed multiple times and is only effective the first time. This is a noop on an unmanaged queue.

func (*Queue) RemoveApplication 

func (sq *Queue) RemoveApplication(app *Application)

RemoveApplication removes the app from the list of tracked applications. Make sure that the app is assigned to this queue and not removed yet. If not found this call is a noop

func (*Queue) RemoveQueue 

func (sq *Queue) RemoveQueue() bool

RemoveQueue remove the queue from the structure. Since nothing is allocated there shouldn't be anything referencing this queue anymore. The real removal is the removal of the queue from the parent's child list. Use a read lock on this queue to prevent other changes but allow status checks etc.

func (*Queue) Reserve 

func (sq *Queue) Reserve(appID string)

Reserve increments the number of reservations for the application adding it to the map if needed. No checks this is only called when a reservation is processed using the app stored in the queue.

func (*Queue) SetMaxResource 

func (sq *Queue) SetMaxResource(max *resources.Resource)

SetMaxResource sets the max resource for the root queue. Called as part of adding or removing a node. Should only happen on the root, all other queues get it from the config via properties.

func (*Queue) SetMaxRunningApps 

func (sq *Queue) SetMaxRunningApps(maxApps uint64)

SetMaxRunningApps allows setting the maximum running apps on a queue

func (*Queue) SetResources 

func (sq *Queue) SetResources(guaranteedResource, maxResource *resources.Resource)

func (*Queue) String 

func (sq *Queue) String() string

func (*Queue) SupportTaskGroup 

func (sq *Queue) SupportTaskGroup() bool

SupportTaskGroup returns true if the queue supports task groups. FIFO policy is required to support this. NOTE: this call does not make sense for a parent queue, and always returns false

func (*Queue) TryAllocate 

func (sq *Queue) TryAllocate(iterator func() NodeIterator, fullIterator func() NodeIterator, getnode func(string) *Node, allowPreemption bool) *AllocationResult

TryAllocate tries to allocate a pending requests. This only gets called if there is a pending request on this queue or its children. This is a depth first algorithm: descend into the depth of the queue tree first. Child queues are sorted based on the configured queue sortPolicy. Queues without pending resources are skipped. Applications are sorted based on the application sortPolicy. Applications without pending resources are skipped. Lock free call this all locks are taken when needed in called functions

func (*Queue) TryIncAllocatedResource 

func (sq *Queue) TryIncAllocatedResource(alloc *resources.Resource) error

TryIncAllocatedResource increments the allocated resources for this queue (recursively). Guard against going over max resources if set

func (*Queue) TryPlaceholderAllocate 

func (sq *Queue) TryPlaceholderAllocate(iterator func() NodeIterator, getnode func(string) *Node) *AllocationResult

TryPlaceholderAllocate tries to replace a placeholders with a real allocation. This only gets called if there is a pending request on this queue or its children. This is a depth first algorithm: descend into the depth of the queue tree first. Child queues are sorted based on the configured queue sortPolicy. Queues without pending resources are skipped. Applications are sorted based on the application sortPolicy. Applications without pending resources are skipped. Lock free call this all locks are taken when needed in called functions

func (*Queue) TryReservedAllocate 

func (sq *Queue) TryReservedAllocate(iterator func() NodeIterator) *AllocationResult

TryReservedAllocate tries to allocate a reservation. This only gets called if there is a pending request on this queue or its children. This is a depth first algorithm: descend into the depth of the queue tree first. Child queues are sorted based on the configured queue sortPolicy. Queues without pending resources are skipped. Applications are currently NOT sorted and are iterated over in a random order. Lock free call this all locks are taken when needed in called functions

func (*Queue) UnReserve 

func (sq *Queue) UnReserve(appID string, releases int)

UnReserve decrements the number of reservations for the application removing it to the map if all reservations are removed. No checks this is only called when a reservation is processed using the app stored in the queue.

func (*Queue) UpdateApplicationPriority 

func (sq *Queue) UpdateApplicationPriority(applicationID string, priority int32)

func (*Queue) UpdateQueuePriority 

func (sq *Queue) UpdateQueuePriority(queueName string, priority int32)

func (*Queue) UpdateQueueProperties 

func (sq *Queue) UpdateQueueProperties()

UpdateQueueProperties updates the queue properties defined as text

type QueuePreemptionSnapshot 

type QueuePreemptionSnapshot struct {
	Parent             *QueuePreemptionSnapshot // snapshot of parent queue
	QueuePath          string                   // fully qualified path to queue
	Leaf               bool                     // true if queue is a leaf queue
	AllocatedResource  *resources.Resource      // allocated resources
	PreemptingResource *resources.Resource      // resources currently flagged for preemption
	MaxResource        *resources.Resource      // maximum resources for this queue
	GuaranteedResource *resources.Resource      // guaranteed resources for this queue
	PotentialVictims   []*Allocation            // list of allocations which could be preempted
	AskQueue           *QueuePreemptionSnapshot // snapshot of ask or preemptor queue
}

QueuePreemptionSnapshot is used to track a snapshot of a queue for preemption

func (*QueuePreemptionSnapshot) AddAllocation 

func (qps *QueuePreemptionSnapshot) AddAllocation(alloc *resources.Resource)

AddAllocation adds an allocation to this snapshot's resource usage

func (*QueuePreemptionSnapshot) Duplicate 

func (qps *QueuePreemptionSnapshot) Duplicate(copy map[string]*QueuePreemptionSnapshot) *QueuePreemptionSnapshot

Duplicate creates a copy of this snapshot into the given map by queue path

func (*QueuePreemptionSnapshot) GetGuaranteedResource 

func (qps *QueuePreemptionSnapshot) GetGuaranteedResource() *resources.Resource

GetGuaranteedResource computes the current guaranteed resources considering parent guaranteed

func (*QueuePreemptionSnapshot) GetMaxResource 

func (qps *QueuePreemptionSnapshot) GetMaxResource() *resources.Resource

GetMaxResource computes the current max resources considering parent max

func (*QueuePreemptionSnapshot) GetPreemptableResource 

func (qps *QueuePreemptionSnapshot) GetPreemptableResource() *resources.Resource

func (*QueuePreemptionSnapshot) GetRemainingGuaranteedResource 

func (qps *QueuePreemptionSnapshot) GetRemainingGuaranteedResource() *resources.Resource

func (*QueuePreemptionSnapshot) RemoveAllocation 

func (qps *QueuePreemptionSnapshot) RemoveAllocation(alloc *resources.Resource)

RemoveAllocation removes an allocation from this snapshot's resource usage

type StateLogEntry 

type StateLogEntry struct {
	Time             time.Time
	ApplicationState string
}

func (*StateLogEntry) DAO 

func (entry *StateLogEntry) DAO() *dao.StateDAOInfo

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.