resourcetuner

Documentation

Index

• func NewResourceBasedTuner(opts ResourceBasedTunerOptions) (worker.WorkerTuner, error)
• type ResourceBasedSlotSupplier
  • func NewResourceBasedSlotSupplier(controller *ResourceController, options ResourceBasedSlotSupplierOptions) (*ResourceBasedSlotSupplier, error)
  • func (r *ResourceBasedSlotSupplier) MarkSlotUsed(worker.SlotMarkUsedInfo)
  • func (r *ResourceBasedSlotSupplier) MaxSlots() int
  • func (r *ResourceBasedSlotSupplier) ReleaseSlot(worker.SlotReleaseInfo)
  • func (r *ResourceBasedSlotSupplier) ReserveSlot(ctx context.Context, info worker.SlotReservationInfo) (*worker.SlotPermit, error)
  • func (r *ResourceBasedSlotSupplier) TryReserveSlot(info worker.SlotReservationInfo) *worker.SlotPermit
• type ResourceBasedSlotSupplierOptions
• type ResourceBasedTunerOptions
• type ResourceController
  • func NewResourceController(options ResourceControllerOptions) *ResourceController
• type ResourceControllerOptions
  • func DefaultResourceControllerOptions() ResourceControllerOptions
• type SystemInfoContext
• type SystemInfoSupplier

Functions

func NewResourceBasedTuner

func NewResourceBasedTuner(opts ResourceBasedTunerOptions) (worker.WorkerTuner, error)

NewResourceBasedTuner creates a WorkerTuner that dynamically adjusts the number of slots based on system resources. Specify the target CPU and memory usage as a value between 0 and 1.

WARNING: Resource based tuning is currently experimental.

Types

type ResourceBasedSlotSupplier

type ResourceBasedSlotSupplier struct {
	// contains filtered or unexported fields
}

ResourceBasedSlotSupplier is a worker.SlotSupplier that issues slots based on system resource usage.

WARNING: Resource based tuning is currently experimental.

func NewResourceBasedSlotSupplier

func NewResourceBasedSlotSupplier(
	controller *ResourceController,
	options ResourceBasedSlotSupplierOptions,
) (*ResourceBasedSlotSupplier, error)

NewResourceBasedSlotSupplier creates a ResourceBasedSlotSupplier given the provided ResourceController and ResourceBasedSlotSupplierOptions. All ResourceBasedSlotSupplier instances must use the same ResourceController.

WARNING: Resource based tuning is currently experimental.

func (*ResourceBasedSlotSupplier) MarkSlotUsed

func (r *ResourceBasedSlotSupplier) MarkSlotUsed(worker.SlotMarkUsedInfo)

func (*ResourceBasedSlotSupplier) MaxSlots

func (r *ResourceBasedSlotSupplier) MaxSlots() int

func (*ResourceBasedSlotSupplier) ReleaseSlot

func (r *ResourceBasedSlotSupplier) ReleaseSlot(worker.SlotReleaseInfo)

func (*ResourceBasedSlotSupplier) ReserveSlot

func (r *ResourceBasedSlotSupplier) ReserveSlot(ctx context.Context, info worker.SlotReservationInfo) (*worker.SlotPermit, error)

func (*ResourceBasedSlotSupplier) TryReserveSlot

func (r *ResourceBasedSlotSupplier) TryReserveSlot(info worker.SlotReservationInfo) *worker.SlotPermit

type ResourceBasedSlotSupplierOptions

type ResourceBasedSlotSupplierOptions struct {
	// MinSlots is minimum number of slots that will be issued without any resource checks.
	MinSlots int
	// MaxSlots is the maximum number of slots that will ever be issued.
	MaxSlots int
	// RampThrottle is time to wait between slot issuance. This value matters (particularly for
	// activities) because how many resources a task will use cannot be determined ahead of time,
	// and thus the system should wait to see how much resources are used before issuing more slots.
	RampThrottle time.Duration
}

ResourceBasedSlotSupplierOptions configures a particular ResourceBasedSlotSupplier.

WARNING: Resource based tuning is currently experimental.

type ResourceBasedTunerOptions

type ResourceBasedTunerOptions struct {
	// TargetMem is the target overall system memory usage as value 0 and 1 that the controller will
	// attempt to maintain. Must be set nonzero.
	TargetMem float64
	// TargetCpu is the target overall system CPU usage as value 0 and 1 that the controller will
	// attempt to maintain. Must be set nonzero.
	TargetCpu float64
	// Passed to ResourceBasedSlotSupplierOptions.RampThrottle for activities.
	// If not set, the default value is 50ms.
	ActivityRampThrottle time.Duration
	// Passed to ResourceBasedSlotSupplierOptions.RampThrottle for workflows.
	// If not set, the default value is 0ms.
	WorkflowRampThrottle time.Duration
}

type ResourceController

type ResourceController struct {
	// contains filtered or unexported fields
}

A ResourceController is used by ResourceBasedSlotSupplier to make decisions about whether slots should be issued based on system resource usage.

WARNING: Resource based tuning is currently experimental.

func NewResourceController

func NewResourceController(options ResourceControllerOptions) *ResourceController

NewResourceController creates a new ResourceController with the provided options. WARNING: It is important that you do not create multiple ResourceController instances. Since the controller looks at overall system resources, multiple instances with different configs can only conflict with one another.

WARNING: Resource based tuning is currently experimental.

type ResourceControllerOptions

type ResourceControllerOptions struct {
	// MemTargetPercent is the target overall system memory usage as value 0 and 1 that the
	// controller will attempt to maintain.
	MemTargetPercent float64
	// CpuTargetPercent is the target overall system CPU usage as value 0 and 1 that the controller
	// will attempt to maintain.
	CpuTargetPercent float64
	// SystemInfoSupplier is the supplier that the controller will use to get system resources.
	// Leave this nil to use the default implementation.
	InfoSupplier SystemInfoSupplier

	MemOutputThreshold float64
	CpuOutputThreshold float64

	MemPGain float64
	MemIGain float64
	MemDGain float64
	CpuPGain float64
	CpuIGain float64
	CpuDGain float64
}

ResourceControllerOptions contains configurable parameters for a ResourceController. It is recommended to use DefaultResourceControllerOptions to create a ResourceControllerOptions and only modify the mem/cpu target percent fields.

WARNING: Resource based tuning is currently experimental.

func DefaultResourceControllerOptions

func DefaultResourceControllerOptions() ResourceControllerOptions

DefaultResourceControllerOptions returns a ResourceControllerOptions with default values.

WARNING: Resource based tuning is currently experimental.

type SystemInfoContext

type SystemInfoContext struct {
	Logger log.Logger
}

type SystemInfoSupplier

type SystemInfoSupplier interface {
	// GetMemoryUsage returns the current system memory usage as a fraction of total memory between
	// 0 and 1.
	GetMemoryUsage(infoContext *SystemInfoContext) (float64, error)
	// GetCpuUsage returns the current system CPU usage as a fraction of total CPU usage between 0
	// and 1.
	GetCpuUsage(infoContext *SystemInfoContext) (float64, error)
}

SystemInfoSupplier implementations provide information about system resources.

WARNING: Resource based tuning is currently experimental.