config

Documentation

Overview

Package config contains configuration for the nebulak8s module - which is global configuration for all Nebula K8s interactions. This config is under the subsection `k8s` and registered under the Plugin config All K8s based plugins can optionally use the nebulak8s module and this configuration allows controlling the defaults For example if for every container execution if some default Environment Variables or Annotations should be used, then they can be configured here An important configuration is ResourceTolerations that are applied to every container execution that needs some resource on the cluster

Index

• Constants
• Variables
• func SetK8sPluginConfig(cfg *K8sPluginConfig) error
• type K8sPluginConfig
  • func GetK8sPluginConfig() *K8sPluginConfig
  • func (cfg K8sPluginConfig) GetPFlagSet(prefix string) *pflag.FlagSet
• type NebulaCoPilotConfig

Constants

const ResourceNvidiaGPU v1.ResourceName = "nvidia.com/gpu"

ResourceNvidiaGPU is the name of the Nvidia GPU resource. Copied from: k8s.io/autoscaler/cluster-autoscaler/utils/gpu/gpu.go

Variables

var (

	// K8sPluginConfigSection provides a singular top level config section for all plugins.
	// If you are a plugin developer writing a k8s plugin, register your config section as a subsection to this.
	K8sPluginConfigSection = config.MustRegisterSubSection(k8sPluginConfigSectionKey, &defaultK8sConfig)
)

Functions

func SetK8sPluginConfig

func SetK8sPluginConfig(cfg *K8sPluginConfig) error

SetK8sPluginConfig should be used for TESTING ONLY, It Sets current value for the config.

Types

type K8sPluginConfig

type K8sPluginConfig struct {
	// InjectFinalizer is a boolean flag that indicates if a finalizer should be injected into every K8s resource launched
	InjectFinalizer bool `json:"inject-finalizer" pflag:",Instructs the plugin to inject a finalizer on startTask and remove it on task termination."`

	// Provide default annotations that should be added to K8s resource
	DefaultAnnotations map[string]string `json:"default-annotations" pflag:"-,Defines a set of default annotations to add to the produced pods."`
	// Provide default labels that should be added to K8s resource
	DefaultLabels map[string]string `json:"default-labels" pflag:"-,Defines a set of default labels to add to the produced pods."`
	// Provide additional environment variable pairs that plugin authors will provide to containers
	DefaultEnvVars map[string]string `json:"default-env-vars" pflag:"-,Additional environment variable that should be injected into every resource"`
	// Provide additional environment variable pairs whose values resolve from the plugin's execution environment.
	DefaultEnvVarsFromEnv map[string]string `json:"default-env-vars-from-env" pflag:"-,Additional environment variable that should be injected into every resource"`

	// default cpu requests for a container
	DefaultCPURequest resource.Quantity `json:"default-cpus" pflag:",Defines a default value for cpu for containers if not specified."`
	// default memory requests for a container
	DefaultMemoryRequest resource.Quantity `json:"default-memory" pflag:",Defines a default value for memory for containers if not specified."`

	// Default Tolerations that will be added to every Pod that is created by Nebula. These can be used in heterogeneous clusters, where one wishes to keep all pods created by Nebula on a separate
	// set of nodes.
	DefaultTolerations []v1.Toleration `` /* 146-byte string literal not displayed */
	// Default Node Selector Labels for pods. These NodeSelector labels are added to all pods, created by Nebula, unless they are marked as interruptible (default of interruptible are different).
	DefaultNodeSelector map[string]string `` /* 159-byte string literal not displayed */
	// Default Affinity that is applied to every pod that Nebula launches
	DefaultAffinity *v1.Affinity `` /* 154-byte string literal not displayed */

	// Default scheduler that should be used for all pods or CRD that accept Scheduler name.
	SchedulerName string `json:"scheduler-name" pflag:",Defines scheduler name."`

	// Tolerations for interruptible k8s pods: These tolerations are added to the pods that can tolerate getting evicted from a node. We
	// can leverage this for better bin-packing and using low-reliability cheaper machines.
	InterruptibleTolerations []v1.Toleration `json:"interruptible-tolerations"  pflag:"-,Tolerations to be applied for interruptible pods"`
	// Node Selector Labels for interruptible pods: Similar to InterruptibleTolerations, these node selector labels are added for pods that can tolerate
	// eviction.
	// Deprecated: Please use InterruptibleNodeSelectorRequirement/NonInterruptibleNodeSelectorRequirement
	InterruptibleNodeSelector map[string]string `json:"interruptible-node-selector" pflag:"-,Defines a set of node selector labels to add to the interruptible pods."`
	// Node Selector Requirements to be added to interruptible and non-interruptible
	// pods respectively
	InterruptibleNodeSelectorRequirement    *v1.NodeSelectorRequirement `json:"interruptible-node-selector-requirement" pflag:"-,Node selector requirement to add to interruptible pods"`
	NonInterruptibleNodeSelectorRequirement *v1.NodeSelectorRequirement `json:"non-interruptible-node-selector-requirement" pflag:"-,Node selector requirement to add to non-interruptible pods"`

	// Tolerations in the cluster that should be applied for a specific resource
	// Currently we support simple resource based tolerations only
	ResourceTolerations map[v1.ResourceName][]v1.Toleration `json:"resource-tolerations"  pflag:"-,Default tolerations to be applied for resource of type 'key'"`

	// Nebula CoPilot Configuration
	CoPilot NebulaCoPilotConfig `json:"co-pilot" pflag:",Co-Pilot Configuration"`

	// DeleteResourceOnFinalize instructs the system to delete the resource on finalize. This ensures that no resources
	// are kept around (potentially consuming cluster resources). This, however, will cause k8s log links to expire as
	// soon as the resource is finalized.
	DeleteResourceOnFinalize bool `` /* 361-byte string literal not displayed */

	// Time to wait for transient CreateContainerError errors to be resolved. If the
	// error persists past this grace period, it will be inferred to be a permanent
	// one, and the corresponding task marked as failed
	CreateContainerErrorGracePeriod config2.Duration `json:"create-container-error-grace-period" pflag:"-,Time to wait for transient CreateContainerError errors to be resolved."`

	// Time to wait for transient CreateContainerConfigError errors to be resolved. If the
	// error persists past this grace period, it will be inferred to be a permanent error.
	// The pod will be deleted, and the corresponding task marked as failed.
	CreateContainerConfigErrorGracePeriod config2.Duration `` /* 136-byte string literal not displayed */

	// Time to wait for transient ImagePullBackoff errors to be resolved. If the
	// error persists past this grace period, it will be inferred to be a permanent
	// one, and the corresponding task marked as failed
	ImagePullBackoffGracePeriod config2.Duration `json:"image-pull-backoff-grace-period" pflag:"-,Time to wait for transient ImagePullBackoff errors to be resolved."`

	// The node label that specifies the attached GPU device.
	GpuDeviceNodeLabel string `json:"gpu-device-node-label" pflag:"-,The node label that specifies the attached GPU device."`

	// The node label that specifies the attached GPU partition size.
	GpuPartitionSizeNodeLabel string `json:"gpu-partition-size-node-label" pflag:"-,The node label that specifies the attached GPU partition size."`

	// Override for node selector requirement added to pods intended for unpartitioned GPU nodes.
	GpuUnpartitionedNodeSelectorRequirement *v1.NodeSelectorRequirement `` /* 151-byte string literal not displayed */

	// Toleration added to pods intended for unpartitioned GPU nodes.
	GpuUnpartitionedToleration *v1.Toleration `json:"gpu-unpartitioned-toleration" pflag:"-,Toleration added to pods intended for unpartitioned GPU nodes."`

	// The name of the GPU resource to use when the task resource requests GPUs.
	GpuResourceName v1.ResourceName `json:"gpu-resource-name" pflag:"-,The name of the GPU resource to use when the task resource requests GPUs."`

	// DefaultPodSecurityContext provides a default pod security context that should be applied for every pod that is launched by NebulaPropeller. This may not be applicable to all plugins. For
	// downstream plugins - i.e. TensorflowOperators may not support setting this, but Spark does.
	DefaultPodSecurityContext *v1.PodSecurityContext `` /* 162-byte string literal not displayed */

	// DefaultSecurityContext provides a default container security context that should be applied for the primary container launched and created by NebulaPropeller. This may not be applicable to all plugins. For
	//	// downstream plugins - i.e. TensorflowOperators may not support setting this, but Spark does.
	DefaultSecurityContext *v1.SecurityContext `` /* 270-byte string literal not displayed */

	// EnableHostNetworkingPod is a binary switch to enable `hostNetwork: true` for all pods launched by Nebula.
	// Refer to - https://kubernetes.io/docs/concepts/policy/pod-security-policy/#host-namespaces.
	// As a follow up, the default pod configurations will now be adjusted using podTemplates per namespace
	EnableHostNetworkingPod *bool `json:"enable-host-networking-pod" pflag:"-,If true, will schedule all pods with hostNetwork: true."`

	// DefaultPodDNSConfig provides a default pod DNS config that that should be applied for the primary container launched and created by NebulaPropeller. This may not be applicable to all plugins. For
	//	// downstream plugins - i.e. TensorflowOperators may not support setting this.
	DefaultPodDNSConfig *v1.PodDNSConfig `` /* 144-byte string literal not displayed */

	// DefaultPodTemplateName that serves as the base PodTemplate for all k8s pods (including
	// individual containers) that are creating by NebulaPropeller.
	DefaultPodTemplateName string `` /* 129-byte string literal not displayed */

	// DefaultPodTemplateResync defines the frequency at which the k8s informer resyncs the default
	// pod template resources.
	DefaultPodTemplateResync config2.Duration `json:"default-pod-template-resync" pflag:",Frequency of resyncing default pod templates"`

	// SendObjectEvents indicates whether to send k8s object events in TaskExecutionEvent updates (similar to kubectl get events).
	SendObjectEvents bool `json:"send-object-events" pflag:",If true, will send k8s object events in TaskExecutionEvent updates."`
}

K8sPluginConfig should be used to configure per-pod defaults for the entire platform. This allows adding global defaults for pods that are being launched. For example, default annotations, labels, if a finalizer should be injected, if taints/tolerations should be used for certain resource types etc.

func GetK8sPluginConfig

func GetK8sPluginConfig() *K8sPluginConfig

GetK8sPluginConfig retrieves the current k8s plugin config or default.

func (K8sPluginConfig) GetPFlagSet

func (cfg K8sPluginConfig) GetPFlagSet(prefix string) *pflag.FlagSet

GetPFlagSet will return strongly types pflags for all fields in K8sPluginConfig and its nested types. The format of the flags is json-name.json-sub-name... etc.

type NebulaCoPilotConfig

type NebulaCoPilotConfig struct {
	// Co-pilot sidecar container name
	NamePrefix string `json:"name" pflag:",Nebula co-pilot sidecar container name prefix. (additional bits will be added after this)"`
	// Docker image FQN where co-pilot binary is installed
	Image string `json:"image" pflag:",Nebula co-pilot Docker Image FQN"`
	// Default Input Path for every task execution that uses co-pilot. This is used only if a input path is not provided by the user and inputs are required for the task
	DefaultInputDataPath string `json:"default-input-path" pflag:",Default path where the volume should be mounted"`
	// Default Output Path for every task execution that uses co-pilot. This is used only if a output path is not provided by the user and outputs are required for the task
	DefaultOutputPath string `json:"default-output-path" pflag:",Default path where the volume should be mounted"`
	// Name of the input volume
	InputVolumeName string `json:"input-vol-name" pflag:",Name of the data volume that is created for storing inputs"`
	// Name of the output volume
	OutputVolumeName string `json:"output-vol-name" pflag:",Name of the data volume that is created for storing outputs"`
	// Time for which the sidecar container should wait after starting up, for the primary process to appear. If it does not show up in this time
	// the process will be assumed to be dead or in a terminal condition and will trigger an abort.
	StartTimeout config2.Duration `` /* 142-byte string literal not displayed */
	// Resources for CoPilot Containers
	CPU     string `json:"cpu" pflag:",Used to set cpu for co-pilot containers"`
	Memory  string `json:"memory" pflag:",Used to set memory for co-pilot containers"`
	Storage string `json:"storage" pflag:",Default storage limit for individual inputs / outputs"`
}

NebulaCoPilotConfig specifies configuration for the Nebula CoPilot system. NebulaCoPilot, allows running nebulakit-less containers in K8s, where the IO is managed by the NebulaCoPilot sidecar process.