v1alpha1

package
v0.1.91-saahil.spark.i... Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 22, 2021 License: Apache-2.0 Imports: 14 Imported by: 0

Documentation

Overview

Package v1alpha1 contains API Schema definitions for the distributed-compute v1alpha1 API group +kubebuilder:object:generate=true +groupName=distributed-compute.dominodatalab.com

Index

Constants

This section is empty.

Variables

View Source 
var (
	// GroupVersion is group version used to register these objects
	GroupVersion = schema.GroupVersion{Group: "distributed-compute.dominodatalab.com", Version: "v1alpha1"}

	// SchemeBuilder is used to add go types to the GroupVersionKind scheme
	SchemeBuilder = &scheme.Builder{GroupVersion: GroupVersion}

	// AddToScheme adds the types in this group-version to the given scheme.
	AddToScheme = SchemeBuilder.AddToScheme
)

Functions

This section is empty.

Types

type Autoscaling 

type Autoscaling struct {
	// MinReplicas is the lower limit for the number of replicas to which the autoscaler can scale down.
	// This value must be greater than zero and less than the MaxReplicas.
	MinReplicas *int32 `json:"minReplicas,omitempty"`

	// MaxReplicas is the upper limit for the number of replicas to which the autoscaler can scale up.
	// This value cannot be less than the replica count for your workload.
	MaxReplicas int32 `json:"maxReplicas"`

	// AverageCPUUtilization is the target value of the average of the resource metric across all relevant pods.
	// This is represented as a percentage of the requested value of the resource for the pods.
	AverageCPUUtilization *int32 `json:"averageCPUUtilization,omitempty"`

	// ScaleDownStabilizationWindowSeconds is the number of seconds for which past recommendations should be considered
	// when scaling down. A shorter window will trigger scale down events quicker, but too short a window may cause
	// replica flapping when metrics used for scaling keep fluctuating.
	ScaleDownStabilizationWindowSeconds *int32 `json:"scaleDownStabilizationWindowSeconds,omitempty"`
}

Autoscaling configuration for scalable workloads.

func (*Autoscaling) DeepCopy 

func (in *Autoscaling) DeepCopy() *Autoscaling

DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Autoscaling.

func (*Autoscaling) DeepCopyInto 

func (in *Autoscaling) DeepCopyInto(out *Autoscaling)

DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.

type FrameworkConfig added in v0.4.0 

type FrameworkConfig struct {
	// Path at which to mount the configmap
	Path    string            `json:"path"`
	Configs map[string]string `json:"configs"`
}

func (*FrameworkConfig) DeepCopy added in v0.4.0 

func (in *FrameworkConfig) DeepCopy() *FrameworkConfig

DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new FrameworkConfig.

func (*FrameworkConfig) DeepCopyInto added in v0.4.0 

func (in *FrameworkConfig) DeepCopyInto(out *FrameworkConfig)

DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.

type IstioConfig 

type IstioConfig struct {
	// MutualTLSMode will be used to create a workload-specific peer
	// authentication policy that takes precedence over a global and/or
	// namespace-wide policy.
	MutualTLSMode string `json:"istioMutualTLSMode,omitempty"`
}

IstioConfig defines operator configuration parameters.

func (*IstioConfig) DeepCopy 

func (in *IstioConfig) DeepCopy() *IstioConfig

DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new IstioConfig.

func (*IstioConfig) DeepCopyInto 

func (in *IstioConfig) DeepCopyInto(out *IstioConfig)

DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.

type KeyTabConfig added in v0.4.0 

type KeyTabConfig struct {
	// Path at which to mount the configmap
	Path string `json:"path"`
	// need to map to a binary string
	KeyTab []byte `json:"configs"`
}

func (*KeyTabConfig) DeepCopy added in v0.4.0 

func (in *KeyTabConfig) DeepCopy() *KeyTabConfig

DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new KeyTabConfig.

func (*KeyTabConfig) DeepCopyInto added in v0.4.0 

func (in *KeyTabConfig) DeepCopyInto(out *KeyTabConfig)

DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.

type OCIImageDefinition 

type OCIImageDefinition struct {
	// Registry where the container image is hosted.
	Registry string `json:"registry,omitempty"`

	// Repository where the container image is stored.
	Repository string `json:"repository,omitempty"`

	// Tag points to a specific container image variant.
	Tag string `json:"tag,omitempty"`

	// PullPolicy used to fetch container image.
	PullPolicy corev1.PullPolicy `json:"pullPolicy,omitempty"`
}

OCIImageDefinition describes where and when to fetch a container image.

func (*OCIImageDefinition) DeepCopy 

func (in *OCIImageDefinition) DeepCopy() *OCIImageDefinition

DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OCIImageDefinition.

func (*OCIImageDefinition) DeepCopyInto 

func (in *OCIImageDefinition) DeepCopyInto(out *OCIImageDefinition)

DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.

type PersistentVolumeClaimTemplate 

type PersistentVolumeClaimTemplate struct {
	// Name is the unique metadata ID of the volume claim.
	Name string `json:"name"`

	// Spec describes the storage attributes of the underlying claim.
	Spec corev1.PersistentVolumeClaimSpec `json:"spec"`
}

PersistentVolumeClaimTemplate describes a claim that pods are allowed to reference. These can either pre-exist or leverage storage classes to provide dynamic provisioning.

func (*PersistentVolumeClaimTemplate) DeepCopy 

func (in *PersistentVolumeClaimTemplate) DeepCopy() *PersistentVolumeClaimTemplate

DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PersistentVolumeClaimTemplate.

func (*PersistentVolumeClaimTemplate) DeepCopyInto 

func (in *PersistentVolumeClaimTemplate) DeepCopyInto(out *PersistentVolumeClaimTemplate)

DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.

type RayCluster 

type RayCluster struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   RayClusterSpec   `json:"spec,omitempty"`
	Status RayClusterStatus `json:"status,omitempty"`
}

RayCluster is the Schema for the rayclusters API.

func (*RayCluster) DeepCopy 

func (in *RayCluster) DeepCopy() *RayCluster

DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RayCluster.

func (*RayCluster) DeepCopyInto 

func (in *RayCluster) DeepCopyInto(out *RayCluster)

DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.

func (*RayCluster) DeepCopyObject 

func (in *RayCluster) DeepCopyObject() runtime.Object

DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.

func (*RayCluster) Default 

func (r *RayCluster) Default()

Default implements webhook.Defaulter so a webhook will be registered for the type

func (*RayCluster) SetupWebhookWithManager 

func (r *RayCluster) SetupWebhookWithManager(mgr ctrl.Manager) error

SetupWebhookWithManager creates and registers this webhook with the manager.

func (*RayCluster) ValidateCreate 

func (r *RayCluster) ValidateCreate() error

ValidateCreate implements webhook.Validator so a webhook will be registered for the type.

func (*RayCluster) ValidateDelete 

func (r *RayCluster) ValidateDelete() error

ValidateDelete implements webhook.Validator so a webhook will be registered for the type. Not used, just here for interface compliance.

func (*RayCluster) ValidateUpdate 

func (r *RayCluster) ValidateUpdate(old runtime.Object) error

ValidateUpdate implements webhook.Validator so a webhook will be registered for the type.

type RayClusterHead 

type RayClusterHead struct {
	RayClusterNode `json:",inline"`
}

RayClusterHead defines head-specific pod settings.

func (*RayClusterHead) DeepCopy 

func (in *RayClusterHead) DeepCopy() *RayClusterHead

DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RayClusterHead.

func (*RayClusterHead) DeepCopyInto 

func (in *RayClusterHead) DeepCopyInto(out *RayClusterHead)

DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.

type RayClusterList 

type RayClusterList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []RayCluster `json:"items"`
}

RayClusterList contains a list of RayCluster resources.

func (*RayClusterList) DeepCopy 

func (in *RayClusterList) DeepCopy() *RayClusterList

DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RayClusterList.

func (*RayClusterList) DeepCopyInto 

func (in *RayClusterList) DeepCopyInto(out *RayClusterList)

DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.

func (*RayClusterList) DeepCopyObject 

func (in *RayClusterList) DeepCopyObject() runtime.Object

DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.

type RayClusterNetworkPolicy 

type RayClusterNetworkPolicy struct {
	// Enabled controls the creation of network policies that limit and provide
	// ingress access to the cluster nodes.
	Enabled *bool `json:"enabled,omitempty"`

	// ClientServerLabels defines the pod selector clause that grant ingress
	// access to the head client server port.
	ClientServerLabels map[string]string `json:"clientServerLabels,omitempty"`

	// DashboardLabels defines the pod selector clause used to grant ingress
	// access to the head dashboard port.
	DashboardLabels map[string]string `json:"dashboardLabels,omitempty"`
}

RayClusterNetworkPolicy defines network policy configuration options.

func (*RayClusterNetworkPolicy) DeepCopy 

func (in *RayClusterNetworkPolicy) DeepCopy() *RayClusterNetworkPolicy

DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RayClusterNetworkPolicy.

func (*RayClusterNetworkPolicy) DeepCopyInto 

func (in *RayClusterNetworkPolicy) DeepCopyInto(out *RayClusterNetworkPolicy)

DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.

type RayClusterNode 

type RayClusterNode struct {
	// Labels applied to ray pods in addition to stock labels.
	Labels map[string]string `json:"labels,omitempty"`

	// Annotations applied to ray pods.
	Annotations map[string]string `json:"annotations,omitempty"`

	// NodeSelector applied to ray pods.
	NodeSelector map[string]string `json:"nodeSelector,omitempty"`

	// Affinity applied to ray pods.
	Affinity *corev1.Affinity `json:"affinity,omitempty"`

	// Tolerations applied to ray pods.
	Tolerations []corev1.Toleration `json:"tolerations,omitempty"`

	// InitContainers added to ray pods.
	InitContainers []corev1.Container `json:"initContainers,omitempty"`

	// Volumes added to ray pods.
	Volumes []corev1.Volume `json:"volumes,omitempty"`

	// VolumeMounts added to ray containers.
	VolumeMounts []corev1.VolumeMount `json:"volumeMounts,omitempty"`

	// VolumeClaimTemplates is a list of claims that ray pods are allowed to
	// reference. You can enable dynamic provisioning of additional storage
	// on-demand by using a storage class provisioner.
	VolumeClaimTemplates []PersistentVolumeClaimTemplate `json:"volumeClaimTemplates,omitempty"`

	// Resources are the requests and limits applied to ray containers.
	Resources corev1.ResourceRequirements `json:"resources,omitempty"`
}

RayClusterNode defines attributes common to all ray node types.

func (*RayClusterNode) DeepCopy 

func (in *RayClusterNode) DeepCopy() *RayClusterNode

DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RayClusterNode.

func (*RayClusterNode) DeepCopyInto 

func (in *RayClusterNode) DeepCopyInto(out *RayClusterNode)

DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.

type RayClusterSpec 

type RayClusterSpec struct {
	// Image used to launch head and worker nodes.
	Image *OCIImageDefinition `json:"image,omitempty"`

	// ImagePullSecrets are references to secrets with credentials to private
	// registries used to pull images.
	ImagePullSecrets []corev1.LocalObjectReference `json:"imagePullSecrets,omitempty"`

	// Autoscaling parameters used to scale up/down ray worker nodes.
	Autoscaling *Autoscaling `json:"autoscaling,omitempty"`

	// NetworkPolicy parameters that grant intra-cluster and external network
	// access to cluster nodes.
	NetworkPolicy RayClusterNetworkPolicy `json:"networkPolicy,omitempty"`

	// Port is the port of the head ray process.
	Port int32 `json:"port,omitempty"`

	// RedisShardPorts is a list of ports for non-primary Redis shards.
	RedisShardPorts []int32 `json:"redisShardPorts,omitempty"`

	// ClientServerPort is the port number to which the ray client server will
	// bind. This port is used by external clients to submit work.
	ClientServerPort int32 `json:"clientServerPort,omitempty"`

	// ObjectManagerPort is the raylet port for the object manager.
	ObjectManagerPort int32 `json:"objectManagerPort,omitempty"`

	// NodeManagerPort is the raylet port for the node manager.
	NodeManagerPort int32 `json:"nodeManagerPort,omitempty"`

	// GCSServerPort is the port for the global control store.
	GCSServerPort int32 `json:"gcsServerPort,omitempty"`

	// WorkerPorts specifies the range of ports used by worker processes.
	WorkerPorts []int32 `json:"workerPorts,omitempty"`

	// ObjectStoreMemoryBytes is initial amount of memory with which to start
	// the object store.
	ObjectStoreMemoryBytes *int64 `json:"objectStoreMemoryBytes,omitempty"`

	// DashboardPort is the port used by the dashboard server.
	DashboardPort int32 `json:"dashboardPort,omitempty"`

	// EnableDashboard starts the dashboard web UI.
	EnableDashboard *bool `json:"enableDashboard,omitempty"`

	// PodSecurityPolicy name can be provided to govern execution of the ray
	// processes within pods.
	PodSecurityPolicy string `json:"podSecurityPolicy,omitempty"`

	// PodSecurityContext added to every ray pod.
	PodSecurityContext *corev1.PodSecurityContext `json:"podSecurityContext,omitempty"`

	// ServiceAccountName will disable the creation of a dedicated cluster
	// service account. The service account referenced by the provided name
	// will be used instead.
	ServiceAccountName string `json:"serviceAccountName,omitempty"`

	// EnvVars added to all every ray pod container.
	EnvVars []corev1.EnvVar `json:"envVars,omitempty"`

	// IstioConfig parameters for ray clusters.
	IstioConfig `json:",inline"`

	// Head node configuration parameters.
	Head RayClusterHead `json:"head,omitempty"`

	// Worker node configuration parameters.
	Worker RayClusterWorker `json:"worker,omitempty"`
}

RayClusterSpec defines the desired state of a RayCluster resource.

func (*RayClusterSpec) DeepCopy 

func (in *RayClusterSpec) DeepCopy() *RayClusterSpec

DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RayClusterSpec.

func (*RayClusterSpec) DeepCopyInto 

func (in *RayClusterSpec) DeepCopyInto(out *RayClusterSpec)

DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.

type RayClusterStatus 

type RayClusterStatus struct {
	// Nodes that comprise the cluster.
	Nodes []string `json:"nodes,omitempty"`

	// WorkerReplicas is the scale.status.replicas subresource field.
	WorkerReplicas int32 `json:"workerReplicas,omitempty"`

	// WorkerSelector is the scale.status.selector subresource field.
	WorkerSelector string `json:"workerSelector,omitempty"`
}

RayClusterStatus defines the observed state of a RayCluster resource.

func (*RayClusterStatus) DeepCopy 

func (in *RayClusterStatus) DeepCopy() *RayClusterStatus

DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RayClusterStatus.

func (*RayClusterStatus) DeepCopyInto 

func (in *RayClusterStatus) DeepCopyInto(out *RayClusterStatus)

DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.

type RayClusterWorker 

type RayClusterWorker struct {
	RayClusterNode `json:",inline"`

	// Replicas configures the total number of workers in the cluster. This
	// field behaves differently when Autoscaling is enabled. If
	// Autoscaling.MinReplicas is unspecified, then the minimum number of
	// replicas will be set to this value. Additionally, you can specify an
	// "initial cluster size" by setting this field to some value above the
	// minimum number of replicas.
	Replicas *int32 `json:"replicas,omitempty"`
}

RayClusterWorker defines worker-specific pod settings.

func (*RayClusterWorker) DeepCopy 

func (in *RayClusterWorker) DeepCopy() *RayClusterWorker

DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RayClusterWorker.

func (*RayClusterWorker) DeepCopyInto 

func (in *RayClusterWorker) DeepCopyInto(out *RayClusterWorker)

DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.

type SparkAdditionalStorage 

type SparkAdditionalStorage struct {
	// Sets the access mode which will be used when mounting the volume to the relevant pod
	AccessModes []corev1.PersistentVolumeAccessMode `json:"accessModes"`

	// Size of volume specified in any legal kubernetes units (i.e. 1Gi, 1000, etc)
	Size string `json:"size"`

	// Controls the storage class of the PersistentVolumeClaim
	StorageClass string `json:"storageClass"`

	// The base of the name that will be used both to name and subsequently mount this volume. This should match
	// an entry in VolumeMounts
	Name string `json:"name"`
}

func (*SparkAdditionalStorage) DeepCopy 

func (in *SparkAdditionalStorage) DeepCopy() *SparkAdditionalStorage

DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new SparkAdditionalStorage.

func (*SparkAdditionalStorage) DeepCopyInto 

func (in *SparkAdditionalStorage) DeepCopyInto(out *SparkAdditionalStorage)

DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.

type SparkCluster 

type SparkCluster struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   SparkClusterSpec   `json:"spec,omitempty"`
	Status SparkClusterStatus `json:"status,omitempty"`
}

SparkCluster is the Schema for the sparkclusters API.

func (*SparkCluster) DeepCopy 

func (in *SparkCluster) DeepCopy() *SparkCluster

DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new SparkCluster.

func (*SparkCluster) DeepCopyInto 

func (in *SparkCluster) DeepCopyInto(out *SparkCluster)

DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.

func (*SparkCluster) DeepCopyObject 

func (in *SparkCluster) DeepCopyObject() runtime.Object

DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.

func (*SparkCluster) Default 

func (r *SparkCluster) Default()

Default implements webhook.Defaulter so a webhook will be registered for the type nolint:gocyclo

func (*SparkCluster) SetupWebhookWithManager 

func (r *SparkCluster) SetupWebhookWithManager(mgr ctrl.Manager) error

SetupWebhookWithManager creates and registers this webhook with the manager.

func (*SparkCluster) ValidateCreate 

func (r *SparkCluster) ValidateCreate() error

ValidateCreate implements webhook.Validator so a webhook will be registered for the type.

func (*SparkCluster) ValidateDelete 

func (r *SparkCluster) ValidateDelete() error

ValidateDelete implements webhook.Validator so a webhook will be registered for the type. Not used, just here for interface compliance.

func (*SparkCluster) ValidateUpdate 

func (r *SparkCluster) ValidateUpdate(old runtime.Object) error

ValidateUpdate implements webhook.Validator so a webhook will be registered for the type.

type SparkClusterDriver added in v0.4.0 

type SparkClusterDriver struct {
	SparkClusterName           string `json:"sparkClusterName,omitempty"`
	ExecutionName              string `json:"executionName,omitempty"`
	DriverPortName             string `json:"driverPortName,omitempty"`
	DriverPort                 int32  `json:"driverPort,omitempty"`
	DriverBlockManagerPortName string `json:"driverBlockManagerPortName,omitempty"`
	DriverBlockManagerPort     int32  `json:"driverBlockManagerPort,omitempty"`
	DriverUIPortName           string `json:"driverUIPortName,omitempty"`
	DriverUIPort               int32  `json:"driverUIPort,omitempty"`
}

SparkClusterDriver defines the configuration for the driver service

func (*SparkClusterDriver) DeepCopy added in v0.4.0 

func (in *SparkClusterDriver) DeepCopy() *SparkClusterDriver

DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new SparkClusterDriver.

func (*SparkClusterDriver) DeepCopyInto added in v0.4.0 

func (in *SparkClusterDriver) DeepCopyInto(out *SparkClusterDriver)

DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.

type SparkClusterHead 

type SparkClusterHead struct {
	SparkClusterNode `json:",inline"`
}

SparkClusterHead defines head-specific pod settings.

func (*SparkClusterHead) DeepCopy 

func (in *SparkClusterHead) DeepCopy() *SparkClusterHead

DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new SparkClusterHead.

func (*SparkClusterHead) DeepCopyInto 

func (in *SparkClusterHead) DeepCopyInto(out *SparkClusterHead)

DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.

type SparkClusterList 

type SparkClusterList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []SparkCluster `json:"items"`
}

SparkClusterList contains a list of SparkCluster resources.

func (*SparkClusterList) DeepCopy 

func (in *SparkClusterList) DeepCopy() *SparkClusterList

DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new SparkClusterList.

func (*SparkClusterList) DeepCopyInto 

func (in *SparkClusterList) DeepCopyInto(out *SparkClusterList)

DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.

func (*SparkClusterList) DeepCopyObject 

func (in *SparkClusterList) DeepCopyObject() runtime.Object

DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.

type SparkClusterNetworkPolicy 

type SparkClusterNetworkPolicy struct {
	// Enabled controls the creation of network policies that limit and provide
	// ingress access to the cluster nodes.
	Enabled *bool `json:"enabled,omitempty"`

	// ClientServerLabels defines the pod selector clause that grant ingress
	// access to the head client server port.
	ClientServerLabels map[string]string `json:"clientServerLabels,omitempty"`

	// DashboardLabels defines the pod selector clause used to grant ingress
	// access to the head dashboard port.
	DashboardLabels map[string]string `json:"dashboardLabels,omitempty"`

	// ExternalPolicyEnabled controls creation of network policies which deal with two way traffic to pods that are
	// external to the cluster entirely. The spark driver, for example, is generally going to live outside the
	// cluster
	ExternalPolicyEnabled *bool `json:"externalPolicyEnabled,omitempty"`

	// ExternalPodLabels defines the pod selector clause for used to granted unfettered
	// access to cluster resources.
	ExternalPodLabels map[string]string `json:"clusterLabels,omitempty"`
}

SparkClusterNetworkPolicy defines network policy configuration options.

func (*SparkClusterNetworkPolicy) DeepCopy 

func (in *SparkClusterNetworkPolicy) DeepCopy() *SparkClusterNetworkPolicy

DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new SparkClusterNetworkPolicy.

func (*SparkClusterNetworkPolicy) DeepCopyInto 

func (in *SparkClusterNetworkPolicy) DeepCopyInto(out *SparkClusterNetworkPolicy)

DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.

type SparkClusterNode 

type SparkClusterNode struct {
	// Labels applied to spark pods in addition to stock labels.
	Labels map[string]string `json:"labels,omitempty"`

	// Annotations applied to spark pods.
	Annotations map[string]string `json:"annotations,omitempty"`

	// NodeSelector applied to spark pods.
	NodeSelector map[string]string `json:"nodeSelector,omitempty"`

	// Affinity applied to spark pods.
	Affinity *corev1.Affinity `json:"affinity,omitempty"`

	// Tolerations applied to spark pods.
	Tolerations []corev1.Toleration `json:"tolerations,omitempty"`

	// InitContainers added to spark pods.
	InitContainers []corev1.Container `json:"initContainers,omitempty"`

	// Volumes added to spark pods.
	Volumes []corev1.Volume `json:"volumes,omitempty"`

	// VolumeMounts added to spark containers.
	VolumeMounts []corev1.VolumeMount `json:"volumeMounts,omitempty"`

	// Resources are the requests and limits applied to spark containers.
	Resources corev1.ResourceRequirements `json:"resources"`

	// Requests for additional storage volumes to be created alongside each pod
	AdditionalStorage []SparkAdditionalStorage `json:"additionalStorage,omitempty"`

	// Extra framework-specific configuration for this cluster
	// For spark this means we'll generate a spark-defaults.conf config map
	// and mount it in to the requested location
	FrameworkConfig *FrameworkConfig `json:"frameworkConfig,omitempty"`

	KeyTabConfig *KeyTabConfig `json:"keyTabConfig,omitempty"`
}

SparkClusterNode defines attributes common to all spark node types.

func (*SparkClusterNode) DeepCopy 

func (in *SparkClusterNode) DeepCopy() *SparkClusterNode

DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new SparkClusterNode.

func (*SparkClusterNode) DeepCopyInto 

func (in *SparkClusterNode) DeepCopyInto(out *SparkClusterNode)

DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.

type SparkClusterSpec 

type SparkClusterSpec struct {
	// Image used to launch head and worker nodes.
	Image *OCIImageDefinition `json:"image,omitempty"`

	// ImagePullSecrets are references to secrets with credentials to private registries used to pull images.
	ImagePullSecrets []corev1.LocalObjectReference `json:"imagePullSecrets,omitempty"`

	// Autoscaling parameters used to scale up/down spark worker nodes.
	Autoscaling *Autoscaling `json:"autoscaling,omitempty"`

	// ClusterPort is the port on which the spark protocol communicates
	ClusterPort int32 `json:"clusterPort,omitempty"`

	// These two are meant for istio compatibility on spark
	TCPMasterWebPort int32 `json:"tcpMasterWebPort,omitempty"`
	TCPWorkerWebPort int32 `json:"tcpWorkerWebPort,omitempty"`

	// IstioConfig parameters for Spark clusters.
	IstioConfig `json:",inline"`

	// Driver is the configuration that is passed along from Nucleus to the DCO to set up the Spark Driver
	Driver SparkClusterDriver `json:"sparkClusterDriver,omitempty"`

	// DashboardPort is the port used by the dashboard server.
	DashboardPort int32 `json:"dashboardPort,omitempty"`

	// EnableDashboard starts the dashboard web UI.
	EnableDashboard *bool `json:"enableDashboard,omitempty"`

	// NetworkPolicyClientLabels will create a pod selector clause for each set of labels.
	// This is used to grant ingress access to one or more groups of external pods and is
	// only applicable when EnableNetworkPolicy is true.
	NetworkPolicy SparkClusterNetworkPolicy `json:"networkPolicy,omitempty"`

	// PodSecurityPolicy name can be provided to govern execution of the spark processes within pods.
	PodSecurityPolicy string `json:"podSecurityPolicy,omitempty"`

	// PodSecurityContext added to every spark pod.
	PodSecurityContext *corev1.PodSecurityContext `json:"podSecurityContext,omitempty"`

	// ServiceAccountName will disable the creation of a dedicated cluster service account.
	// The service account referenced by the provided name will be used instead.
	ServiceAccountName string `json:"serviceAccountName,omitempty"`

	// EnvVars added to every spark pod container.
	EnvVars []corev1.EnvVar `json:"envVars,omitempty"`

	// Master node configuration parameters.
	Master SparkClusterHead `json:"head,omitempty"`

	// Worker node configuration parameters.
	Worker SparkClusterWorker `json:"worker,omitempty"`
}

SparkClusterSpec defines the desired state of a SparkCluster resource.

func (*SparkClusterSpec) DeepCopy 

func (in *SparkClusterSpec) DeepCopy() *SparkClusterSpec

DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new SparkClusterSpec.

func (*SparkClusterSpec) DeepCopyInto 

func (in *SparkClusterSpec) DeepCopyInto(out *SparkClusterSpec)

DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.

type SparkClusterStatus 

type SparkClusterStatus struct {
	// Nodes that comprise the cluster.
	Nodes []string `json:"nodes,omitempty"`

	// WorkerReplicas is the scale.status.replicas subresource field.
	WorkerReplicas int32 `json:"workerReplicas,omitempty"`
	// WorkerSelector is the scale.status.selector subresource field.
	WorkerSelector string `json:"workerSelector,omitempty"`
}

SparkClusterStatus defines the observed state of a SparkCluster resource.

func (*SparkClusterStatus) DeepCopy 

func (in *SparkClusterStatus) DeepCopy() *SparkClusterStatus

DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new SparkClusterStatus.

func (*SparkClusterStatus) DeepCopyInto 

func (in *SparkClusterStatus) DeepCopyInto(out *SparkClusterStatus)

DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.

type SparkClusterWorker 

type SparkClusterWorker struct {
	SparkClusterNode `json:",inline"`

	// Replicas configures the total number of workers in the cluster.
	// This field behaves differently when Autoscaling is enabled. If Autoscaling.MinReplicas is unspecified, then the
	// minimum number of replicas will be set to this value. Additionally, you can specify an "initial cluster size" by
	// setting this field to some value above the minimum number of replicas.
	Replicas *int32 `json:"replicas,omitempty"`
}

SparkClusterWorker defines worker-specific pod settings.

func (*SparkClusterWorker) DeepCopy 

func (in *SparkClusterWorker) DeepCopy() *SparkClusterWorker

DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new SparkClusterWorker.

func (*SparkClusterWorker) DeepCopyInto 

func (in *SparkClusterWorker) DeepCopyInto(out *SparkClusterWorker)

DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.