base

package
v1.0.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 17, 2024 License: Apache-2.0 Imports: 30 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	MountModeSelectAll  = "All"
	MountModeSelectNone = "None"
)

Variables

View Source 
var SupportedMountModes = []MountMode{MountPodMountMode, SidecarMountMode}

Functions

func CheckReferenceDataset added in v0.9.0 

func CheckReferenceDataset(dataset *datav1alpha1.Dataset) (check bool, err error)

func GetDatasetRefName added in v0.9.0 

func GetDatasetRefName(name, namespace string) string

func GetMetadataListFromAnnotation added in v0.9.0 

func GetMetadataListFromAnnotation(accessor metav1.ObjectMetaAccessor) (ret []datav1alpha1.Metadata)

func GetPhysicalDatasetFromMounts added in v1.0.0 

func GetPhysicalDatasetFromMounts(mounts []datav1alpha1.Mount) []types.NamespacedName

func GetPhysicalDatasetSubPath added in v1.0.0 

func GetPhysicalDatasetSubPath(virtualDataset *datav1alpha1.Dataset) []string

func GetRuntimeStatus added in v0.9.0 

func GetRuntimeStatus(client client.Client, runtimeType, name, namespace string) (status *datav1alpha1.RuntimeStatus, err error)

func InstallDataOperationHelmIfNotExist added in v0.9.0 

func InstallDataOperationHelmIfNotExist(ctx cruntime.ReconcileRequestContext, operation dataoperation.OperationInterface,
	yamlGenerator DataOperatorYamlGenerator) (err error)

func ParseMountModeSelectorFromStr added in v1.0.3 

func ParseMountModeSelectorFromStr(mountModeStr string) (mountModeSelector, error)

func RecordDatasetMetrics added in v0.9.0 

func RecordDatasetMetrics(result MetadataSyncResult, datasetNamespace, datasetName string, log logr.Logger)

RecordDatasetMetrics records dataset-related metrics from the given MetadataSyncResult

func ReleaseTargetDataset added in v0.9.0 

func ReleaseTargetDataset(ctx cruntime.ReconcileRequestContext, operation dataoperation.OperationInterface) error

ReleaseTargetDataset release target dataset OperationRef field which marks the data operation being performed.

func SafeClose added in v0.9.0 

func SafeClose(ch chan MetadataSyncResult) (closed bool)

SafeClose closes the metadataSyncResultChannel but ignores panic when the channel is already closed. Returns true if the channel is already closed.

func SafeSend added in v0.9.0 

func SafeSend(ch chan MetadataSyncResult, result MetadataSyncResult) (closed bool)

SafeSend sends result to the metadataSyncResultChannel but ignores panic when the channel is already closed Returns true if the channel is already closed.

func SetDataOperationInTargetDataset added in v0.9.0 

func SetDataOperationInTargetDataset(ctx cruntime.ReconcileRequestContext, operation dataoperation.OperationInterface, engine *TemplateEngine) error

SetDataOperationInTargetDataset set status of target dataset to mark the data operation being performed.

Types

type CachePath added in v0.5.0 

type CachePath struct {
	Path string

	Quota *resource.Quantity
}

type Conventions added in v1.0.3 

type Conventions interface {
	GetPersistentVolumeName() string

	GetLabelNameForMemory() string

	GetLabelNameForDisk() string

	GetLabelNameForTotal() string

	GetCommonLabelName() string

	GetFuseLabelName() string

	GetRuntimeLabelName() string

	GetDatasetNumLabelName() string

	GetWorkerStatefulsetName() string
}

Conventions defines naming convention for all runtime. Conventions includes all the literal string used in Fluid to identify the relationship between a dataset(runtime) and its component(master, worker, fuse, persistentVolume, etc.).

type DataOperator added in v0.9.0 

type DataOperator interface {
	Operate(ctx cruntime.ReconcileRequestContext, opStatus *datav1alpha1.OperationStatus, operation dataoperation.OperationInterface) (ctrl.Result, error)
}

DataOperator is a common interface of TemplateEngine for Data Operations like DataBackup/DataLoad/DataMigrate etc.

type DataOperatorYamlGenerator added in v0.9.0 

type DataOperatorYamlGenerator interface {
	GetDataOperationValueFile(ctx cruntime.ReconcileRequestContext, operation dataoperation.OperationInterface) (valueFileName string, err error)
}

DataOperatorYamlGenerator is the implementation of DataOperator interface for runtime engine using TemplateEngine.

type Engine 

type Engine interface {
	// ID returns the id
	ID() string

	// Shutdown and clean up the engine
	Shutdown() error

	// Setup the engine
	Setup(ctx cruntime.ReconcileRequestContext) (ready bool, err error)

	// Setup the Volume
	CreateVolume() (err error)

	// Destroy the Volume
	DeleteVolume() (err error)

	// Sync syncs the alluxio runtime
	Sync(ctx cruntime.ReconcileRequestContext) error

	// Validate checks all spec fields of the Dataset and the Runtime
	Validate(ctx cruntime.ReconcileRequestContext) (err error)

	// DataOperator is a common interface for Data Operations like DataBackup/DataLoad/DataMigrate etc.
	DataOperator
}

Engine interface defines the interfaces that should be implemented by a distributed data caching Engine. Thread safety is required from implementations of this interface.

type Fuse added in v0.5.0 

type Fuse struct {
	NodeSelector map[string]string

	// CleanPolicy decides when to clean fuse pods.
	CleanPolicy datav1alpha1.FuseCleanPolicy

	// Metrics
	MetricsScrapeTarget mountModeSelector
}

type Implement 

type Implement interface {
	UnderFileSystemService

	DataOperatorYamlGenerator

	// CheckMasterReady checks if the master ready
	CheckMasterReady() (ready bool, err error)

	// CheckWorkersReady checks if the workers ready
	CheckWorkersReady() (ready bool, err error)

	// ShouldSetupMaster checks if we need to setup the master
	ShouldSetupMaster() (should bool, err error)

	// ShouldSetupWorkers checks if we need to setup the workers
	ShouldSetupWorkers() (should bool, err error)

	// ShouldCheckUFS checks if we should check the ufs
	ShouldCheckUFS() (should bool, err error)

	// SetupMaster setup the cache master
	SetupMaster() (err error)

	// SetupWorkers setup the cache worker
	SetupWorkers() (err error)

	// UpdateDatasetStatus update the status of Dataset according to the given phase
	UpdateDatasetStatus(phase datav1alpha1.DatasetPhase) (err error)

	// PrepareUFS prepare the mounts and metadata if it's not ready
	PrepareUFS() (err error)

	// ShouldUpdateUFS check if we need to update the ufs and return all ufs to update
	// If the ufs have changed and the engine supports add/remove mount points dynamically,
	// then we need to UpdateOnUFSChange
	ShouldUpdateUFS() (ufsToUpdate *utils.UFSToUpdate)

	// UpdateOnUFSChange update the mount point of Dataset if ufs change
	// if an engine doesn't support UpdateOnUFSChange, it need to return false
	UpdateOnUFSChange(ufsToUpdate *utils.UFSToUpdate) (ready bool, err error)

	// Shutdown and clean up the engine
	Shutdown() error

	// CheckRuntimeHealthy checks runtime healthy
	CheckRuntimeHealthy() (err error)

	// UpdateCacheOfDataset updates cache of the dataset
	UpdateCacheOfDataset() (err error)

	// CheckAndUpdateRuntimeStatus checks and updates the status
	CheckAndUpdateRuntimeStatus() (ready bool, err error)

	// CreateVolume create the pv and pvc for the Dataset
	CreateVolume() error

	// SyncReplicas syncs the replicas
	SyncReplicas(ctx cruntime.ReconcileRequestContext) error

	// SyncMetadata syncs all metadata from UFS
	SyncMetadata() (err error)

	// DeleteVolume Destroy the Volume
	DeleteVolume() (err error)

	// BindToDataset binds the engine to dataset
	BindToDataset() (err error)

	// CheckRuntimeReady checks if the runtime is ready
	CheckRuntimeReady() (ready bool)

	// SyncRuntime syncs the runtime spec
	SyncRuntime(ctx cruntime.ReconcileRequestContext) (changed bool, err error)

	// SyncScheduleInfoToCacheNodes Sync the scheduleInfo to cacheNodes
	SyncScheduleInfoToCacheNodes() (err error)

	// Validate checks all spec fields of the Dataset and the Runtime
	Validate(ctx cruntime.ReconcileRequestContext) (err error)
}

Implement is what the real engine should implement if it use the TemplateEngine

type Level added in v0.5.0 

type Level struct {
	MediumType common.MediumType

	VolumeType common.VolumeType

	VolumeSource datav1alpha1.VolumeSource

	CachePaths []CachePath

	High string

	Low string
}

type MetadataSyncResult added in v0.9.0 

type MetadataSyncResult struct {
	Done      bool
	StartTime time.Time
	UfsTotal  string
	FileNum   string
	Err       error
}

MetadataSyncResult describes result for asynchronous metadata sync

type MountMode added in v1.0.3 

type MountMode string
const (
	MountPodMountMode MountMode = "MountPod"
	SidecarMountMode  MountMode = "Sidecar"
)

Supported mount modes

type RuntimeInfo added in v0.5.0 

type RuntimeInfo struct {
	// contains filtered or unexported fields
}

The real Runtime Info should implement

func (*RuntimeInfo) GetAnnotations added in v1.0.3 

func (info *RuntimeInfo) GetAnnotations() map[string]string

func (*RuntimeInfo) GetCommonLabelName added in v0.6.0 

func (info *RuntimeInfo) GetCommonLabelName() string

func (*RuntimeInfo) GetDatasetNumLabelName added in v0.6.0 

func (info *RuntimeInfo) GetDatasetNumLabelName() string

GetDatasetNumLabelname get the label to record how much datasets on a node

func (*RuntimeInfo) GetFuseCleanPolicy added in v0.7.0 

func (info *RuntimeInfo) GetFuseCleanPolicy() datav1alpha1.FuseCleanPolicy

func (*RuntimeInfo) GetFuseContainerTemplate added in v1.0.0 

func (info *RuntimeInfo) GetFuseContainerTemplate() (template *common.FuseInjectionTemplate, err error)

GetFuseContainerTemplate collects the fuse container spec from the runtime's fuse daemonSet spec. The function summarizes fuse related information into the template and returns it. The template then can be freely modified according to need of the serverless platform.

func (*RuntimeInfo) GetFuseLabelName added in v0.7.0 

func (info *RuntimeInfo) GetFuseLabelName() string

GetFuseLabelName gets the label indicating a fuse running on some node.

func (*RuntimeInfo) GetFuseMetricsScrapeTarget added in v1.0.3 

func (info *RuntimeInfo) GetFuseMetricsScrapeTarget() mountModeSelector

func (*RuntimeInfo) GetFuseNodeSelector added in v1.0.1 

func (info *RuntimeInfo) GetFuseNodeSelector() (nodeSelector map[string]string)

GetFuseNodeSelector gets the fuse deploy mode

func (*RuntimeInfo) GetLabelNameForDisk added in v0.6.0 

func (info *RuntimeInfo) GetLabelNameForDisk() string

func (*RuntimeInfo) GetLabelNameForMemory added in v0.6.0 

func (info *RuntimeInfo) GetLabelNameForMemory() string

func (*RuntimeInfo) GetLabelNameForTotal added in v0.6.0 

func (info *RuntimeInfo) GetLabelNameForTotal() string

func (*RuntimeInfo) GetMetadataList added in v0.9.0 

func (info *RuntimeInfo) GetMetadataList() []datav1alpha1.Metadata

func (*RuntimeInfo) GetName added in v0.5.0 

func (info *RuntimeInfo) GetName() string

GetName gets name

func (*RuntimeInfo) GetNamespace added in v0.5.0 

func (info *RuntimeInfo) GetNamespace() string

GetNamespace gets namespace

func (*RuntimeInfo) GetPersistentVolumeName added in v0.6.0 

func (info *RuntimeInfo) GetPersistentVolumeName() string

func (*RuntimeInfo) GetRuntimeLabelName added in v0.6.0 

func (info *RuntimeInfo) GetRuntimeLabelName() string

func (*RuntimeInfo) GetRuntimeType added in v0.5.0 

func (info *RuntimeInfo) GetRuntimeType() string

GetRuntimeType gets runtime type

func (*RuntimeInfo) GetTieredStoreInfo added in v0.6.0 

func (info *RuntimeInfo) GetTieredStoreInfo() TieredStoreInfo

func (*RuntimeInfo) GetWorkerStatefulsetName added in v1.0.3 

func (info *RuntimeInfo) GetWorkerStatefulsetName() string

func (*RuntimeInfo) IsDeprecatedNodeLabel added in v0.6.0 

func (info *RuntimeInfo) IsDeprecatedNodeLabel() bool

IsDeprecatedNodeLabel checks if using deprecated node label

func (*RuntimeInfo) IsDeprecatedPVName added in v0.6.0 

func (info *RuntimeInfo) IsDeprecatedPVName() bool

func (*RuntimeInfo) IsExclusive added in v0.5.0 

func (info *RuntimeInfo) IsExclusive() bool

IsExclusive determines if the runtime is exlusive

func (*RuntimeInfo) SetClient added in v0.7.0 

func (info *RuntimeInfo) SetClient(client client.Client)

func (*RuntimeInfo) SetDeprecatedNodeLabel added in v0.6.0 

func (info *RuntimeInfo) SetDeprecatedNodeLabel(deprecated bool)

SetDeprecatedNodeLabel set the DeprecatedNodeLabel

func (*RuntimeInfo) SetDeprecatedPVName added in v0.6.0 

func (info *RuntimeInfo) SetDeprecatedPVName(deprecated bool)

func (*RuntimeInfo) SetFuseNodeSelector added in v1.0.1 

func (info *RuntimeInfo) SetFuseNodeSelector(nodeSelector map[string]string)

SetFuseNodeSelector setups the fuse deploy mode

func (*RuntimeInfo) SetupFuseCleanPolicy added in v0.7.0 

func (info *RuntimeInfo) SetupFuseCleanPolicy(policy datav1alpha1.FuseCleanPolicy)

func (*RuntimeInfo) SetupWithDataset added in v0.5.0 

func (info *RuntimeInfo) SetupWithDataset(dataset *datav1alpha1.Dataset)

SetupWithDataset determines if need to setup with the info of dataset

type RuntimeInfoInterface added in v0.5.0 

type RuntimeInfoInterface interface {
	Conventions

	GetTieredStoreInfo() TieredStoreInfo

	GetName() string

	GetNamespace() string

	GetRuntimeType() string

	IsExclusive() bool

	SetFuseNodeSelector(nodeSelector map[string]string)

	SetupFuseCleanPolicy(policy datav1alpha1.FuseCleanPolicy)

	SetupWithDataset(dataset *datav1alpha1.Dataset)

	GetFuseNodeSelector() (nodeSelector map[string]string)

	GetFuseCleanPolicy() datav1alpha1.FuseCleanPolicy

	SetDeprecatedNodeLabel(deprecated bool)

	IsDeprecatedNodeLabel() bool

	SetDeprecatedPVName(deprecated bool)

	IsDeprecatedPVName() bool

	GetFuseContainerTemplate() (template *common.FuseInjectionTemplate, err error)

	SetClient(client client.Client)

	GetMetadataList() []datav1alpha1.Metadata

	GetAnnotations() map[string]string

	GetFuseMetricsScrapeTarget() mountModeSelector
}

Runtime Information interface defines the interfaces that should be implemented by Alluxio Runtime or other implementation . Thread safety is required from implementations of this interface.

func BuildRuntimeInfo added in v0.5.0 

func BuildRuntimeInfo(name string,
	namespace string,
	runtimeType string,
	opts ...RuntimeInfoOption) (runtime RuntimeInfoInterface, err error)

func GetRuntimeInfo added in v0.6.0 

func GetRuntimeInfo(client client.Client, name, namespace string) (runtimeInfo RuntimeInfoInterface, err error)

GetRuntimeInfo gets the RuntimeInfo according to name and namespace of it

type RuntimeInfoOption added in v0.9.0 

type RuntimeInfoOption func(info *RuntimeInfo) error

func WithAnnotations added in v1.0.3 

func WithAnnotations(annotations map[string]string) RuntimeInfoOption

func WithClientMetrics added in v1.0.3 

func WithClientMetrics(clientMetrics datav1alpha1.ClientMetrics) RuntimeInfoOption

func WithMetadataList added in v0.9.0 

func WithMetadataList(metadataList []datav1alpha1.Metadata) RuntimeInfoOption

func WithTieredStore added in v1.0.3 

func WithTieredStore(tieredStore datav1alpha1.TieredStore) RuntimeInfoOption

type RuntimeInterface added in v0.7.0 

type RuntimeInterface interface {

	// Replicas gets the replicas of runtime worker
	Replicas() int32

	// GetStatus gets the status of runtime
	GetStatus() *datav1alpha1.RuntimeStatus

	client.Object
}

type TemplateEngine 

type TemplateEngine struct {
	Implement
	Id string
	client.Client
	Log     logr.Logger
	Context cruntime.ReconcileRequestContext
	// contains filtered or unexported fields
}

func NewTemplateEngine 

func NewTemplateEngine(impl Implement,
	id string,

	context cruntime.ReconcileRequestContext) *TemplateEngine

NewTemplateEngine creates template engine

func (*TemplateEngine) CreateVolume 

func (t *TemplateEngine) CreateVolume() (err error)

Setup the CSI

func (*TemplateEngine) DeleteVolume 

func (t *TemplateEngine) DeleteVolume() (err error)

Setup the CSI

func (*TemplateEngine) ID 

func (t *TemplateEngine) ID() string

ID returns the id of the engine

func (*TemplateEngine) Operate added in v0.9.0 

func (t *TemplateEngine) Operate(ctx cruntime.ReconcileRequestContext, opStatus *datav1alpha1.OperationStatus,
	operation dataoperation.OperationInterface) (ctrl.Result, error)

func (*TemplateEngine) Setup 

func (b *TemplateEngine) Setup(ctx cruntime.ReconcileRequestContext) (ready bool, err error)

Setup the ddc engine

func (*TemplateEngine) Shutdown 

func (t *TemplateEngine) Shutdown() error

Shutdown and clean up the engine

func (*TemplateEngine) Sync 

func (t *TemplateEngine) Sync(ctx cruntime.ReconcileRequestContext) (err error)

SyncReplicas syncs the replicas

func (*TemplateEngine) Validate added in v1.0.2 

func (t *TemplateEngine) Validate(ctx cruntime.ReconcileRequestContext) (err error)

type TieredStoreInfo added in v0.6.0 

type TieredStoreInfo struct {
	Levels []Level
}

type UnderFileSystemService 

type UnderFileSystemService interface {
	UsedStorageBytes() (int64, error)

	FreeStorageBytes() (int64, error)

	TotalStorageBytes() (int64, error)

	TotalFileNums() (int64, error)
}

UnderFileSystemService interface defines the interfaces that should be implemented by a underlayer fileSystem service for the data. The implementation is the underlayer file system connector. It is responsible for checking ufs and preload the data. Thread safety is required from implementations of this interface.

Source Files

View all Source files

Directories

Path Synopsis
Package base is a generated GoMock package.
 Package base is a generated GoMock package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.