app

package
v0.5.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 11, 2022 License: Apache-2.0 Imports: 49 Imported by: 0

README

The controllers are:

  • FybrikApplication controller - creates/updates a blueprint specification from the information in the FybrikApplication and from the policy compiler
  • Blueprint controller - reconciles the desired cluster state by orchestrating a blueprint It also configures lower level things like isolation policies.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func AllocateBucket 

func AllocateBucket(c client.Client, log logr.Logger, owner types.NamespacedName, id string, geo string) (*storage.ProvisionedBucket, error)

AllocateBucket allocates a bucket in the relevant geo The buckets are created as temporary, i.e. to be removed after the owner Dataset is deleted After a successful copy and registering a dataset, the bucket will become persistent

func AnalyzeError 

func AnalyzeError(application *api.FybrikApplication, assetID string, err error)

AnalyzeError analyzes whether the given error is fatal, or a retrial attempt can be made. Reasons for retrial can be either communication problems with external services, or kubernetes problems to perform some action on a resource. A retrial is achieved by returning an error to the reconcile method

func ConstructOpenAPIReq added in v0.5.0 

func ConstructOpenAPIReq(datasetID string, input *app.FybrikApplication, operation *taxonomymodels.PolicyManagerRequestAction) *taxonomymodels.PolicyManagerRequest

func CopyMap 

func CopyMap(m map[string]interface{}) map[string]interface{}

CopyMap copies a map

func GetSupportedReadSources 

func GetSupportedReadSources(module *app.FybrikModule) []*app.InterfaceDetails

GetSupportedReadSources returns a list of supported READ interfaces of a module

func LookupPolicyDecisions 

func LookupPolicyDecisions(datasetID string, policyManager connectors.PolicyManager, input *app.FybrikApplication, op *taxonomymodels.PolicyManagerRequestAction) ([]*taxonomymodels.PolicyManagerResultItem, error)

LookupPolicyDecisions provides a list of governance actions for the given dataset and the given operation

func SecretToCredentialMap 

func SecretToCredentialMap(cl client.Client, secretRef types.NamespacedName) (map[string]interface{}, error)

SecretToCredentialMap fetches a secret and converts into a map matching credentials proto

func SecretToCredentials 

func SecretToCredentials(cl client.Client, secretRef types.NamespacedName) (*pb.Credentials, error)

SecretToCredentials fetches a secret and constructs Credentials structure

func SetMapField 

func SetMapField(obj map[string]interface{}, k string, v interface{}) bool

SetMapField updates a map

Types

type BlueprintReconciler 

type BlueprintReconciler struct {
	client.Client
	Name   string
	Log    logr.Logger
	Scheme *runtime.Scheme
	Helmer helm.Interface
}

BlueprintReconciler reconciles a Blueprint object

func NewBlueprintReconciler 

func NewBlueprintReconciler(mgr ctrl.Manager, name string, helmer helm.Interface) *BlueprintReconciler

NewBlueprintReconciler creates a new reconciler for Blueprint resources

func (*BlueprintReconciler) Reconcile 

func (r *BlueprintReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)

Reconcile receives a Blueprint CRD

func (*BlueprintReconciler) SetupWithManager 

func (r *BlueprintReconciler) SetupWithManager(mgr ctrl.Manager) error

SetupWithManager registers Blueprint controller

type ContextInterface 

type ContextInterface interface {
	ResourceExists(ref *app.ResourceReference) bool
	CreateOrUpdateResource(owner *app.ResourceReference, ref *app.ResourceReference, plotterSpec *app.PlotterSpec, labels map[string]string) error
	DeleteResource(ref *app.ResourceReference) error
	GetResourceStatus(ref *app.ResourceReference) (app.ObservedState, error)
	CreateResourceReference(owner *app.ResourceReference) *app.ResourceReference
	GetManagedObject() runtime.Object
}

ContextInterface is an interface for communication with a generated resource (e.g. Blueprint)

type FybrikApplicationReconciler 

type FybrikApplicationReconciler struct {
	client.Client
	Name              string
	Log               logr.Logger
	Scheme            *runtime.Scheme
	PolicyManager     connectors.PolicyManager
	DataCatalog       connectors.DataCatalog
	ResourceInterface ContextInterface
	ClusterManager    multicluster.ClusterLister
	Provision         storage.ProvisionInterface
}

FybrikApplicationReconciler reconciles a FybrikApplication object

func NewFybrikApplicationReconciler 

func NewFybrikApplicationReconciler(mgr ctrl.Manager, name string,
	policyManager connectors.PolicyManager, catalog connectors.DataCatalog, cm multicluster.ClusterLister, provision storage.ProvisionInterface) *FybrikApplicationReconciler

NewFybrikApplicationReconciler creates a new reconciler for FybrikApplications

func (*FybrikApplicationReconciler) GetAllModules 

func (r *FybrikApplicationReconciler) GetAllModules() (map[string]*api.FybrikModule, error)

GetAllModules returns all CRDs of the kind FybrikModule mapped by their name

func (*FybrikApplicationReconciler) Reconcile 

func (r *FybrikApplicationReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)

Reconcile reconciles FybrikApplication CRD It receives FybrikApplication CRD and selects the appropriate modules that will run The outcome is a Plotter containing multiple Blueprints that run on different clusters

func (*FybrikApplicationReconciler) RegisterAsset 

func (r *FybrikApplicationReconciler) RegisterAsset(catalogID string, info *app.DatasetDetails, input *app.FybrikApplication) (string, error)

RegisterAsset registers a new asset in the specified catalog Input arguments: - catalogID: the destination catalog identifier - info: connection and credential details Returns: - an error if happened - the new asset identifier

func (*FybrikApplicationReconciler) SetupWithManager 

func (r *FybrikApplicationReconciler) SetupWithManager(mgr ctrl.Manager) error

SetupWithManager registers FybrikApplication controller

type ModuleManager 

type ModuleManager struct {
	Client             client.Client
	Log                logr.Logger
	Modules            map[string]*app.FybrikModule
	Clusters           []multicluster.Cluster
	Owner              types.NamespacedName
	PolicyManager      connectors.PolicyManager
	WorkloadGeography  string
	Provision          storage.ProvisionInterface
	VaultConnection    vault.Interface
	ProvisionedStorage map[string]NewAssetInfo
}

ModuleManager builds a set of modules based on the requirements (governance actions, data location) and the existing set of FybrikModules

func (*ModuleManager) AddFlowInfoForAsset added in v0.5.0 

func (m *ModuleManager) AddFlowInfoForAsset(item modules.DataInfo, appContext *app.FybrikApplication, plotterSpec *app.PlotterSpec, flowType app.DataFlow) error

SelectModuleInstances selects the necessary read/copy/write modules for the plotter for a given data set Adds the asset details, flows and templates to the given plotter spec. Write path is not yet implemented

func (*ModuleManager) GetCopyDestination 

func (m *ModuleManager) GetCopyDestination(item modules.DataInfo, destinationInterface *app.InterfaceDetails, geo string) (*app.DataStore, error)

GetCopyDestination creates a Dataset for bucket allocation by implicit copies or ingest.

func (*ModuleManager) GetProcessingGeography 

func (m *ModuleManager) GetProcessingGeography(applicationContext *app.FybrikApplication) (string, error)

GetProcessingGeography determines the geography of the workload cluster. If no cluster has been specified for a workload, a local cluster is assumed.

type NewAssetInfo 

type NewAssetInfo struct {
	Storage *storage.ProvisionedBucket
	Details *pb.DatasetDetails
}

NewAssetInfo points to the provisoned storage and hold information about the new asset

type PlotterInterface 

type PlotterInterface struct {
	Client client.Client
}

PlotterInterface context implementation for communication with a single Plotter resource

func NewPlotterInterface 

func NewPlotterInterface(cl client.Client) *PlotterInterface

NewPlotterInterface creates a new plotter interface for FybrikApplication controller

func (*PlotterInterface) CreateOrUpdateResource 

func (c *PlotterInterface) CreateOrUpdateResource(owner *app.ResourceReference, ref *app.ResourceReference, plotterSpec *app.PlotterSpec, labels map[string]string) error

CreateOrUpdateResource creates a new Plotter resource or updates an existing one

func (*PlotterInterface) CreateResourceReference 

func (c *PlotterInterface) CreateResourceReference(owner *app.ResourceReference) *app.ResourceReference

CreateResourceReference returns an identifier (name and namespace) of the generated resource.

func (*PlotterInterface) DeleteResource 

func (c *PlotterInterface) DeleteResource(ref *app.ResourceReference) error

DeleteResource deletes the generated Plotter resource

func (*PlotterInterface) GetManagedObject 

func (c *PlotterInterface) GetManagedObject() runtime.Object

GetManagedObject returns the type of the managed runtime object

func (*PlotterInterface) GetResourceSignature 

func (c *PlotterInterface) GetResourceSignature(ref *app.ResourceReference) *app.Plotter

GetResourceSignature returns the namespaced information of the generated Plotter resource

func (*PlotterInterface) GetResourceStatus 

func (c *PlotterInterface) GetResourceStatus(ref *app.ResourceReference) (app.ObservedState, error)

GetResourceStatus returns the generated Plotter status

func (*PlotterInterface) ResourceExists 

func (c *PlotterInterface) ResourceExists(ref *app.ResourceReference) bool

ResourceExists checks whether the Plotter resource generated by FybrikApplication controller is active

type PlotterModulesSpec added in v0.5.0 

type PlotterModulesSpec struct {
	ClusterName     string
	VaultAuthPath   string
	AssetID         string
	ModuleName      string
	ModuleArguments *app.StepParameters
	FlowType        app.DataFlow
	Chart           app.ChartSpec
	Scope           app.CapabilityScope
}

PlotterModulesSpec consists of module details extracted from the Plotter structure

type PlotterReconciler 

type PlotterReconciler struct {
	client.Client
	Name           string
	Log            logr.Logger
	Scheme         *runtime.Scheme
	ClusterManager multicluster.ClusterManager
}

PlotterReconciler reconciles a Plotter object

func NewPlotterReconciler 

func NewPlotterReconciler(mgr ctrl.Manager, name string, manager multicluster.ClusterManager) *PlotterReconciler

NewPlotterReconciler creates a new reconciler for Plotter resources

func (*PlotterReconciler) GenerateBlueprint added in v0.5.0 

func (r *PlotterReconciler) GenerateBlueprint(instances []modules.ModuleInstanceSpec, clusterName string) app.BlueprintSpec

GenerateBlueprint creates the Blueprint spec based on the datasets and the governance actions required, which dictate the modules that must run in the fybrik Credentials for accessing data set are stored in a credential management system (such as vault) and the paths for accessing them are included in the blueprint. The credentials themselves are not included in the blueprint.

func (*PlotterReconciler) GenerateBlueprints added in v0.5.0 

func (r *PlotterReconciler) GenerateBlueprints(instances []modules.ModuleInstanceSpec) map[string]app.BlueprintSpec

GenerateBlueprints creates Blueprint specs (one per cluster)

func (*PlotterReconciler) Reconcile 

func (r *PlotterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)

Reconcile receives a Plotter CRD

func (*PlotterReconciler) RefineInstances added in v0.5.0 

func (r *PlotterReconciler) RefineInstances(instances []modules.ModuleInstanceSpec) []modules.ModuleInstanceSpec

RefineInstances collects all instances of the same read/write module with non "Asset" scope and creates a new instance instead, with accumulated arguments.

func (*PlotterReconciler) SetupWithManager 

func (r *PlotterReconciler) SetupWithManager(mgr ctrl.Manager) error

SetupWithManager registers Plotter controller

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.