app

package
v0.6.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 8, 2022 License: Apache-2.0 Imports: 60 Imported by: 0

README

The controllers are:

  • FybrikApplication controller - creates/updates a blueprint specification from the information in the FybrikApplication and from the policy compiler
  • Blueprint controller - reconciles the desired cluster state by orchestrating a blueprint It also configures lower level things like isolation policies.

Documentation

Index

Constants

View Source 
const (
	// ReadyCondition means that access to a dataset is granted
	ReadyConditionIndex int64 = 0
	// DenyCondition means that access to a dataset is denied
	DenyConditionIndex int64 = 1
	// ErrorCondition means that an error was encountered during blueprint construction
	ErrorConditionIndex int64 = 2
)

Condition indices are static. Conditions always present in the status.

View Source 
const (
	ApplicationTaxonomy = "/tmp/taxonomy/fybrik_application.json"
	DataCatalogTaxonomy = "/tmp/taxonomy/datacatalog.json#/definitions/GetAssetResponse"
)
View Source 
const (
	ModuleTaxonomy                 = "/tmp/taxonomy/fybrik_module.json"
	ModuleValidationConditionIndex = 0
)
View Source 
const (
	PolicyManagerTaxonomy = "/tmp/taxonomy/policymanager.json#/definitions/GetPolicyDecisionsResponse"
)
View Source 
const Transform = "transform"

Temporary hard-coded capability representing Actions of read/copy capability. A change to modules is requested to add "transform" as an additional capability to the existing read and copy modules.

Variables

This section is empty.

Functions

func AllocateBucket 

func AllocateBucket(c client.Client, log zerolog.Logger, bucketName string, geo string) (*storage.ProvisionedBucket, error)

AllocateBucket allocates a bucket in the relevant geo The buckets are created as temporary, i.e. to be removed after the owner Dataset is deleted After a successful copy and registering a dataset, the bucket will become persistent

func AnalyzeError 

func AnalyzeError(appContext ApplicationContext, assetID string, err error)

AnalyzeError analyzes whether the given error is fatal, or a retrial attempt can be made. Reasons for retrial can be either communication problems with external services, or kubernetes problems to perform some action on a resource. A retrial is achieved by returning an error to the reconcile method

func CheckDependencies added in v0.6.0 

func CheckDependencies(module *app.FybrikModule, moduleMap map[string]*app.FybrikModule) ([]*app.FybrikModule, []string)

CheckDependencies returns dependent modules

func ConstructOpenAPIReq added in v0.5.0 

func ConstructOpenAPIReq(datasetID string, input *app.FybrikApplication, operation *policymanager.RequestAction) *policymanager.GetPolicyDecisionsRequest

func CopyMap 

func CopyMap(m map[string]interface{}) map[string]interface{}

CopyMap copies a map

func CreateDataRequest added in v0.6.0 

func CreateDataRequest(application *api.FybrikApplication, dataCtx api.DataContext, assetMetadata *datacatalog.ResourceMetadata) adminconfig.DataRequest

CreateDataRequest generates a new DataRequest object for a specific asset based on FybrikApplication and asset metadata

func GetDependencies added in v0.6.0 

func GetDependencies(module *app.FybrikModule, moduleMap map[string]*app.FybrikModule) ([]*app.FybrikModule, error)

GetDependencies returns dependencies of a selected module

func LookupPolicyDecisions 

func LookupPolicyDecisions(datasetID string, policyManager connectors.PolicyManager, appContext ApplicationContext, op *policymanager.RequestAction) ([]taxonomy.Action, error)

LookupPolicyDecisions provides a list of governance actions for the given dataset and the given operation

func SecretToCredentialMap 

func SecretToCredentialMap(cl client.Client, secretRef types.NamespacedName) (map[string]interface{}, error)

SecretToCredentialMap fetches a secret and converts into a map matching credentials proto

func SecretToCredentials 

func SecretToCredentials(cl client.Client, secretRef types.NamespacedName) (string, error)

SecretToCredentials fetches a secret and constructs Credentials structure

func SetMapField 

func SetMapField(obj map[string]interface{}, k string, v interface{}) bool

SetMapField updates a map

func SupportsDependencies added in v0.6.0 

func SupportsDependencies(module *app.FybrikModule, moduleMap map[string]*app.FybrikModule) bool

SupportsDependencies checks whether the module supports the dependency requirements

func ValidateFybrikModule added in v0.6.0 

func ValidateFybrikModule(module *fapp.FybrikModule, taxonomyFile string) error

func ValidatePolicyDecisionsResponse added in v0.6.0 

func ValidatePolicyDecisionsResponse(response *policymanager.GetPolicyDecisionsResponse, taxonomyFile string) error

Types

type ApplicationContext added in v0.6.1 

type ApplicationContext struct {
	Log         zerolog.Logger
	Application *api.FybrikApplication
	UUID        string
}

type BlueprintReconciler 

type BlueprintReconciler struct {
	client.Client
	Name   string
	Log    zerolog.Logger
	Scheme *runtime.Scheme
	Helmer helm.Interface
}

BlueprintReconciler reconciles a Blueprint object

func NewBlueprintReconciler 

func NewBlueprintReconciler(mgr ctrl.Manager, name string, helmer helm.Interface) *BlueprintReconciler

NewBlueprintReconciler creates a new reconciler for Blueprint resources

func (*BlueprintReconciler) Reconcile 

func (r *BlueprintReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)

Reconcile receives a Blueprint CRD

func (*BlueprintReconciler) SetupWithManager 

func (r *BlueprintReconciler) SetupWithManager(mgr ctrl.Manager) error

SetupWithManager registers Blueprint controller

type ContextInterface 

type ContextInterface interface {
	ResourceExists(ref *app.ResourceReference) bool
	CreateOrUpdateResource(owner *app.ResourceReference, ref *app.ResourceReference, plotterSpec *app.PlotterSpec, labels map[string]string, uuid string) error
	DeleteResource(ref *app.ResourceReference) error
	GetResourceStatus(ref *app.ResourceReference) (app.ObservedState, error)
	CreateResourceReference(owner *app.ResourceReference) *app.ResourceReference
	GetManagedObject() runtime.Object
}

ContextInterface is an interface for communication with a generated resource (e.g. Blueprint)

type DataInfo added in v0.6.0 

type DataInfo struct {
	// Source connection details
	DataDetails *datacatalog.GetAssetResponse
	// Pointer to the relevant data context in the Fybrik application spec
	Context *app.DataContext
	// Evaluated config policies
	Configuration adminconfig.EvaluatorOutput
	// Workload cluster
	WorkloadCluster multicluster.Cluster
	// Governance actions to perform on this asset
	Actions []taxonomy.Action
}

DataInfo defines all the information about the given data set that comes from the fybrikapplication spec and from the connectors.

type Edge added in v0.6.0 

type Edge struct {
	Source          *Node
	Sink            *Node
	Module          *app.FybrikModule
	CapabilityIndex int
}

Edge represents a module capability that gets data via source and returns data via sink interface

type FybrikApplicationReconciler 

type FybrikApplicationReconciler struct {
	client.Client
	Name              string
	Log               zerolog.Logger
	Scheme            *runtime.Scheme
	PolicyManager     pmclient.PolicyManager
	DataCatalog       dcclient.DataCatalog
	ResourceInterface ContextInterface
	ClusterManager    multicluster.ClusterLister
	Provision         storage.ProvisionInterface
	ConfigEvaluator   adminconfig.EvaluatorInterface
}

FybrikApplicationReconciler reconciles a FybrikApplication object

func NewFybrikApplicationReconciler 

func NewFybrikApplicationReconciler(mgr ctrl.Manager, name string,
	policyManager pmclient.PolicyManager, catalog dcclient.DataCatalog, cm multicluster.ClusterLister,
	provision storage.ProvisionInterface, evaluator adminconfig.EvaluatorInterface) *FybrikApplicationReconciler

NewFybrikApplicationReconciler creates a new reconciler for FybrikApplications

func (*FybrikApplicationReconciler) GetAllModules 

func (r *FybrikApplicationReconciler) GetAllModules() (map[string]*api.FybrikModule, error)

GetAllModules returns all CRDs of the kind FybrikModule mapped by their name

func (*FybrikApplicationReconciler) GetWorkloadCluster added in v0.6.0 

func (r *FybrikApplicationReconciler) GetWorkloadCluster(appContext ApplicationContext) (multicluster.Cluster, error)

GetWorkloadCluster returns a workload cluster If no cluster has been specified for a workload, a local cluster is assumed.

func (*FybrikApplicationReconciler) Reconcile 

func (r *FybrikApplicationReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)

Reconcile reconciles FybrikApplication CRD It receives FybrikApplication CRD and selects the appropriate modules that will run The outcome is a Plotter containing multiple Blueprints that run on different clusters

func (*FybrikApplicationReconciler) RegisterAsset 

func (r *FybrikApplicationReconciler) RegisterAsset(catalogID string, info *app.DatasetDetails, input *app.FybrikApplication) (string, error)

RegisterAsset registers a new asset in the specified catalog Input arguments: - catalogID: the destination catalog identifier - info: connection and credential details Returns: - an error if happened - the new asset identifier

func (*FybrikApplicationReconciler) SetupWithManager 

func (r *FybrikApplicationReconciler) SetupWithManager(mgr ctrl.Manager) error

SetupWithManager registers FybrikApplication controller

func (*FybrikApplicationReconciler) ValidateAssetResponse added in v0.6.0 

func (r *FybrikApplicationReconciler) ValidateAssetResponse(response *datacatalog.GetAssetResponse, taxonomyFile string, datasetID string) error

type FybrikModuleReconciler added in v0.6.0 

type FybrikModuleReconciler struct {
	client.Client
	Name   string
	Log    zerolog.Logger
	Scheme *runtime.Scheme
}

FybrikModuleReconciler reconciles a FybrikModule object

func NewFybrikModuleReconciler added in v0.6.0 

func NewFybrikModuleReconciler(mgr ctrl.Manager, name string) *FybrikModuleReconciler

NewFybrikModuleReconciler creates a new reconciler for FybrikModules

func (*FybrikModuleReconciler) Reconcile added in v0.6.0 

func (r *FybrikModuleReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)

Reconcile validates FybrikModule CRD

func (*FybrikModuleReconciler) SetupWithManager added in v0.6.0 

func (r *FybrikModuleReconciler) SetupWithManager(mgr ctrl.Manager) error

SetupWithManager registers Module controller

type ModuleInstanceSpec added in v0.6.0 

type ModuleInstanceSpec struct {
	Chart       *app.ChartSpec
	Args        *app.ModuleArguments
	AssetIDs    []string
	ClusterName string
	ModuleName  string
	Scope       app.CapabilityScope
}

ModuleInstanceSpec consists of the module spec and arguments

type NewAssetInfo 

type NewAssetInfo struct {
	Storage *storage.ProvisionedBucket
}

NewAssetInfo points to the provisoned storage and hold information about the new asset

type Node added in v0.6.0 

type Node struct {
	Connection *app.InterfaceDetails
	Virtual    bool
}

Node represents an access point to data (as a physical source/sink, or a virtual endpoint) A virtual endpoint is activated by the workload for read/write actions.

type PlotterGenerator added in v0.6.0 

type PlotterGenerator struct {
	Client                client.Client
	Log                   zerolog.Logger
	Modules               map[string]*app.FybrikModule
	Clusters              []multicluster.Cluster
	Owner                 types.NamespacedName
	PolicyManager         pmclient.PolicyManager
	Provision             storage.ProvisionInterface
	VaultConnection       vault.Interface
	ProvisionedStorage    map[string]NewAssetInfo
	StorageAccountRegions []string
}

PlotterGenerator constructs a plotter based on the requirements (governance actions, data location) and the existing set of FybrikModules

func (*PlotterGenerator) AddFlowInfoForAsset added in v0.6.0 

func (p *PlotterGenerator) AddFlowInfoForAsset(item DataInfo, application *app.FybrikApplication, plotterSpec *app.PlotterSpec) error

Adds the asset details, flows and templates to the given plotter spec.

func (*PlotterGenerator) FindPaths added in v0.6.0 

func (p *PlotterGenerator) FindPaths(item *DataInfo, appContext *app.FybrikApplication) []Solution

FindPaths finds all valid data paths between the data source and the workload First, data paths are constructed using interface connections, starting from data source. Then, transformations are added to the found paths, and clusters are matched to satisfy restrictions from admin config policies. Optimization is done by the shortest path (the paths are sorted by the length). To be changed in future versions.

func (*PlotterGenerator) GetCopyDestination added in v0.6.0 

func (p *PlotterGenerator) GetCopyDestination(item DataInfo, destinationInterface *app.InterfaceDetails, geo string) (*app.DataStore, error)

GetCopyDestination creates a Dataset for bucket allocation by implicit copies or ingest.

type PlotterInterface 

type PlotterInterface struct {
	Client client.Client
}

PlotterInterface context implementation for communication with a single Plotter resource

func NewPlotterInterface 

func NewPlotterInterface(cl client.Client) *PlotterInterface

NewPlotterInterface creates a new plotter interface for FybrikApplication controller

func (*PlotterInterface) CreateOrUpdateResource 

func (c *PlotterInterface) CreateOrUpdateResource(owner *app.ResourceReference, ref *app.ResourceReference, plotterSpec *app.PlotterSpec, labels map[string]string, uuid string) error

CreateOrUpdateResource creates a new Plotter resource or updates an existing one

func (*PlotterInterface) CreateResourceReference 

func (c *PlotterInterface) CreateResourceReference(owner *app.ResourceReference) *app.ResourceReference

CreateResourceReference returns an identifier (name and namespace) of the generated resource.

func (*PlotterInterface) DeleteResource 

func (c *PlotterInterface) DeleteResource(ref *app.ResourceReference) error

DeleteResource deletes the generated Plotter resource

func (*PlotterInterface) GetManagedObject 

func (c *PlotterInterface) GetManagedObject() runtime.Object

GetManagedObject returns the type of the managed runtime object

func (*PlotterInterface) GetResourceSignature 

func (c *PlotterInterface) GetResourceSignature(ref *app.ResourceReference) *app.Plotter

GetResourceSignature returns the namespaced information of the generated Plotter resource

func (*PlotterInterface) GetResourceStatus 

func (c *PlotterInterface) GetResourceStatus(ref *app.ResourceReference) (app.ObservedState, error)

GetResourceStatus returns the generated Plotter status

func (*PlotterInterface) ResourceExists 

func (c *PlotterInterface) ResourceExists(ref *app.ResourceReference) bool

ResourceExists checks whether the Plotter resource generated by FybrikApplication controller is active

type PlotterModulesSpec added in v0.5.0 

type PlotterModulesSpec struct {
	ClusterName     string
	VaultAuthPath   string
	AssetID         string
	ModuleName      string
	ModuleArguments *api.StepParameters
	FlowType        api.DataFlow
	Chart           api.ChartSpec
	Scope           api.CapabilityScope
}

PlotterModulesSpec consists of module details extracted from the Plotter structure

type PlotterReconciler 

type PlotterReconciler struct {
	client.Client
	Name           string
	Log            zerolog.Logger
	Scheme         *runtime.Scheme
	ClusterManager multicluster.ClusterManager
}

PlotterReconciler reconciles a Plotter object

func NewPlotterReconciler 

func NewPlotterReconciler(mgr ctrl.Manager, name string, manager multicluster.ClusterManager) *PlotterReconciler

NewPlotterReconciler creates a new reconciler for Plotter resources

func (*PlotterReconciler) GenerateBlueprint added in v0.5.0 

func (r *PlotterReconciler) GenerateBlueprint(instances []ModuleInstanceSpec, clusterName string, plotter *app.Plotter) app.BlueprintSpec

GenerateBlueprint creates the Blueprint spec based on the datasets and the governance actions required, which dictate the modules that must run in the fybrik Credentials for accessing data set are stored in a credential management system (such as vault) and the paths for accessing them are included in the blueprint. The credentials themselves are not included in the blueprint.

func (*PlotterReconciler) GenerateBlueprints added in v0.5.0 

func (r *PlotterReconciler) GenerateBlueprints(instances []ModuleInstanceSpec, plotter *app.Plotter) map[string]app.BlueprintSpec

GenerateBlueprints creates Blueprint specs (one per cluster)

func (*PlotterReconciler) Reconcile 

func (r *PlotterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)

Reconcile receives a Plotter CRD

func (*PlotterReconciler) RefineInstances added in v0.5.0 

func (r *PlotterReconciler) RefineInstances(instances []ModuleInstanceSpec) []ModuleInstanceSpec

RefineInstances collects all instances of the same read/write module with non "Asset" scope and creates a new instance instead, with accumulated arguments.

func (*PlotterReconciler) SetupWithManager 

func (r *PlotterReconciler) SetupWithManager(mgr ctrl.Manager) error

SetupWithManager registers Plotter controller

type ResolvedEdge added in v0.6.0 

type ResolvedEdge struct {
	Edge
	Actions              []taxonomy.Action
	Cluster              string
	StorageAccountRegion string
}

ResolvedEdge extends an Edge by adding actions that a module should perform, and the cluster where the module will be deployed TODO(shlomitk1): add plugins/transformation capabilities to this structure

type Solution added in v0.6.0 

type Solution struct {
	DataPath []ResolvedEdge
}

Solution is a final solution enabling a plotter construction. It represents a full data flow between the data source and the workload.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.