rcs-ocm-deployer
rcs-ocm-deployer
is an operator designed to deploy Capp
(ContainerApp) workloads created on the Hub Cluster on the most suitable Managed Cluster using OCM
(Open Cluster Management) APIs of Placement
and ManifestWork
. It also includes a OCM AddOn
to sync the status of Capp
between the Managed Clusters and the Hub Cluster.
RCS
RCS
(Run Container Service) is an open-source implementation of Container as a Service solution.
It utilizes the OCM
project to create and manage workloads across multiple clusters, providing cloud-like features and technologies to users running on-premise.
It offers an auto cluster scheduler based on cluster usage and availability, requires basic configuration, and provides an auto-scaler template based on a single metric, among other features.
RCS
aims to simplify and streamline the management of containerized applications, making it easier for developers to focus on writing code.
Capp
This operator works together with the container-app-operator
. For more information about Capp
and about its API, please check the Capp repo.
Open Cluster Management
This project uses the Placement
and ManifestWork
APIs of the Open Cluster Management (OCM) project. For more information please refer to the OCM documentation.
High-level architecture
-
The client creates the Capp
CR in a Namespace on the Hub Cluster.
-
The rcs-ocm-deployer
controller on Hub Cluster watches for Capp
CRs and reconciles them.
-
The controller chooses the Managed Cluster with the highest score to deploy the Capp
workload using the Placement
API. The score is computed by the rcs-score
addon which creates an AddOnPlacementScore
.
-
Each Managed Cluster on the Hub Cluster has a dedicated namespace. The controller creates a ManifestWork
object on the Hub Cluster in the namespace of the chosen Managed Cluster.
-
The Managed Cluster pulls the ManifestWork
from the Hub Cluster using the work agent
and creates a Capp
CR on the Managed Cluster.
-
A Capp
controller runs on Managed Cluster, watches for Capp
CRs and reconciles them.
-
A status is returned from the Managed Cluster to the Hub Cluster using an OCM AddOn
.
The controllers
-
placement
: The controller adds an annotation containing the chosen Managed Cluster to deploy the Capp
workload on, in accordance to the placementDecision
and the desired Site
.
-
sync
: The controller controls the lifecycle of the ManifestWork
CR in the namespace of the chosen Managed Cluster. The ManifestWork
contains the Capp
CR as well as all the Secrets
and Volumes
referenced in the Capp
CR, thus making sure that all the Secrets
and Volumes
also exist on the Managed Cluster, in the same namespace the Capp CR
exists in on the Hub Cluster.
-
addOns
:
status addon
: By applying this add-on to the Hub Cluster, the Capp
status will automatically be synced back from Managed/Spoke Clusters to the Hub cluster. It has two components:
agent
- deployed on the Managed/Spoke clusters; responsible for syncing the Capp
status between the Managed Cluster and the Hub Cluster.
manager
- deployed on the Hub Cluster; responsible for deploying the agent
on the Managed/Spoke clusters.
score addon
: Based on the resource-usage-collect-addon, this add-on implements an AddonPlacementScore
which gives a score to each Managed Cluster. The score is in the range of [-100, 100] and is computed from collecting resource usage information (CPU Request and Memory Request).
Getting Started
Prerequisites
The following should be installed on your Linux machine:
Automatic Approach
Simply run the following to get a Hub cluster ready to have rcs-ocm-deployer
deployed on it, and 2 Managed Cluster with container-app-operator
already installed on them, with all add-ons installed as well.
$ make local-quickstart IMG=ghcr.io/dana-team/rcs-ocm-deployer:<release>
Manual Approach
Setting Up a Hub Cluster and Managed Clusters locally
The OCM project contains a script that allows spinning up a KinD
-based environment, containing of a Hub Cluster and 2 Managed Clusters.
The script
You can clone the repository and set up the environment. This script would create 3 clusters with the following Kubernetes contexts:
kind-hub
kind-cluster1
kind-cluster2
To switch between contexts, use:
$ kubectl config use-context <context-name>
Run the script:
$ git clone https://github.com/open-cluster-management-io/OCM
$ bash ./OCM/solutions/setup-dev-environment/local-up.sh
Create ManagedClusterSet
ManagedClusterSet
is an OCM cluster-scoped API in the Hub Cluster for grouping a few managed clusters into a "set".
To create a ManagedClusterSet
, run the following command on the Hub Cluster:
$ clusteradm create clusterset <clusterSet-name>
To add a Managed Cluster to the ManagedClusterSet
, run the following command on the Hub Cluster:
$ clusteradm clusterset set <clusterSet-name> --clusters <cluster-name>
To bind the cluster set to a namespace, run the following command on the Hub Cluster:
$ clusteradm clusterset bind <clusterSet-name> --namespace <clusterSet-namespace>
Create Placement
For the controller to work, it is needed to create a Placement
CR. This Placement
selects all clusters and is used to install the addons
on all the Managed Clusters:
apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
name: all-clusters
namespace: <clusterSet-namespace>
spec:
clusterSets:
- <clusterSet-name>
You can also create a Placement such that the clusters are selected using the customized scores. This uses the AddonPlacementScore
which is deployed later:
apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
name: <placement-name>
namespace: <clusterSet-namespace>
spec:
numberOfClusters: 1
clusterSets:
- <clusterSet-name>
prioritizerPolicy:
mode: Exact
configurations:
- scoreCoordinate:
type: AddOn
addOn:
resourceName: rcs-score
scoreName: cpuAvailable
weight: 1
Install the Capp CRD
The Capp
CRD needs to be installed on the Hub Cluster:
$ git clone https://github.com/dana-team/container-app-operator
$ cd container-app-operator
$ make install
Install cert-manager
To use rcs-ocm-deployer
, you need to have cert-manager
installed on your cluster. Follow the instruction here.
Deploying the controller
$ make deploy IMG=ghcr.io/dana-team/rcs-ocm-deployer:<release>
Configuration Using RCSConfig CRD
The rcs-ocm-deployer
operator utilizes the RCSConfig
CRD to manage its configuration and deployment options.
An instance of the RCSConfig
CRD named rcs-config
should exist in the rcs-deployer-system
namespace. This CRD instance contains the necessary configuration for the operator.
apiVersion: rcs.dana.io/v1alpha1
kind: RCSConfig
metadata:
name: rcs-config
namespace: rcs-deployer-system
spec:
placements:
- placement-1st
- placement-2nd
# Add more placement names as needed
placementsNamespace: your-placements-namespace
Ensure that the spec section includes a list of placements
and specifies the placementsNamespace
as required for your setup.
Deploy the add-ons
Status add-on
Deploy the add-on the Hub
cluster:
$ make deploy-status-addon IMG=ghcr.io/dana-team/rcs-ocm-deployer:<release>
$ kubectl -n open-cluster-management get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
capp-status-addon 1/1 1 1 14s
Patch the ClusterManagementAddon
CR of the status addon, called capp-status
to add the created Placement
to the add-on, so that it's deployed on all clusters.
$ kubectl patch clustermanagementaddon capp-status-addon --type merge -p '{"spec":{"installStrategy":{"type":"Placements","placements":[{"name":"all-clusters","namespace":"<clusterSet-namespace>"}]}}}'
The controller will automatically install the add-on agent
on all Managed/Spoke Clusters. Validate the add-on agent is installed on a Managed/Spoke` cluster:
$ kubectl -n open-cluster-management-agent-addon get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
capp-status-addon-agent 1/1 1 1 2m24s
You can also validate and check the status of the add-on on the Hub cluster:
$ kubectl -n <managed-cluster> get managedclusteraddon
NAME AVAILABLE DEGRADED PROGRESSING
capp-status-addon True
Score add-on
Deploy the add-on the Hub
cluster:
$ make deploy-score-addon IMG=ghcr.io/dana-team/rcs-ocm-deployer:<release>
Deploy the AddOnDeploymentConfig
which uses customizedVariables
to pass the max
value and min
value of the customized scores as environment variables to the add-on deployment on the Managed Clusters.
apiVersion: addon.open-cluster-management.io/v1alpha1
kind: AddOnDeploymentConfig
metadata:
name: rcs-score-deploy-config
namespace: open-cluster-management-hub
spec:
agentInstallNamespace: open-cluster-management-agent-addon
customizedVariables:
- name: MAX_CPU_COUNT
value: "<MAX_CPU_COUNT>"
- name: MIN_CPU_COUNT
value: "<MIN_CPU_COUNT>>"
- name: MAX_MEMORY_BYTES
value: "<MAX_MEMORY_BYTES>"
- name: MIN_MEMORY_BYTES
value: "<MIN_MEMEORY_BYTES>"
Patch the ClusterManagementAddon
CR of the score, called rcs-score
to add the created Placement
and AddonDeploymentConfigs
to the add-on, so that it's deployed on all clusters in the Placement
with the default configuration.
kubectl patch clustermanagementaddon rcs-score --type merge -p \
'{"spec":{"installStrategy":{"type":"Placements","placements":[{"name":"<placement-name>","namespace":"<clusterSet-namespace>","configs":[{"group":"addon.open-cluster-management.io","resource":"addondeploymentconfigs","name":"rcs-score-deploy-config","namespace":"open-cluster-management-hub"}]}]}}}'
Build your own image
$ make docker-build docker-push IMG=<registry>/rcs-ocm-deployer:<tag>
Example Capp
apiVersion: rcs.dana.io/v1alpha1
kind: Capp
metadata:
name: capp-sample
namespace: capp-sample
spec:
configurationSpec:
template:
spec:
containers:
- env:
- name: APP_NAME
value: capp-env-var
image: 'quay.io/danateamorg/example-python-app:v1-flask'
name: capp-sample
volumeMounts:
- name: test-nfspvc
mountPath: /data
routeSpec:
hostname: capp.dev
tlsEnabled: true
tlsSecret: cappTlsSecretName
volumesSpec:
nfsVolumes:
- server: test
path: /test
name: test-nfspvc
capacity:
storage: 200Gi
logSpec:
type: elastic
host: 10.11.12.13
index: main
user: elastic
passwordSecret: es-elastic-user
scaleMetric: concurrency
state: enabled