Plex Sandbox Operator
Introduction
The Plex Sandbox Operator is an operator for Kubernetes that enables authenticated users to create their own isolated environments.
Installation
Kustomize
This repository contains a deploy folder which contains all of the manifests required to deploy the operator, as well as a kustomization.yaml
file.
If you would like to apply your own customizations, reference the deploy
folder and the version in your kustomization.yaml
.
Kustomize v2
(version used in kubectl apply -k .
)
bases:
- git::https://github.com/plexsystems/sandbox-operator.git//deploy?ref=v0.10.1
Kustomize v3
Latest version of Kustomize if installed as a standalone. Also version embedded in flux.
resources:
- git::https://github.com/plexsystems/sandbox-operator.git//deploy?ref=v0.10.1
The example folder shows one example of how to customize the operator.
Bundle
A bundle.yaml is provided in the root of the repository which can then be applied via kubectl apply
.
Created ClusterRole and ClusterRoleBinding
A ClusterRole
resource and a ClusterRoleBinding
resource will be created to enable authenticated users to create Sandbox resources.
ClusterRole (sandbox-user)
Verbs |
API Groups |
Resources |
create, list, get |
operators.plex.dev |
sandboxes |
ClusterRoleBinding (sandbox-users)
API Group |
Name |
Subjects |
rbac.authorization.k8s.io |
sandbox-users |
system:authenticated |
Sandbox CRD
A CustomResourceDefinition
named Sandbox
will be created.
An example manifest for the Sandbox CRD is as follows:
apiVersion: operators.plex.dev/v1alpha1
kind: Sandbox
metadata:
name: test
spec:
size: small
owners:
- foo@bar.com
Configuration
Clients
The Sandbox operator can leverage different clients, depending upon how authenitcation is configured for your cluster.
Azure
If Azure credentials are provided to the operators environment, it will perform a lookup of each user in the owners
field and fetch that users ObjectID
inside of Azure using the Microsoft Graph API.
This enables users to create Sandboxes with friendly names in the owners
field (such as the owners email address) and have the operator itself handle the mapping to the ObjectID
when creating the Kubernetes resources.
To use the Azure client, include the following environment variables:
AZURE_CLIENT_ID
AZURE_TENANT_ID
AZURE_CLIENT_SECRET
Your Azure Service Principal will need the following Application permission for the Azure Active Directory Graph API (00000002-0000-0000-c000-000000000000):
Directory.Read.All (5778995a-e1bf-45b8-affa-663a9f3f4d04)
Default
If no credentials are provided, the operator will create the Role
and ClusterRole
bindings using the values listed in the owners
field.
Docker Pull Secrets
By default, the operator will not create any secrets in the provisioned namespace.
If the PULL_SECRET_NAME
environment variable is set, the operator will copy your clusters pull secret to the provisioned namespace and patch the default service account.
PULL_SECRET_NAME
should be the name of the pull secret that exists in your cluster. By default, the operator will look for your secret in the default
namespace.
To have the operator look in a different namespace for the pull secret, use the PULL_SECRET_NAMESPACE
environment variable.
Creating a Sandbox
To create a Sandbox, apply a Sandbox CRD to the target cluster.
The following will create a Sandbox called foo
(the resulting namespace being sandbox-foo
), and assign the RBAC policies to user foo@bar.com
.
foo.yaml
apiVersion: operators.plex.dev/v1alpha1
kind: Sandbox
metadata:
name: foo
spec:
size: small
owners:
- foo@bar.com
$ kubectl apply -f foo.yaml
sandboxes.operators.plex.dev "foo" created
Created Resources
Assuming the name of the created Sandbox is named foo
, the following resources will be created per Sandbox:
Namespace (sandbox-foo)
ClusterRole (sandbox-foo-admin)
Verbs |
API Groups |
Resources |
ResourceNames |
* |
operators.plex.dev |
sandboxes |
sandbox-foo |
This is created so that only users defined in the owners
field can delete their Sandboxes.
ClusterRoleBinding (sandbox-foo-admins)
One ClusterRoleBinding
per name in the owners
field. Bindings are added and removed as users are added and removed from the owners
field.
Role (sandbox-foo-owner)
Verbs |
API Groups |
Resources |
* |
core |
pods, pods/log, pods/portforward, services, services/finalizers, endpoints, persistentvolumeclaims, events, configmaps, replicationcontrollers |
create |
core |
secrets |
* |
apps, extensions |
deployments, daemonsets, replicasets, statefulsets |
* |
autoscaling |
horizontalpodautoscalers |
* |
batch |
jobs, cronjobs |
create, list, get |
rbac.authorization.k8s.io |
roles, rolebindings |
RoleBinding (sandbox-foo-owners)
One RoleBinding
per name in the owners
field. Bindings are added and removed as users are added and removed from the owners
field.
ResourceQuota (sandbox-foo-resourcequota)
The ResourceQuota
that is applied to the Namespace
depends on the size
of the Sandbox
that was created. Defaults to small
if no size is given.
Small
Resource Name |
Quantity |
ResourceRequestsCPU |
0.25 |
ResourceLimitsCPU |
0.5 |
ResourceRequestsMemory |
250Mi |
ResourceLimitsMemory |
500Mi |
ResourceRequestsStorage |
10Gi |
ResourcePersistentVolumeClaims |
2 |
Large
Resource Name |
Quantity |
ResourceRequestsCPU |
1 |
ResourceLimitsCPU |
2 |
ResourceRequestsMemory |
2Gi |
ResourceLimitsMemory |
8Gi |
ResourceRequestsStorage |
40Gi |
ResourcePersistentVolumeClaims |
8 |
Managing Owners of a Sandbox
After the Sandbox has been created, you can add or remove owners that are associated to it.
For example, to add more@bar.com
as an owner, add their name to the list of owners and apply the changes:
apiVersion: operators.plex.dev/v1alpha1
kind: Sandbox
metadata:
name: foo
spec:
size: small
owners:
- foo@bar.com
- more@bar.com
$ kubectl apply -f sandbox-foo.yaml
sandboxes.operators.plex.dev "foo" configured
This will cause the operator to add ClusterRoleBinding
and RoleBinding
resources to match the owners list.
When owners
are removed from the Sandbox, their ClusterRoleBinding
and RoleBinding
will also be removed.
Deleting a Sandbox
To delete a Sandbox, delete the Sandbox resource from the cluster:
$ kubectl delete sandbox foo
sandboxes.operators.plex.dev "foo" deleted
Deleting a Sandbox will delete the Namespace
as well as the ClusterRole
and ClusterRoleBinding
resources.
Metrics
The operator exposes two metric ports for the /metrics
endpoint:
- Port
8383
exposes metrics for the operator itself
- Port
8686
exposes metrics for the Sandbox
CRD
Additionally, if prometheus-operator is installed into the cluster, a ServiceMonitor
is created for the operator.
Development
No external tooling is required to develop and build the operator. However, some tooling is required to run the integration tests:
Testing
The provided Makefile
contains commands that assist with running the tests for the operator.
Unit tests
make test-unit
will use an in-memory kubernetes client to validate and test your changes without the need for an external Kubernetes cluster.
Integration tests
make test-integration
will create a Kubernetes cluster for you, using Kind, and deploy the operator to it. The integration tests will then be ran against the newly created cluster.
Testing different Kubernetes versions
To test the operator with different versions of Kubernetes, you can use the KUBERNETES_VERSION
variable when calling make
.
For example, to test on Kubernetes v1.16.3, run the following command:
make test-integration KUBERNETES_VERSION=v1.16.3
Contributing
We ❤ pull requests. If you have a question, feedback, or would like to contribute — please feel free to create an issue or open a pull request!