kubectl fluence
This repository implements a single kubectl plugin for interacting with the fluence scheduler.
This is currently intended to help with debugging and development.
Usage
Deploy Fluence
You will need to deploy Fluence with the following flag for scheduler.enableExternalService=true
. See
the kind-config.yaml for how to deploy a kind cluster with ingress.
helm install \
--set scheduler.image=ghcr.io/vsoch/fluence:latest \
--set scheduler.enableExternalService=true \
--set scheduler.sidecarimage=ghcr.io/vsoch/fluence-sidecar:latest \
schedscheduler-plugins as-a-second-scheduler/
Google Cloud
Note that Google Cloud will require an extra command to expose the service. I find that creating the cluster like this works:
GOOGLE_PROJECT=myproject
gcloud container clusters create test-cluster \
--threads-per-core=1 \
--placement-type=COMPACT \
--num-nodes=6 \
--region=us-central1-a \
--project=${GOOGLE_PROJECT} \
--enable-network-policy \
--tags=test-cluster \
--enable-intra-node-visibility \
--machine-type=c2d-standard-8
Here is how to expose the port for ingress (you typically only need to do this once for an entire project):
gcloud compute firewall-rules create test-cluster --allow tcp:4242
You will need to use kubectl get pods -o wide
to see the physical node fluence is running on, and then
get the ip address for that node with kubectl get nodes -o wide
.
kubectl fluence --host 34.121.134.15
Resource Graph Summary
NODES CORES/NODE MEMORY/NODE
5 3 27648
1 3 26624
TOTAL 6 6 54272
That's the first glimpose of having one node with a slightly different type (I wonder why that is)? Possibly
the control plane has less memory available? And of course cleanup:
gcloud container clusters delete test-cluster --region=us-central1-a
Note that I haven't gotten it to work on Google Cloud - I am not sure why.
Cluster Resources
A simple kubectl fluence
will show cluster resources.
Resource Graph Summary
COUNT CORES MEMORY
1 10 30720
TOTAL 1 10 30720
Development
Note that for this setup if you are developing locally with kind, you will need to enable the ingress. Here is kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
kubeadmConfigPatches:
- |
kind: InitConfiguration
nodeRegistration:
kubeletExtraArgs:
node-labels: "ingress-ready=true"
extraPortMappings:
- containerPort: 4242
hostPort: 4242
protocol: TCP
Make Protocol Buffers
Install the needed tools to your local bin (already on the PATH for the development container):
make protoc
If you aren't in the dev container:
export PATH=$PATH:$PWD/bin
Then compile:
make proto
Build Plugin
To install the plugin from GitHub:
go get github.com/converged-computing/fluence-kubectl
Or build it locally. This assumes you have a working KUBECONFIG
make
This will generate the binary, and you'll need to copy it into somewhere on the path to access.
sudo cp ./kubectl-fluence /usr/local/bin
Shell Completion
For shell completion, copy kubectl_complete-fluence also somewhere on your
path and make it executable.
/usr/local/bin ./kubectl_complete-fluence
sudo cp ./kubectl_complete-fluence /usr/local/bin
Build and Install
To do both of the above (and warning, the install requests sudo!)
make install
Uninstall
To uninstall, just delete both.
sudo rm -rf /usr/local/bin/kubectl-fluence /usr/local/bin/kubectl_complete-fluence
TODO
- I don't like that the grpcs are now shared across modules and projects - I'd like a simple strategy (maybe submodules) that can get them from the same source instead.
- We need a way to get group name / metadata directly from fluxion.
- We can add the Info/Stats exposed endpoints to the fluxion GRPC to get better views of the cluster (stats)
Thank You
License
HPCIC DevTools is distributed under the terms of the MIT license.
All new contributions must be made under this license.
See LICENSE,
COPYRIGHT, and
NOTICE for details.
SPDX-License-Identifier: (MIT)
LLNL-CODE- 842614