README
¶
Monogon Operating System
A cluster operating system. Codename: metropolis. Linux kernel, stateless userland, API-driven management, high integrity. Designed to run Kubernetes and other workload scheduling systems.
Documentation
The canonical documentation for Monogon OS is the Monogon OS Handbook.
Developer Quick Start
Follow the setup instructions in the top-level README.md. We recommend using nix-shell either via Nix installed on an existing distribution or on NixOS.
Start a test cluster by running:
$ bazel run //metropolis:launch-cluster
This will build all the required code and run a fully userspace test cluster consisting of four qemu VMs (three for Monogon OS nodes, one for a swich/router emulator). No root access on the host required.
.--------. .--------. .--------.
| node 0 | | node 1 | | node 2 |
'--------' '--------' '--------'
^ ^ ^
| | | Virtual Ethernet (10.1.0.0/24)
V V V
.-------------------------.
| nanoswitch |
|-------------------------| .-------------------.
| Router, switch, |--->| Internet via host |
| SOCKS proxy | '-------------------'
'-------------------------'
^
| gRPC over SOCKS to nodes
.----------.
| metroctl |
'----------'
The launch tool will output information on how to connect to the cluster:
Launch: Cluster running!
To access cluster use: metroctl --config /tmp/metroctl-3733429479 --proxy 127.0.0.1:42385 --endpoints 10.1.0.2 --endpoints 10.1.0.3 --endpoints 10.1.0.4
Or use this handy wrapper: /tmp/metroctl-3733429479/metroctl.sh
To access Kubernetes, use kubectl --context=launch-cluster
You can use the metroctl wrapper to then look at the node list per the Monogon OS cluster control plane:
$ alias metroctl=/tmp/metroctl-3733429479/metroctl.sh
$ metroctl node describe
NODE ID STATE ADDRESS HEALTH ROLES TPM VERSION HEARTBEAT
metropolis-067651202d00b79fffe92df0001aabff UP 10.1.0.4 HEALTHY yes v0.1.0-dev494.g0d8a8a4f.dirty 1s
metropolis-7ccd2437c50696ea9a9b6543dc163f84 UP 10.1.0.3 HEALTHY yes v0.1.0-dev494.g0d8a8a4f.dirty 3s
metropolis-ec101152c48c5f761534c1910cf66200 UP 10.1.0.2 HEALTHY ConsensusMember,KubernetesController yes v0.1.0-dev494.g0d8a8a4f.dirty 3s
We have a node running the Monogon OS control plane (ConsensusMember role) and Kubernetes control plane ( KubernetesController role), but no Kubernetes worker nodes. But changing that is a simple API call (or metroctl invocation) away:
$ metroctl node add role KubernetesWorker metropolis-067651202d00b79fffe92df0001aabff
2024/02/12 17:42:33 Updated node metropolis-067651202d00b79fffe92df0001aabff.
$ metroctl node add role KubernetesWorker metropolis-7ccd2437c50696ea9a9b6543dc163f84
2024/02/12 17:42:36 Updated node metropolis-7ccd2437c50696ea9a9b6543dc163f84.
$ metroctl node describe
NODE ID STATE ADDRESS HEALTH ROLES TPM VERSION HEARTBEAT
metropolis-067651202d00b79fffe92df0001aabff UP 10.1.0.4 HEALTHY KubernetesWorker yes v0.1.0-dev494.g0d8a8a4f.dirty 0s
metropolis-7ccd2437c50696ea9a9b6543dc163f84 UP 10.1.0.3 HEALTHY KubernetesWorker yes v0.1.0-dev494.g0d8a8a4f.dirty 3s
metropolis-ec101152c48c5f761534c1910cf66200 UP 10.1.0.2 HEALTHY ConsensusMember,KubernetesController yes v0.1.0-dev494.g0d8a8a4f.dirty 3s
And just like that, we can now see these nodes in Kubernetes, too:
$ kubectl --context=launch-cluster get nodes
NAME STATUS ROLES AGE VERSION
metropolis-067651202d00b79fffe92df0001aabff Ready <none> 15s v1.24.2+mngn
metropolis-7ccd2437c50696ea9a9b6543dc163f84 Ready <none> 13s v1.24.2+mngn
$ kubectl --context=launch-cluster run -it --image=ubuntu:22.04 test -- bash
If you don't see a command prompt, try pressing enter.
root@test:/# uname -a
Linux test 6.1.69-metropolis #1 SMP PREEMPT_DYNAMIC Tue Jan 30 14:43:23 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
root@test:/#
With the test launch tooling, you can now start iterating on the codebase. Regardless of whether you're changing the
Linux kernel config or implementing a new RPC, testing your changes interactively is a single bazel command away.
End-to-end tests
We have an end-to-end test suite. It's run automatically on CI. Any new logic should be exercised there.
$ bazel run //metropolis/test/e2e:e2e_test
These tests operate on a fully virtualized cluster just like the launch tooling, so be patient.
Directories
¶
| Path | Synopsis |
|---|---|
|
cli
|
|
|
flagdefs
Package flagdefs contains shared flag definitions for Metropolis.
|
Package flagdefs contains shared flag definitions for Metropolis. |
|
metroctl/core
Package core contains parts of metroctl which do significant amounts of work beyond just providing a CLI for them.
|
Package core contains parts of metroctl which do significant amounts of work beyond just providing a CLI for them. |
|
Installer creates a Metropolis image at a suitable block device based on the installer bundle present in the installation medium's ESP, after which it reboots.
|
Installer creates a Metropolis image at a suitable block device based on the installer bundle present in the installation medium's ESP, after which it reboots. |
|
test/testos
TestOS is a tiny "operating system" which is packaged the exact same way as an actual Metropolis node but only outputs a single flag before exiting.
|
TestOS is a tiny "operating system" which is packaged the exact same way as an actual Metropolis node but only outputs a single flag before exiting. |
|
core
This uses the unstable overrideWrite interface to also emit all runtime writes to a dedicated runtime file descriptor to catch and debug crash dumps.
|
This uses the unstable overrideWrite interface to also emit all runtime writes to a dedicated runtime file descriptor to catch and debug crash dumps. |
|
core/cluster
Package cluster implements low-level clustering logic, especially logic regarding to bootstrapping, registering into and joining a cluster.
|
Package cluster implements low-level clustering logic, especially logic regarding to bootstrapping, registering into and joining a cluster. |
|
core/clusternet
Package clusternet implements a Cluster Networking mesh service running on all Metropolis nodes.
|
Package clusternet implements a Cluster Networking mesh service running on all Metropolis nodes. |
|
core/consensus/client
package client implements a higher-level client for consensus/etcd that is to be used within the Metropolis node code for unprivileged access (ie.
|
package client implements a higher-level client for consensus/etcd that is to be used within the Metropolis node code for unprivileged access (ie. |
|
core/curator
package curator implements the Curator, a service responsible for management of the Metropolis cluster that it is running on.
|
package curator implements the Curator, a service responsible for management of the Metropolis cluster that it is running on. |
|
core/devmgr
Package devmgr is the userspace pendant to the kernel device management system.
|
Package devmgr is the userspace pendant to the kernel device management system. |
|
core/metrics/fake_exporter
fake_exporter is a tiny Prometheus-compatible metrics exporter which exports a single metric with a value configured at startup.
|
fake_exporter is a tiny Prometheus-compatible metrics exporter which exports a single metric with a value configured at startup. |
|
core/mgmt
Package mgmt implements the node-local management service, a.k.a.
|
Package mgmt implements the node-local management service, a.k.a. |
|
core/network/dhcp4c
Package dhcp4c implements a DHCPv4 Client as specified in RFC2131 (with some notable deviations).
|
Package dhcp4c implements a DHCPv4 Client as specified in RFC2131 (with some notable deviations). |
|
core/network/dhcp4c/callback
Package callback contains minimal callbacks for configuring the kernel with options received over DHCP.
|
Package callback contains minimal callbacks for configuring the kernel with options received over DHCP. |
|
core/network/dhcp4c/transport
Package transport contains Linux-based transports for the DHCP broadcast and unicast specifications.
|
Package transport contains Linux-based transports for the DHCP broadcast and unicast specifications. |
|
core/network/hostsfile
Package hostsfile implements a service which owns and writes all node-local files/interfaces used by the system to resolve the local node's name and the names of other nodes in the cluster:
|
Package hostsfile implements a service which owns and writes all node-local files/interfaces used by the system to resolve the local node's name and the names of other nodes in the cluster: |
|
core/roleserve
Package roleserve implements the roleserver/“Role Server”.
|
Package roleserve implements the roleserver/“Role Server”. |
|
core/time
Package time implements a supervisor runnable which is responsible for keeping both the system clock and the RTC accurate.
|
Package time implements a supervisor runnable which is responsible for keeping both the system clock and the RTC accurate. |
|
kubernetes/authproxy
Package authproxy implements an authenticating proxy in front of the K8s API server converting Metropolis credentials into authentication headers.
|
Package authproxy implements an authenticating proxy in front of the K8s API server converting Metropolis credentials into authentication headers. |
|
kubernetes/clusternet
Package clusternet implements a WireGuard-based overlay network for Kubernetes.
|
Package clusternet implements a WireGuard-based overlay network for Kubernetes. |
|
kubernetes/hyperkube
This is the entry point for our multicall Kubernetes binary.
|
This is the entry point for our multicall Kubernetes binary. |
|
kubernetes/metricsproxy
Package metricsproxy implements an authenticating proxy in front of the K8s controller-manager and scheduler providing unauthenticated access to the metrics via local ports
|
Package metricsproxy implements an authenticating proxy in front of the K8s controller-manager and scheduler providing unauthenticated access to the metrics via local ports |
|
kubernetes/nfproxy
Package nfproxy is a Kubernetes Service IP proxy based exclusively on the Linux nftables interface.
|
Package nfproxy is a Kubernetes Service IP proxy based exclusively on the Linux nftables interface. |
|
kubernetes/pki
package pki builds upon osbase/pki/ to provide an etcd-backed implementation of all x509 PKI Certificates/CAs required to run Kubernetes.
|
package pki builds upon osbase/pki/ to provide an etcd-backed implementation of all x509 PKI Certificates/CAs required to run Kubernetes. |
|
kubernetes/plugins/kvmdevice
Package kvmdevice implements a Kubernetes device plugin for the virtual KVM device.
|
Package kvmdevice implements a Kubernetes device plugin for the virtual KVM device. |
|
kubernetes/reconciler
The reconciler ensures that a base set of K8s resources is always available in the cluster.
|
The reconciler ensures that a base set of K8s resources is always available in the cluster. |
|
proto
|
|
|
test
|
|
|
e2e/httpserver
httpserver serves a test HTTP endpoint for E2E testing.
|
httpserver serves a test HTTP endpoint for E2E testing. |
|
e2e/k8s_cts
This package launches a Metropolis cluster with two nodes and spawns in the CTS container.
|
This package launches a Metropolis cluster with two nodes and spawns in the CTS container. |
|
launch
cluster builds on the launch package and implements launching Metropolis nodes and clusters in a virtualized environment using qemu.
|
cluster builds on the launch package and implements launching Metropolis nodes and clusters in a virtualized environment using qemu. |
|
localregistry
Package localregistry implements a read-only OCI Distribution / Docker V2 container image registry backed by local layers.
|
Package localregistry implements a read-only OCI Distribution / Docker V2 container image registry backed by local layers. |
|
nanoswitch
nanoswitch is a virtualized switch/router combo intended for testing.
|
nanoswitch is a virtualized switch/router combo intended for testing. |
|
util
This file implements test helper functions that augment the way any given test is run.
|
This file implements test helper functions that augment the way any given test is run. |
|
vm
|
|
|
smoketest
This is a small smoke test which will run in a container on top of Metropolis Kubernetes.
|
This is a small smoke test which will run in a container on top of Metropolis Kubernetes. |
Click to show internal directories.
Click to hide internal directories.