Ever wondered how to deploy your WebRTC infrastructure into the
cloud? Frightened away by the
complexities of Kubernetes container networking, and the surprising ways in which it may interact
with your UDP/RTP media? Tried to read through the endless stream of Stack
Overflow
questions
asking
how
to
scale
WebRTC services with Kubernetes, just to get (mostly) insufficient answers? Want to safely connect
your users behind a NAT, without relying on expensive third-party TURN
services?
Worry no more! STUNner allows you to deploy any WebRTC service into Kubernetes, smoothly
integrating it into the cloud-native ecosystem. STUNner exposes a
standards-compliant STUN/TURN gateway for clients to access your virtualized WebRTC infrastructure
running in Kubernetes, maintaining full browser compatibility and requiring minimal or no
modification to your existing WebRTC codebase. STUNner implements the standard Kubernetes Gateway
API so you can configure it in the familiar YAML-engineering
style via Kubernetes manifests.
See the full documentation here.
Table of Contents
- Description
- Features
- Getting started
- Tutorials
- Caveats
- Milestones
Description
Currently WebRTC
lacks
a
virtualization
story: there
is no easy way to deploy a WebRTC media service into Kubernetes to benefit from the
resiliency,
scalability,
and high
availability
features we have come to expect from modern network services. Worse yet, the entire industry relies
on a handful of public STUN
servers and hosted TURN
services to connect clients behind a NAT/firewall,
which may create a useless dependency on externally operated services, introduce a bottleneck,
raise security concerns, and come with a non-trivial price tag.
The main goal of STUNner is to allow anyone to deploy their own WebRTC infrastructure into
Kubernetes, without relying on any external service other than the cloud-provider's standard hosted
Kubernetes offering. This is achieved by STUNner acting as a gateway for ingesting WebRTC media
traffic into the Kubernetes cluster, exposing a public-facing STUN/TURN server that WebRTC clients
can connect to.
STUNner can act as a STUN/TURN server that WebRTC clients can use as a scalable NAT traversal
facility (headless model), or it can serve as a fully-fledged ingress gateway for clients to reach
a media server deployed behind STUNner (media-plane model). This makes it possible to deploy WebRTC
application servers and media servers into ordinary Kubernetes pods, taking advantage of
Kubernetes's excellent tooling to manage, scale, monitor and troubleshoot the WebRTC infrastructure
like any other cloud-bound workload.
Don't worry about the performance implications of processing all your media through a TURN server:
STUNner is written in Go so it is extremely fast, it is co-located with your
media server pool so you don't pay the round-trip time to a far-away public STUN/TURN server, and
STUNner can be easily scaled up if needed, just like any other "normal" Kubernetes service.
Features
Kubernetes has been designed and optimized for the typical HTTP/TCP Web workload, which makes
streaming workloads, and especially UDP/RTP based WebRTC media, feel like a foreign citizen.
STUNner aims to change this state-of-the-art, by exposing a single public STUN/TURN server port for
ingesting all media traffic into a Kubernetes cluster in a controlled and standards-compliant
way.
-
Seamless integration with Kubernetes. STUNner can be deployed into any Kubernetes cluster,
even into restricted ones like GKE Autopilot, using a single command. Manage your HTTP/HTTPS
application servers with your favorite service mesh, and STUNner takes care
of all UDP/RTP media. STUNner implements the Kubernetes Gateway
API so you configure it in exactly the same way as
the
rest
of
your
workload through easy-to-use YAML
manifests.
-
Expose a WebRTC media server on a single external UDP port. Get rid of the Kubernetes
hacks, like privileged pods and
hostNetwork
/hostPort
services, typically recommended as a prerequisite to containerizing your
WebRTC media plane. Using STUNner a WebRTC deployment needs only two public-facing ports, one
HTTPS port for the application server and a single UDP port for all your media.
-
No reliance on external services for NAT traversal. Can't afford a decent hosted TURN
service for client-side NAT traversal? Can't get a decent
audio/video quality because the third-party TURN service poses a bottleneck? STUNner can be
deployed into the same cluster as the rest of your WebRTC infrastructure, and any WebRTC client
can connect to it directly without the use of any external STUN/TURN service whatsoever, apart
from STUNner itself.
-
Easily scale your WebRTC infrastructure. Tired of manually provisioning your WebRTC media
servers? STUNner lets you deploy the entire WebRTC infrastructure into ordinary Kubernetes pods,
thus scaling the media plane is as easy as issuing a kubectl scale
command. STUNner itself can
be scaled with similar ease, completely separately from the media servers.
-
Secure perimeter defense. No need to open thousands of UDP/TCP ports on your media server for
potentially malicious access; with STUNner all media is received through a single ingress port
that you can tightly monitor and control.
- Simple code and extremely small size. Written in pure Go using the battle-tested
pion/webrtc framework, STUNner is just a couple of hundred
lines of fully open-source code. The server is extremely lightweight: the typical STUNner
container image size is only about 5 Mbytes.
Getting Started
STUNner comes with a Helm chart to fire up a fully functional STUNner-based
WebRTC media gateway in minutes. Note that the default installation does not contain an application
server and a media server: STUNner in itself is not a WebRTC service, it is merely an enabler for
you to deploy your own WebRTC infrastructure into Kubernetes. Once installed, STUNner makes sure
that your media servers are readily reachable to WebRTC clients, despite running with a private IP
address inside a Kubernetes pod. See the tutorials for some ideas on how to deploy an
actual WebRTC application behind STUNner.
With a minimal understanding of WebRTC and Kubernetes, deploying STUNner should take less than 5
minutes.
Installation
The simplest way to deploy STUNner is through Helm. STUNner configuration
parameters are available for customization as Helm
Values. We recommend deploying STUNner
into a separate namespace and we usually name this namespace as stunner
, so as to isolate it from
the rest of the workload.
helm repo add stunner https://l7mp.io/stunner
helm repo update
helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=<your-namespace>
helm install stunner stunner/stunner --create-namespace --namespace=<your-namespace>
Find out more about the charts in the STUNner-helm repository.
Configuration
The standard way to interact with STUNner is via Kubernetes via the standard Gateway
API version
v1alpha2. This is akin to the way you
configure all Kubernetes workloads: specify your intents in YAML files and issue a kubectl apply
, and the STUNner gateway operator will
automatically reconcile the STUNner dataplane for the new configuration.
-
Given a fresh STUNner install, the first step is to register STUNner with the Kubernetes Gateway
API. This amounts to creating a
GatewayClass,
which serves as the root level configuration for your STUNner
deployment.
Each GatewayClass must specify a controller that will manage the Gateway objects created under
the class hierarchy. In our case this must be set to stunner.l7mp.io/gateway-operator
for
STUNner to pick up the GatewayClass. In addition, a GatewayClass can refer to further
implementation-specific configuration via a parametersRef
; in our case, this will be a
GatewayConfig object to be specified next.
kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: GatewayClass
metadata:
name: stunner-gatewayclass
spec:
controllerName: "stunner.l7mp.io/gateway-operator"
parametersRef:
group: "stunner.l7mp.io"
kind: GatewayConfig
name: stunner-gatewayconfig
namespace: stunner
description: "STUNner is a WebRTC media gateway for Kubernetes"
EOF
-
The next step is to set some general configuration for STUNner,
most importantly the STUN/TURN authentication credentials. This requires loading
a GatewayConfig custom resource into Kubernetes.
Below we set the plaintext
authentication mechanism for STUNner, using the
username/password pair user-1/pass-1
, and the authentication realm stunner.l7mp.io
. See the
package docs for further
configuration options available via GatewayConfigs.
kubectl apply -f - <<EOF
apiVersion: stunner.l7mp.io/v1alpha1
kind: GatewayConfig
metadata:
name: stunner-gatewayconfig
namespace: stunner
spec:
realm: stunner.l7mp.io
authType: plaintext
userName: "user-1"
password: "pass-1"
EOF
Note that these two steps are required only once per STUNner installation.
-
At this point, we are ready to expose STUNner to clients! This occurs
by loading a
Gateway
resource into Kubernetes.
In the below example, we open a STUN/TURN listener service on the UDP listener port 3478.
STUNner will automatically expose this listener on a public IP address and port (by creating a
LoadBalancer
service for each
Gateway), await clients to connect to this listener and, once authenticated, forward client
connections to an arbitrary service backend inside the cluster. Note that we set the
gatewayClassName
to the name of the above GatewayClass; this is the way STUNner will know
which class hierarchy the Gateway belongs to so that it can set up the corresponding STUN/TURN
credentials for the new listener.
kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
name: udp-gateway
namespace: stunner
spec:
gatewayClassName: stunner-gatewayclass
listeners:
- name: udp-listener
port: 3478
protocol: UDP
EOF
-
The final step is to tell STUNner what to do with the client connections received on the
Gateway. This occurs by attaching a
UDPRoute
resource to the Gateway and specifying the target service in the backendRef
. A UDPRoute can be
attached to any Gateway by setting the parentRef
to the Gateway's name, there is just one
rule: the Gateway and the UDPRoute must both live in the same Kubernetes namespace.
The below UDPRoute will configure STUNner to route client
connections received on the Gateway called udp-gateway
to the
WebRTC media server pool identified by the Kubernetes service media-plane
in the default
namespace.
kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: UDPRoute
metadata:
name: media-plane
namespace: stunner
spec:
parentRefs:
- name: udp-gateway
rules:
- backendRefs:
- name: media-plane
namespace: default
EOF
And that's all: once configured, STUNner will make all this happen automatically, and you don't
need to worry about client-side NAT traversal and request routing because STUNner has you covered!
Even better, every time you change a Gateway API resource in Kubernetes, say, you update the
GatewayConfig to reset your STUN/TURN credentials or change the protocol or port in one of your
Gateways, the STUNner gateway operator will
automatically pick up your modifications and update the underlying dataplane in a matter of
milliseconds. Kubernetes is beautiful, isn't it?
Check your config
The current STUNner dataplane configuration is always made available in a convenient ConfigMap
called stunnerd-config
(you can choose the name in the GatewayConfig). The STUNner dataplane pods
themselves will use the very same ConfigMap to reconcile their internal state, so you can consider
its content to be the ground truth.
STUNner comes with a small utility to dump the running configuration in human readable format (you
must have jq
installed in your PATH to be able to use it). Chdir
into the main STUNner directory and issue.
cmd/stunnerctl/stunnerctl running-config stunner/stunnerd-config
STUN/TURN authentication type: plaintext
STUN/TURN username: user-1
STUN/TURN password: pass-1
Listener: udp-listener
Protocol: UDP
Public address: 34.118.36.108
Public port: 3478
As it turns out, STUNner has successfully assigned a public IP and port to our Gateway and set the
STUN/TURN credentials based on the GatewayConfig. You can use the below to dump the entire running
configuration; jq
is there just to pretty-print JSON.
kubectl get cm -n stunner stunnerd-config -o jsonpath="{.data.stunnerd\.conf}" | jq .
Testing
We have successfully configured STUNner to route client connections to the media-plane
service
but at the moment there is no backend that would respond. Below we will use a simplistic UDP
greeter service for testing: every time you send some input, the greeter service will respond with
a heartwarming welcome message.
-
Fire up the UDP greeter service.
The below manifest spawns the service in the default
namespace and wraps it in a Kubernetes
service called media-plane
(recall, this is the target service STUNner will route connections
to!). Note that the type of the media-plane
service is ClusterIP
, which means that
Kubernetes will not expose the service to the Internet: the only way for clients to obtain a
response is via STUNner.
kubectl apply -f deploy/manifests/udp-greeter.yaml
-
We also need the ClusterIP assigned by Kubernetes to the media-plane
service.
export PEER_IP=$(kubectl get svc media-plane -o jsonpath='{.spec.clusterIP}')
-
We also need a STUN/TURN client to actually initiate a connection. STUNner comes with a handy
STUN/TURN client called turncat
that we can use for this
purpose. Once built, we can fire up turncat
to listen
on the standard input and send everything it receives to STUNner. Type any input and press
Enter, and you should see a nice greeting from your cluster!
./turncat - k8s://stunner/stunnerd-config:udp-listener udp://${PEER_IP}:9001
Hello STUNner
Greetings from STUNner!
Observe that we haven't specified the STUNner public IP address and port for turncat
: it is
clever enough to read the running configuration from Kubernetes
directly. Just specify the special STUNner URI k8s://stunner/stunnerd-config:udp-listener
,
identifying the namespace and the name for the STUNner ConfigMap and the name of the listener to
connect to, and turncat
will do the heavy lifting.
Note that your actual WebRTC clients will not need to use turncat
to reach the cluster: all
modern Web browsers and WebRTC clients come with a STUN/TURN client included. Here, turncat
is
used only to simulate what a real WebRTC client would do when trying to reach STUNner.
Reconcile
Any time you see fit, you can update the STUNner configuration through the Gateway API: STUNner
will automatically reconcile the underlying dataplane for the new configuration.
For instance, you may decide to open up your WebRTC infrastructure on TCP as well; say, because an
enterprise NAT on the client network path has gone berserk and started to actively filter UDP/TURN
traffic. The below steps will do just that: open another gateway on STUNner, this time on the TCP
port 3478, and reattach the UDPRoute to both Gateways so that no matter which protocol a client may
choose the connection will be routed to the media-plane
service (i.e., the UDP greeter) by
STUNner.
-
Add the new TCP Gateway.
kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
name: tcp-gateway
namespace: stunner
spec:
gatewayClassName: stunner-gatewayclass
listeners:
- name: tcp-listener
port: 3478
protocol: TCP
EOF
-
Update the UDPRoute so that it attaches to both Gateways.
kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: UDPRoute
metadata:
name: media-plane
namespace: stunner
spec:
parentRefs:
- name: udp-gateway
- name: tcp-gateway
rules:
- backendRefs:
- name: media-plane
namespace: default
EOF
-
Fire up turncat
again, but this time let it connect through TCP. This is achieved
by specifying the name of the TCP listener (tcp-listener
) in the STUNner URI.
./turncat -l all:INFO - k8s://stunner/stunnerd-config:tcp-listener udp://${PEER_IP}:9001
[...] turncat INFO: Turncat client listening on -, TURN server: TCP://34.118.18.210:3478, peer: udp://10.120.0.127:9001
[...]
Hello STUNner
Greetings from STUNner!
We have set the turncat
loglevel to INFO to learn that this time turncat
has connected via
the TURN server TCP://34.118.18.210:3478
. And that's it: STUNner automatically routes the
incoming TCP connection to the UDP greeter service, silently converting from TCP to UDP in the
background and back again on return.
Configuring WebRTC clients
Real WebRTC clients will need a valid ICE server configuration to use STUNner as the TURN
server. STUNner is compatible with all client-side TURN auto-discovery
mechanisms. When no auto-discovery mechanism is
available, clients will need to be manually configured to stream audio/video media over STUNner.
The below JavaScript snippet will direct a WebRTC client to use STUNner as the TURN server. Make
sure to substitute the placeholders (like <STUNNER_PUBLIC_ADDR>
) with the correct configuration
from the running STUNner config; don't forget that stunnerctl
is always there for you to help.
var ICE_config = {
iceServers: [
{
url: 'turn:<STUNNER_PUBLIC_ADDR>:<STUNNER_PUBLIC_PORT>?transport=udp',
username: <STUNNER_USERNAME>,
credential: <STUNNER_PASSWORD>,
},
],
};
var pc = new RTCPeerConnection(ICE_config);
Note that STUNner comes with a small Node.js
library that simplifies generating ICE
configurations and STUNner credentials in the application server.
Tutorials
STUNner comes with a series of tutorials to demonstrate its use to deploy different WebRTC
applications into Kubernetes.
Basics
- Opening a UDP tunnel via STUNner: This introductory tutorial
shows how to tunnel an external connection via STUNner to a UDP service deployed into
Kubernetes. The demo can be used to quickly check and benchmark a STUNner installation.
Headless deployment mode
- Direct one to one video call via STUNner: This
tutorial showcases STUNner acting as a TURN server for two WebRTC clients to establish
connections between themselves, without the mediation of a media server.
- One to one video call with Kurento: This tutorial
shows how to use STUNner to connect WebRTC clients to a media server deployed into Kubernetes
behind STUNner in the media-plane deployment model. All this happens
without modifying the media server code in any way, just by adding 5-10 lines of
straightforward JavaScript to configure clients to use STUNner as the TURN server.
- Magic mirror with Kurento: This tutorial has been
adopted from the Kurento magic
mirror
demo, deploying a basic WebRTC loopback server behind STUNner with some media processing
added. In particular, the application uses computer vision and augmented reality techniques to
add a funny hat on top of faces.
- Video-conferencing with LiveKit: This tutorial helps you deploy
the LiveKit WebRTC media server behind STUNner. The docs also show how to
obtain a valid TLS certificate to secure your signaling connections, courtesy of the
cert-manager project, nip.io and Let's
Encrypt.
- Video-conferencing with Jitsi: This tutorial helps you deploy a
fully fledged Jitsi video-conferencing service into Kubernetes behind
STUNner. The docs also show how to obtain a valid TLS certificate to secure your signaling
connections, using cert-manager, nip.io and Let's
Encrypt.
- Cloud-gaming with Cloudretro: This tutorial lets you play Super
Mario or Street Fighter in your browser, courtesy of the amazing
CloudRetro project and, of course, STUNner. The demo also presents a
simple multi-cluster setup, where clients can reach the game-servers in their geographical
locality to minimize latency.
- Remote desktop access with Neko: This demo showcases STUNner
providing an ingress gateway service to a remote desktop application. We use
neko.io to run a browser in a secure container inside the Kubernetes
cluster, and stream the desktop to clients via STUNner.
Documentation
See the full documentation here.
Caveats
STUNner is a work-in-progress. Some features are missing, others may not work as expected. The
notable limitations at this point are as follows.
- STUNner targets only a partial implementation of the Kubernetes Gateway API. In particular,
only GatewayClass, Gateway and UDPRoute resources are supported. This is intended: STUNner
deliberately ignores some unnecessary complexity in the Gateway
API and deviates from the prescribed behavior in some cases,
all in the name of simplifying the configuration process. The STUNner Kubernetes gateway
operator docs contain a detailed
list on the differences.
- Certain Kubernetes control plane operations will trigger a STUN/TURN server restart in STUNner,
which leads to dropping all active client connections. In particular, adding, removing or
modifying Gateways currently requires a full server restart. Modifications to a GatewayConfig or
UDPRoute however are reconciled seamlessly. We plan to remove this restriction in a later
release; for now it is best to refrain from intrusive changes on live STUNner deployments.
- STUNner supports arbitrary scale-up without dropping active calls, but scale-down might
disconnect calls established through the STUNner pods and/or media server replicas being removed
from the load-balancing pool. Note that this problem is
universal in WebRTC, but we plan to
do something about it in a later STUNner release so stay tuned.
- STUNner supports multiple parallel GatewayClass hierarchies, each deployed into a separate
namespace with a separate GatewayClass an a separate dataplane. This mode can be useful for
testing new STUNner versions or canary-upgrades and A/B testing of a new media server version. At
the moment, however, this mode is not supported: it should work but we don' test it.
- Currently TURN over TLS/DTLS is supported only in the standalone mode; we
will remove this restriction in the next release.
Milestones
- v0.9: Demo release: STUNner basic UDP/TURN connectivity + helm chart + tutorials
- v0.10: Dataplane: Long-term STUN/TURN credentials and STUN/TURN over
TCP/TLS/DTLS in standalone mode.
- v0.11: Control plane: Kubernetes gateway operator and dataplane reconciliation.
- v0.12: Security: Hide plain text credentials in the Gateway API and expose TLS/DTLS.
- v0.13: Observability: Prometheus + Grafana dashboard.
- v1.0: GA
- v1.2: Performance: eBPF acceleration
- v2.0: Service mesh: adaptive scaling & resiliency
Help
STUNner development is coordinated in Discord, feel free to join.
License
Copyright 2021-2022 by its authors. Some rights reserved. See AUTHORS.
MIT License - see LICENSE for full text.
Acknowledgments
Initial code adopted from pion/stun and
pion/turn.