admissioncontrol

README

Admission Control

🕵️🕵️🕵️

A micro-framework for building and deploying dynamic Admission Controllers for your Kubernetes clusters. It reduces the boilerplate needed to inspect, validate and/or reject the admission of objects to your cluster, allowing you to focus on writing the specific business logic you want to enforce.

• Can be used as the target of both ValidatingWebhookConfiguration and MutatingWebhookConfiguration - handlers can return simple allow/deny responses, or patches (mutations) to submitted resources.
• Provides an extensible AdmissionHandler type that accepts a custom admission function (called an AdmitFunc), making it easy for you to add new validating or mutating webhook endpoints.
• Provides sample Deployment, Service and ValidatingWebhookConfiguration definitions for you to build off of, and an example webhook server as additional guidance.

---

• Using the Framework
• Built-In AdmitFuncs
• Creating Your Own AdmitFunc
• Configuring & Deploying a Server
• Pre-requisites
• Setup
  • Configuring a Server
  • Generating TLS Certificates
  • Deploying the Admission Controller
• Troubleshooting
• Contributing
• License

---

Using the Framework

Built-In AdmitFuncs

Admission Control provides a number of useful built-in AdmitFuncs, including:

• EnforcePodAnnotations - ensures that admitted Pods have (at least) the required set of annotations. Annotation values are matched using a matchFunc (a func(string) bool) that allows flexible matching. For example, a matchFunc could wrap the IsDomainName function from miekg/dns, or reference a []string of accepted values. It is strongly suggested you use a namespaceSelector as part of your webhook configuration to only apply this to specific namespaces, and/or set the ignoreNamespaces argument to include kube-system, as annotation validation will otherwise include system Pods.
• DenyPublicLoadBalancers - prevents exposing Services of type: LoadBalancer outside of the cluster, instead requiring the LB to be annotated as internal-only, by looking for the well-known annotations for major cloud providers.
• DenyIngresses - similar to the above, it prevents creating Ingresses (except in the namespaces you allow). This can be useful for limiting which namespaces can expose services via common Ingress types.

More built-ins are coming soon, and suggestions are welcome! ⏳

Creating Your Own AdmitFunc

The core type of the library is the AdmitFunc - a function that takes a k8s AdmissionReview object and returns an (*AdmissionResponse, error) tuple. You can provide a closure that returns an AdmitFunc type if you need to inject additional dependencies into your handler, and/or use a constructor function to do the same.

The AdmissionReview type wraps the AdmissionRequest, which can be serialized into a concrete type—such as a Pod or Service—and subsequently validated.

An example AdmitFunc looks like this:

// DenyDefaultLoadBalancerSourceRanges denies any kind: Service of type:
// LoadBalancer that does not explicitly set .spec.loadBalancerSourceRanges -
// which defaults to 0.0.0.0/0 (e.g. Internet traffic, if routable).
//
// This prevents LoadBalancers from being accidentally exposed to the Internet.
func DenyDefaultLoadBalancerSourceRanges() AdmitFunc {
    // Return a function of type AdmitFunc
    return func(admissionReview *admission.AdmissionReview) (*admission.AdmissionResponse, error) {
        kind := admissionReview.Request.Kind.Kind
        // Create an *admission.AdmissionResponse that denies by default.
        resp := newDefaultDenyResponse()

        // Create an object to deserialize our requests' object into
        service := core.Service{}
        deserializer := serializer.NewCodecFactory(runtime.NewScheme()).UniversalDeserializer()
        if _, _, err := deserializer.Decode(admissionReview.Request.Object.Raw, nil, &service); err != nil {
          return nil, err
        }

        // Allow non-LoadBalancer Services to pass through.
        if service.Spec.Type != "LoadBalancer" {
          resp.Allowed = true
          resp.Result.Message = fmt.Sprintf(
            "received a non-LoadBalancer type (%s)",
            service.Spec.Type,
          )
          return resp, nil
        }

        // Inspect the service.Spec.LoadBalancerSourceRanges field
        // If unset, reject it.
        // Returning an error from an AdmitFunc will automatically deny admission of that requests' object.
        if service.Spec.LoadBalancerSourceRanges == nil {
          return resp, fmt.Errorf("LoadBalancers without explicitly configured LoadBalancerSourceRanges are not allowed.")
        }

        // Set resp.Allowed to true before returning your AdmissionResponse
        resp.Allowed = true
        return resp, nil
    }
}

You can see that we deserialize the raw object in our AdmissionReview into an object (based on its Kind), inspect and validate the fields we're interested in, and either return an error (rejecting admission) or set resp.Allowed = true and allow admission.

Tips:

• Having your AdmitFuncs focus on "one" thing is best practice: it allows you to be more granular in how you apply constraints to your cluster
• Returning an AdmitFunc from a constructor/closure will allow you to inject dependencies and/or configuration into your handler.

You can then create an AdmissionHandler and pass it the AdmitFunc. Use your favorite HTTP router, and associate a path with your handler:

	// We're using "gorilla/mux" as our router here.
	r := mux.NewRouter().StrictSlash(true)
	admissions := r.PathPrefix("/admission-control").Subrouter()
	admissions.Handle("/deny-default-load-balancer-source-ranges", &admissioncontrol.AdmissionHandler{
		AdmitFunc:  admissioncontrol.DenyDefaultLoadBalancerSourceRanges(),
		Logger:     logger,
	}).Methods(http.MethodPost)

The example server admissiond provides a more complete example of how to configure & serve your admission controller endpoints.

---

Configuring & Deploying a Server

There are two ways to deploy an admission controller:

1. Within your Kubernetes cluster ("in-cluster"), where it runs as a Pod and is exposed as a Service to the rest of the cluster. This requires you to provision a TLS keypair, as admission controllers can only be accessed over TLS (HTTPS).
2. Out-of-cluster, where it is accessible over HTTPS by the cluster. The admission controller could be hosted on another cluster, or more commonly, via a serverless platform like Cloud Run or Azure Container Instances. See the CloudRun.Dockerfile for an example of how to build an image for Cloud Run.

The documentation below covers deploying within a Kubernetes cluster (option 1).

Pre-requisites

You'll need:

• Access to a Kubernetes cluster (GKE, minikube, AKS, etc) with support for admission webhooks (v1.9+)
• cfssl as part of the process of generating a TLS key-pair, and some familiarity with creating TLS (SSL) certificates (CSRs, PEM-encoded certificates, keys).
• Experience writing Go - for implementing your own AdmitFuncs (refer to the example DenyPublicServices AdmitFunc included).
• Experience building OCI (Docker) containers via docker build or similar.

Setup

Setting up an Admission Controller in your Kubernetes cluster has three major steps:

1. Generate a TLS keypair—Kubernetes only allows HTTPS (TLS) communication to Admission Controllers, whether in-cluster or hosted externally—and make the key & certificate available as a Secret within your cluster.

2. Create a Deployment with your Admission-Control-based server, mounting the TLS keypair in your Secret as a volume in the container.

3. Configure a ValidatingWebhookConfiguration that tells Kubernetes which objects should be validated, and the endpoint (URL) on your Service to validate them against.

Your single server can act as the admission controller for any number of ValidatingWebhookConfiguration or MutatingWebhookConfiguration - each configuration can point to a specific URL on the same server.

Configuring a Server

⚠ Reminder: Admission webhooks must support HTTPS (TLS) connections; k8s does not allow webhooks to be reached over plain-text HTTP. If running in-cluster, the Service fronting the controller must be reachable via TCP port 443. External webhooks only need to satisfy the HTTPS requirement, but can be reached on any valid TCP port.

Having your k8s cluster create a TLS certificate for you will dramatically simplify the configuration, as self-signed certificates require you to provide a .webhooks.clientConfig.caBundle value for verification.

The key steps include:

1. Generate a TLS keypair for the admission controller by issuing a CertificateSigningRequest against the Kubernetes cluster, and obtain the CA certificate from the k8s cluster.
2. Creating a Deployment and a Service that makes the admission controller available to the cluster.
3. Creating a ValidatingWebhookConfiguration that points matching k8s API requests to a route on your admission controller. i.e. you may want to configure different validation policies between Services and Pods.

---

Generating TLS Certificates

As noted above, we need to make our webhook endpoint available over HTTPS (TLS), which requires generating a CA cert (required as the caBundle value), key and certificate. You can can choose to have your k8s cluster sign & provide a cert for you, or otherwise provide your own self-signed cert & CA cert.

We're going to have our cluster issue a certificate for us, which simplifies the process:

1. Create a k8s CertificateSigningRequest for the hostname(s) you will deploy the Service as. There is an example CSR in demo-certs/csr.yaml for the admission-control-service.default.svc hostname. This hostname must match the .webhooks.name[].clientConfig.service.name described in your ValidatingWebhookConfiguration.

2. Approve and then fetch the certificate from the k8s API server.

3. Create a Secret that contains the TLS key-pair - the key you created alongside the CSR in step 1, and the certificate you fetched via kubectl get csr <name> ... - e.g. kubectl create secret tls <name> --cert=cert.crt --key=key.key.

4. Retrieve the k8s cluster CA cert - this will be the .webhooks.clientConfig.caBundle value in our ValidatingWebhookConfiguration: `

kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}'

Specifically, you'll want to make sure your manifest looks like this:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: deny-public-services
webhooks:
  - name: deny-public-services.questionable.services
    # <snip, for brevity>
    clientConfig:
      service:
        # This is the hostname our certificate needs in its Subject Alternative
        # Name array - name.namespace.svc
        # If the certificate does NOT have this name, TLS validation will fail.
        name: admission-control-service
        namespace: default
        path: "/admission-control/deny-public-services"
      # This will be the CA cert from your k8s cluster, or the CA cert you
      # generated if you took the DIY approach.
      caBundle: "<your-base64-encoded-PEM-certificate-here>"

With the TLS certificates in hand, you can now move on to deploying the controller.

---

Deploying the Admission Controller

With the TLS certificates generated & the associated Secret created, we can update our Deployment, Service and ValidatingWebhookConfiguration in-kind. Refer to the samples/ directory if you need a reference config.

1. Create a Deployment and make sure the -host flag passed to the admissiond container matches the hostname (ServerName) you used in the CSR.
2. Update the .spec.containers[].volumes.secret.secretName to refer to the Secret you created in step 3.
3. Create a Service that exposes the Deployment in step no. 4 to the cluster. Remember: the name of the Service should match one of the names in step 1.
4. Create a ValidatingWebhookConfiguration that matches the objects (kinds, versions) and actions (create, update, delete), and configure the .webhooks.clientConfig.service map to point to the Service you created.

Note: A set of example manifests - both admissiond-deployment.yml and deny-public-admissions-config.yml- are available in the samples/ directory.

To deploy the built-in server to your cluster with its existing validation endpoints, you'll need to build the container image and push it to an image registry that your k8s cluster can access.

If you're using Google Container Registry, you can push images to the same project as your GKE cluster:

docker build -t yourco/admissiond .
docker tag yourco/admissiond gcr.io/$PROJECTNAME/admissiond
docker push gcr.io/$PROJECTNAME/admissiond

Make sure to update/copy samples/admission-control-service.yaml with the new container image URL before deploying it:

# An example Deployment we'll try to expose
kubectl apply -f samples/hello-app.yaml
# Install the Admission Controller into the cluster
kubectl apply -f samples/admission-control-service.yaml
# Add our ValidatingWebhookConfiguration
kubectl apply -f samples/deny-public-webhook-config.yaml

Let's now attempt to deploy a kind: Service of type: LoadBalancer without the internal-only annotations:

kubectl apply -f samples/public-service.yaml

You should see the following output:

Error from server (hello-service does not have the cloud.google.com/load-balancer-type: Internal annotation.): error when creating "samples/public-service.yaml": admission webhook "deny-public-services.questionable.services" denied the request: Services of type: LoadBalancer without an internal annotation are not allowed on this cluster

Perfect! 🎉

---

Troubleshooting

If you run into problems setting up the admission-controller, make sure that:

• Your certificates are valid / key-pairs match
• You've inspected the Pod logs - e.g. via kubectl logs -f -l app=admission-control - all HTTP handler errors are logged to the configured logger.
• Your ValidatingWebhookConfiguration is matching the right API versions, namespaces & objects vs. what you have configured as an AdmitFunc endpoint in the admission-control server.

If you're stuck, open an issue with the output of:

kubectl version
# replace the label if you've authored your own Deployment manifest
kubectl logs -f -l app=admission-control

... and any relevant error messages from attempting to kubectl apply -f <manifest> that match your ValidatingWebhookConfiguration.

Contributing

This project is open to contributions!

As a courtesy: please open an issue with a brief proposal of your idea first (and the use-cases surrounding it!) before diving into implementation.

License

Apache 2.0 licensed. Copyright Google, LLC (2019). See the LICENSE file for details.

Documentation

Index

• func LoggingMiddleware(logger log.Logger) func(http.Handler) http.Handler
• type AdmissionError
  • func (e AdmissionError) Error() string
• type AdmissionHandler
  • func (ah *AdmissionHandler) ServeHTTP(w http.ResponseWriter, r *http.Request)
• type AdmissionServer
  • func NewServer(srv *http.Server, logger log.Logger) (*AdmissionServer, error)
  • func (as *AdmissionServer) Run(ctx context.Context) error
  • func (as *AdmissionServer) Stop() error
• type AdmitFunc
  • func DenyIngresses(ignoredNamespaces []string) AdmitFunc
  • func DenyPublicLoadBalancers(ignoredNamespaces []string, provider CloudProvider) AdmitFunc
  • func EnforcePodAnnotations(ignoredNamespaces []string, requiredAnnotations map[string]func(string) bool) AdmitFunc
• type CloudProvider

Functions

func LoggingMiddleware

func LoggingMiddleware(logger log.Logger) func(http.Handler) http.Handler

LoggingMiddleware logs the incoming HTTP request & its duration.

Types

type AdmissionError

type AdmissionError struct {
	Allowed bool
	Message string
	Debug   string
}

AdmissionError represents an error (rejection, serialization error, etc) from an AdmissionHandler endpoint/handler.

func (AdmissionError) Error

func (e AdmissionError) Error() string

type AdmissionHandler

type AdmissionHandler struct {
	// The AdmitFunc to invoke for this handler.
	AdmitFunc AdmitFunc
	// A kitlog.Logger compatible interface
	Logger log.Logger
	// LimitBytes limits the size of objects the webhook will handle.
	LimitBytes int64
	// contains filtered or unexported fields
}

AdmissionHandler represents the configuration & associated endpoint for an k8s ValidatingAdmissionController (or MutatingAdmissionController) webhook.

Multiple instances can be created with distinct AdmitFuncs to handle different admission requirements.

func (*AdmissionHandler) ServeHTTP

func (ah *AdmissionHandler) ServeHTTP(w http.ResponseWriter, r *http.Request)

type AdmissionServer

type AdmissionServer struct {

	// GracePeriod is defines how long the server allows for in-flight connections
	// to complete before exiting.
	GracePeriod time.Duration
	// contains filtered or unexported fields
}

AdmissionServer represents a HTTP server configuration for serving an Admission Controller.

Use NewServer to create a new AdmissionServer.

func NewServer

func NewServer(srv *http.Server, logger log.Logger) (*AdmissionServer, error)

NewServer creates an unstarted AdmissionServer, ready to be started (via the 'Run' method).

The provided *http.Server must have its Handler field set, as well as a valid and non-nil TLSConfig. Kubernetes requires that Admission Controllers are only reachable over HTTPS (TLS), whether running in-cluster or externally.

func (*AdmissionServer) Run

func (as *AdmissionServer) Run(ctx context.Context) error

Run the AdmissionServer; starting the configured *http.Server, and blocking indefinitely.

Run will return under three explicit cases:

1. An interrupt (SIGINT; "Ctrl+C") or termination (SIGTERM) signal, such as the SIGTERM most process managers send: e.g. as Kubernetes sends to a Pod: https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods

2. When an error is returned from the listener on our server (fails to bind to a port, terminal network issue, etc.)

3. When we receive a cancellation signal from the parent context; e.g. by calling the returned CancelFunc from calling context.WithCancel(ctx)

This allows us to stop accepting connections, allow in-flight connections to finish gracefully (up to the configured grace period), and then close the server. You may also call the .Stop() method on the server to trigger a shutdown.

func (*AdmissionServer) Stop

func (as *AdmissionServer) Stop() error

Stop stops the AdmissionServer, if running, waiting for configured grace period.

type AdmitFunc

type AdmitFunc func(reviewRequest *admission.AdmissionReview) (*admission.AdmissionResponse, error)

AdmitFunc is a type for building Kubernetes admission webhooks. An AdmitFunc should check whether an admission request is valid, and shall return an admission response that sets AdmissionResponse.Allowed to true or false as needed.

Users wishing to build their own admission handlers should satisfy the AdmitFunc type, and pass it to an AdmissionHandler for serving over HTTP.

Note: this mirrors the type in k8s source: https://github.com/kubernetes/kubernetes/blob/v1.13.0/test/images/webhook/main.go#L43-L44

func DenyIngresses

func DenyIngresses(ignoredNamespaces []string) AdmitFunc

DenyIngresses denies any kind: Ingress from being deployed to the cluster, except for any explicitly allowed namespaces (e.g. istio-system).

Providing an empty/nil list of ignoredNamespaces will reject Ingress objects across all namespaces.

Kinds other than Ingress will be allowed.

func DenyPublicLoadBalancers

func DenyPublicLoadBalancers(ignoredNamespaces []string, provider CloudProvider) AdmitFunc

DenyPublicLoadBalancers denies any non-internal public cloud load balancers (kind: Service of type: LoadBalancer) by looking for their "internal" load balancer annotations. This prevents accidentally exposing Services to the Internet for Kubernetes clusters designed to be internal-facing only.

The required annotations are documented at https://kubernetes.io/docs/concepts/services-networking/#internal-load-balancer

Services with a .spec.type other than LoadBalancer will NOT be rejected by this handler.

Providing an empty/nil list of ignoredNamespaces will reject LoadBalancers across all namespaces.

func EnforcePodAnnotations

func EnforcePodAnnotations(ignoredNamespaces []string, requiredAnnotations map[string]func(string) bool) AdmitFunc

EnforcePodAnnotations ensures that Pods have the required annotations by looking for a strict (case-sensitive) key-match, and then running the matchFunc (a func(string) bool) over the value.

This allows the caller to perform flexible matching - checking for valid DNS names or a list of accepted values - rather than having to iterate over all possible values, which may not be possible.

EnforcePodAnnotations can inspect Pods, Deployments, StatefulSets, DaemonSets & Jobs.

Unknown object kinds are rejected. You can create multiple versions of this AdmitFunc for a given ValidatingAdmissionWebhook configuration if you wish to apply different configurations per kind or namespace.

type CloudProvider

type CloudProvider int

CloudProvider represents supported cloud platforms for provider-specific configuration.

const (
	// GCP is a constant for Google Cloud Platform specific logic.
	GCP CloudProvider = iota
	// Azure is a constant for cloud-specific logic.
	Azure
	// AWS is a constant for Amazon Web Services specific logic.
	AWS
	// OpenStack is a constant for cloud-specific logic.
	OpenStack
)