external-dns-operator

README

ExternalDNS Operator

The ExternalDNS Operator allows you to deploy and manage ExternalDNS, a cluster-internal component which makes Kubernetes resources discoverable through public DNS servers. For more information about the initial motivation, see ExternalDNS Operator OpenShift Enhancement Proposal.

• Deploying the ExternalDNS Operator
  • Preparing the environment
  • Installing the ExternalDNS Operator by building and pushing the Operator image to a registry
  • Installing the ExternalDNS Operator using a custom catalog image on OperatorHub
• Using custom operand image
• Running end-to-end tests manually
• Proxy support
• Metrics
• Status of providers
• Known limitations
  • Length of the domain name

Deploying the ExternalDNS Operator

The following procedure describes how to deploy the ExternalDNS Operator for AWS.

Preparing the environment

Prepare your environment for the installation commands.

• Select the container runtime you want to build the images with (podman or docker):

  export CONTAINER_ENGINE=podman
  

• Select the name settings of the image:

  export REGISTRY=quay.io
  export REPOSITORY=myuser
  export VERSION=1.0.0
  

• Login to the image registry:

  ${CONTAINER_ENGINE} login ${REGISTRY} -u ${REPOSITORY}
  

Installing the ExternalDNS Operator by building and pushing the Operator image to a registry

1. Build and push the Operator image to a registry:

   export IMG=${REGISTRY}/${REPOSITORY}/external-dns-operator:${VERSION}
   make image-build image-push
   

2. Optional: you may need to link the registry secret to external-dns-operator service account if the image is not public (Doc link):

   a. Create a secret with authentication details of your image registry:

   oc -n external-dns-operator create secret generic extdns-pull-secret  --type=kubernetes.io/dockercfg  --from-file=.dockercfg=${XDG_RUNTIME_DIR}/containers/auth.json
   

   b. Link the secret to external-dns-operator service account:

   oc -n external-dns-operator secrets link external-dns-operator extdns-pull-secret --for=pull
   

3. Run the following command to deploy the ExternalDNS Operator:

   make deploy
   

4. The previous step deploys the validation webhook, which requires TLS authentication for the webhook server. The manifests deployed through the make deploy command do not contain a valid certificate and key. You must provision a valid certificate and key through other tools. You can use a convenience script, hack/generate-certs.sh to generate the certificate bundle and patch the validation webhook config.
   Important: Do not use the hack/generate-certs.sh script in a production environment.
   Run the hack/generate-certs.sh script with the following inputs:

   hack/generate-certs.sh --service webhook-service --webhook validating-webhook-configuration \
   --secret webhook-server-cert --namespace external-dns-operator
   

   Note: you may need to wait for the retry of the volume mount in the operator's POD

5. Now you can deploy an instance of ExternalDNS:

   • Run the following command to create the credentials secret for AWS:

     kubectl -n external-dns-operator create secret generic aws-access-key \
             --from-literal=aws_access_key_id=${ACCESS_KEY_ID} \
             --from-literal=aws_secret_access_key=${ACCESS_SECRET_KEY}
     

     Note: See this guide for instructions specific to other providers.

   • Run the following command:

     # for AWS
     kubectl apply -k config/samples/aws
     

     Note: For other providers, see config/samples/.

Installing the ExternalDNS Operator using a custom catalog image on OperatorHub

1. Build and push the operator image to the registry:

   export IMG=${REGISTRY}/${REPOSITORY}/external-dns-operator:${VERSION}
   make image-build image-push
   

2. Build and push the bundle image to the registry:

   export BUNDLE_IMG=${REGISTRY}/${REPOSITORY}/external-dns-operator-bundle:${VERSION}
   make bundle-image-build bundle-image-push
   

3. Build and push the catalog image to the registry:

   export CATALOG_IMG=${REGISTRY}/${REPOSITORY}/external-dns-operator-catalog:${VERSION}
   make catalog-image-build catalog-image-push
   

4. Optional: you may need to link the registry secret to the pod of external-dns-operator created in the openshift-marketplace namespace if the image is not made public (Doc link). If you are using podman then these are the instructions:

   a. Create a secret with authentication details of your image registry:

   oc -n openshift-marketplace create secret generic extdns-olm-secret  --type=kubernetes.io/dockercfg  --from-file=.dockercfg=${XDG_RUNTIME_DIR}/containers/auth.json
   

   b. Link the secret to default service account:

   oc -n openshift-marketplace secrets link default extdns-olm-secret --for=pull
   

5. Create the CatalogSource object:

   cat <<EOF | oc apply -f -
   apiVersion: operators.coreos.com/v1alpha1
   kind: CatalogSource
   metadata:
     name: external-dns-operator
     namespace: openshift-marketplace
   spec:
     sourceType: grpc
     image: ${CATALOG_IMG}
   EOF
   

6. Create the operator namespace:

   oc create namespace external-dns-operator
   oc label namespace external-dns-operator openshift.io/cluster-monitoring=true
   

7. Create the OperatorGroup object to scope the operator to external-dns-operator namespace:

   cat <<EOF | oc apply -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: external-dns-operator
     namespace: external-dns-operator
   spec:
     targetNamespaces:
     - external-dns-operator
   EOF
   

8. Create the Subscription object:

   cat <<EOF | oc apply -f -
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: external-dns-operator
     namespace: external-dns-operator
   spec:
     channel: stable-v1
     name: external-dns-operator
     source: external-dns-operator
     sourceNamespace: openshift-marketplace
   EOF
   

   Note: The steps starting from the 7th can be replaced with the following actions in the web console: Navigate to Operators -> OperatorHub, search for the ExternalDNS Operator, and install it in the external-dns-operator namespace.

9. Now you can deploy an instance of ExternalDNS:

   # for AWS
   oc apply -k config/samples/aws
   

   Note: For other providers, see config/samples/.

Using custom operand image

1. Optional: you may need to link the registry secret to the operand's service account if your custom image is not public (Doc link):

   a. Create a secret with authentication details of your image registry:

   oc -n external-dns-operator create secret generic extdns-pull-secret --type=kubernetes.io/dockercfg --from-file=.dockercfg=${XDG_RUNTIME_DIR}/containers/auth.json
   

   b. Find the service account of your operand:

   oc -n external-dns-operator get sa | grep external-dns
   

   c. Link the secret to found service account:

   oc -n external-dns-operator secrets link external-dns-sample-aws extdns-pull-secret --for=pull
   

2. Patch RELATED_IMAGE_EXTERNAL_DNS environment variable's value with your custom operand image:

   • In the operator's deployment:

   # "external-dns-operator" container has index 0
   # "RELATED_IMAGE_EXTERNAL_DNS" environment variable has index 1
   oc -n external-dns-operator patch deployment external-dns-operator --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/env/1/value", "value":"<CUSTOM_IMAGE_TAG>"}]'
   

   • Or in the operator's subscription:

   oc -n external-dns-operator patch subscription external-dns-operator --type='json' -p='[{"op": "add", "path": "/spec/config", "value":{"env":[{"name":"RELATED_IMAGE_EXTERNAL_DNS","value":"<CUSTOM_IMAGE_TAG>"}]}}]'
   

Running end-to-end tests manually

1. Deploy the operator as described above

2. Set the necessary environment variables

   For AWS:

   export KUBECONFIG=/path/to/mycluster/kubeconfig
   export DNS_PROVIDER=AWS
   export AWS_ACCESS_KEY_ID=my-aws-access-key
   export AWS_SECRET_ACCESS_KEY=my-aws-access-secret
   

   For Infoblox:

   export KUBECONFIG=/path/to/mycluster/kubeconfig
   export DNS_PROVIDER=INFOBLOX
   export INFOBLOX_GRID_HOST=myinfoblox.myorg.com
   export INFOBLOX_WAPI_USERNAME=my-infoblox-username
   export INFOBLOX_WAPI_PASSWORD=my-infoblox-password
   

   For the other providers: check out e2e directory.

3. Run the test suite

   make test-e2e
   

Proxy support

Configuring proxy support for ExternalDNS Operator

Metrics

The ExternalDNS Operator exposes controller-runtime metrics using custom resources expected by Prometheus Operator. ServiceMonitor object is created in the operator's namespace (by default external-dns-operator), make sure that your instance of the Prometheus Operator is properly configured to find it.
You can check .spec.serviceMonitorNamespaceSelector and .spec.serviceMonitorSelector fields of prometheus resource and edit the operator's namespace or service monitor accordingly:

kubectl -n monitoring get prometheus k8s --template='{{.spec.serviceMonitorNamespaceSelector}}{{"\n"}}{{.spec.serviceMonitorSelector}}{{"\n"}}'
map[matchLabels:map[openshift.io/cluster-monitoring:true]]
map[]

For OpenShift:

oc -n openshift-monitoring get prometheus k8s --template='{{.spec.serviceMonitorNamespaceSelector}}{{"\n"}}{{.spec.serviceMonitorSelector}}{{"\n"}}'
map[matchLabels:map[openshift.io/cluster-monitoring:true]]
map[]

Status of providers

We define the following stability levels for DNS providers:

• GA: Integration and smoke tests before release are done on the real platforms. API is stable with a guarantee of no breaking changes.
• TechPreview: Maintainers have no access to resources to execute integration tests on the real platform, API may be a subject to a change.

Provider	Status
AWS Route53	GA
AWS Route53 on GovCloud	TechPreview
AzureDNS	GA
GCP Cloud DNS	GA
Infoblox	GA
BlueCat	TechPreview

Known limitations

Length of the domain name

ExternalDNS Operator uses the TXT registry which implies the usage of the new format and the prefix for the TXT records. This reduces the maximum length of the domain name for the TXT records.
Since the TXT record accompanies every DNS record, there cannot be a DNS record without a corresponding TXT record, therefore the DNS record's domain name gets hit by the same limit:

DNS record: <domain-name-from-source>
TXT record: external-dns-<record-type>-<domain-name-from-source>

Please be aware that instead of the standardized 63 characters long maximum length, the domain names generated from the ExternalDNS sources have the following limits:

• for the CNAME record type:
  • 44 characters
  • 42 characters for wildcard records on AzureDNS (OCPBUGS-819)
• for the A record type:
  • 48 characters
  • 46 characters for wildcard records on AzureDNS (OCPBUGS-819)

Example

ExternalDNS CR which uses Service of type ExternalName (to force CNAME record type) as the source for the DNS records:

$ oc get externaldns aws -o yaml
apiVersion: externaldns.olm.openshift.io/v1beta1
kind: ExternalDNS
metadata:
  name: aws
spec:
  provider:
    type: AWS
  zones:
  - Z06988883Q0H0RL6UMXXX
  source:
    type: Service
    service:
        serviceType:
        - ExternalName
    fqdnTemplate:
    - "{{.Name}}.test.example.io"

The service of the following name will not pose any problems because its name is 44 characters long:

$ oc get svc
NAME                                                          TYPE           CLUSTER-IP     EXTERNAL-IP              PORT(S)             AGE
hello-openshift-aaaaaaaaaa-bbbbbbbbbb-cccccc                  ExternalName   <none>         somednsname.example.io   8080/TCP,8888/TCP

$ aws route53 list-resource-record-sets --hosted-zone-id=Z06988883Q0H0RL6UMXXX
RESOURCERECORDSETS	external-dns-cname-hello-openshift-aaaaaaaaaa-bbbbbbbbbb-cccccc.test.example.io.	300	TXT
RESOURCERECORDS	"heritage=external-dns,external-dns/owner=external-dns-aws,external-dns/resource=service/test-long-dnsname/hello-openshift-aaaaaaaaaa-bbbbbbbbbb-cccccc"
RESOURCERECORDSETS	external-dns-hello-openshift-aaaaaaaaaa-bbbbbbbbbb-cccccc.test.example.io.	300	TXT
RESOURCERECORDS	"heritage=external-dns,external-dns/owner=external-dns-aws,external-dns/resource=service/test-long-dnsname/hello-openshift-aaaaaaaaaa-bbbbbbbbbb-cccccc"
RESOURCERECORDSETS	hello-openshift-aaaaaaaaaa-bbbbbbbbbb-cccccc.test.example.io.	300	CNAME
RESOURCERECORDS	somednsname.example.io

The service of a longer name will result in no changes on the DNS provider and the errors similar to the below ones in the ExternalDNS instance:

$ oc -n external-dns-operator logs external-dns-aws-7ddbd9c7f8-2jqjh
...
time="2022-09-02T08:53:57Z" level=info msg="Desired change: CREATE external-dns-cname-hello-openshift-aaaaaaaaaa-bbbbbbbbbb-ccccccc.test.example.io TXT [Id: /hostedzone/Z06988883Q0H0RL6UMXXX]"
time="2022-09-02T08:53:57Z" level=info msg="Desired change: CREATE external-dns-hello-openshift-aaaaaaaaaa-bbbbbbbbbb-ccccccc.test.example.io TXT [Id: /hostedzone/Z06988883Q0H0RL6UMXXX]"
time="2022-09-02T08:53:57Z" level=info msg="Desired change: CREATE hello-openshift-aaaaaaaaaa-bbbbbbbbbb-ccccccc.test.example.io A [Id: /hostedzone/Z06988883Q0H0RL6UMXXX]"
time="2022-09-02T08:53:57Z" level=error msg="Failure in zone test.example.io. [Id: /hostedzone/Z06988883Q0H0RL6UMXXX]"
time="2022-09-02T08:53:57Z" level=error msg="InvalidChangeBatch: [FATAL problem: DomainLabelTooLong (Domain label is too long) encountered with 'external-dns-a-hello-openshift-aaaaaaaaaa-bbbbbbbbbb-ccccccc']\n\tstatus code: 400, request id: e54dfd5a-06c6-47b0-bcb9-a4f7c3a4e0c6"
...