README ¶
Disco
Disco is a service discovery for MinIO.
The motivation is that some stateful services have the need for a static IP address to discover peers it's peers, Kubernetes offers this functionality however kube-dns
has a large delay in announcing the new Pods, therefore we propose this layer to act immediately upon the creation of the desired pod.
Pods are automatically added to the DNS as long at the have the annotation io.min.disco
. This annotation supports jsonpath expressions.
For example, after setting up Disco
as a service we can configure a new Statefulset to use it to resolve the name of the peer replicas. So if we setup a MinIO
instance that looks for it's peers at at hostname zone-1-{0...3}.zone-1
we can indicate this to the Disco
via the annotation:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: zone-1
namespace: default
spec:
podManagementPolicy: Parallel
replicas: 4
selector:
matchLabels:
app: minio
controller: zone-1
serviceName: zone-1
template:
metadata:
labels:
app: minio
controller: zone-1
annotations:
io.min.disco: '{.metadata.name}.{.metadata.labels.controller}'
spec:
dnsPolicy: "None"
dnsConfig:
nameservers:
- 10.110.109.99
containers:
- name: minio
args:
- server
- http://zone-1-{0...3}.zone-1/data{1...4}
Deploy
You can deploy with kustomize
kustomize build deployment/base | kubectl apply -f -
After deploying disco
you should find out the IP for the minio-disco
service since that is going to be used to configure the top level minio.local
domain.
$ kubectl get svc minio-disco -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
minio-disco ClusterIP 10.109.234.52 <none> 53/UDP,53/TCP 12m app=minio-disco
Here we can see the IP is 10.109.234.52
so we are going to add that to the Corefile
stored in the coredns
configmap inside the kube-system
namespace.
$ kubectl -n kube-system edit configmap corends
and add at the end of Corefile
minio.local:53 {
errors
cache 30
forward . 10.109.234.52
}
The file should look like this
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
data:
Corefile: |
.:53 {
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
minio.local:53 {
errors
cache 30
forward . 10.109.234.52
}
Afterwards, restart the coredns pods on the kube-system
namespace
$ kubectl -n kube-system delete pod $(kubectl -n kube-system get pods | grep coredns | awk '{print $1}')
License
Use of disco
is governed by the AGPLv3 license that can be found in the LICENSE file.
Documentation ¶
There is no documentation for this package.