consistenthash

README

Distributed Build with Consistent Hashing

Demo for efficiently using BuildKit daemon-local cache with multi-node cluster

Deploy

$ kubectl apply -f ../statefulset.rootless.yaml
$ kubectl scale --replicas=10 statefulset/buildkitd

Consistent hashing

Define the key string for consistent hashing.

For example, the key can be defined as <REPO NAME>:<CONTEXT PATH>, e.g. github.com/example/project:some/directory.

Then determine the pod that corresponds to the key:

$ go build -o consistenthash .
$ pod=$(./show-running-pods.sh | consistenthash $key)

You can connect to the pod using export BUILDKIT_HOST=kube-pod://$pod.

Documentation

Overview

Package main provides simple consistenthash commandline utility.

Demo:

$ seq -f node-%.f 1 100 > /tmp/nodes
$ consistenthash < /tmp/nodes apple
node-42
$ consistenthash < /tmp/nodes banana
node-48
$ consistenthash < /tmp/nodes chocolate
node-2

Modify /tmp/nodes and confirm that the result rarely changes.