elasticsearch

command
v0.19.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 9, 2015 License: Apache-2.0 Imports: 10 Imported by: 0

README

Elasticsearch for Kubernetes

This directory contains the source for a Docker image that creates an instance of Elasticsearch 1.5.2 which can be used to automatically form clusters when used with replication controllers. This will not work with the library Elasticsearch image because multicast discovery will not find the other pod IPs needed to form a cluster. This image detects other Elasticsearch pods running in a specified namespace with a given label selector. The detected instances are used to form a list of peer hosts which are used as part of the unicast discovery mechansim for Elasticsearch. The detection of the peer nodes is done by a program which communicates with the Kubernetes API server to get a list of matching Elasticsearch pods. To enable authenticated communication this image needs a secret to be mounted at /etc/apiserver-secret with the basic authentication username and password.

Here is an example replication controller specification that creates 4 instances of Elasticsearch which is in the file music-rc.yaml.

apiVersion: v1beta3
kind: ReplicationController
metadata:
  labels:
    name: music-db
    namespace: mytunes
  name: music-db
spec:
  replicas: 4
  selector:
    name: music-db
  template:
    metadata:
      labels:
         name: music-db
    spec:
      containers:
      - name: es
        image: kubernetes/elasticsearch:1.0
        env:
          - name: "CLUSTER_NAME"
            value: "mytunes-db"
          - name: "SELECTOR"
            value: "name=music-db"
          - name: "NAMESPACE"
            value: "mytunes"
        ports:
        - name: es
          containerPort: 9200
        - name: es-transport
          containerPort: 9300
        volumeMounts:
        - name: apiserver-secret
          mountPath: /etc/apiserver-secret
          readOnly: true
      volumes:
      - name: apiserver-secret
        secret:
          secretName: apiserver-secret

The CLUSTER_NAME variable gives a name to the cluster and allows multiple separate clusters to exist in the same namespace. The SELECTOR variable should be set to a label query that identifies the Elasticsearch nodes that should participate in this cluster. For our example we specify name=music-db to match all pods that have the label name set to the value music-db. The NAMESPACE variable identifies the namespace to be used to search for Elasticsearch pods and this should be the same as the namespace specified for the replication controller (in this case mytunes).

Before creating pods with the replication controller a secret containing the bearer authentication token should be set up. A template is provided in the file apiserver-secret.yaml:

apiVersion: v1beta3
kind: Secret
metadata:
  name: apiserver-secret
  namespace: NAMESPACE
data:
  token: "TOKEN"

Replace NAMESPACE with the actual namespace to be used and TOKEN with the basic64 encoded versions of the bearer token reported by kubectl config view e.g.

$ kubectl config view
...
- name: kubernetes-logging_kubernetes-basic-auth
...
  token: yGlDcMvSZPX4PyP0Q5bHgAYgi1iyEHv2
 ...   
$ echo yGlDcMvSZPX4PyP0Q5bHgAYgi1iyEHv2 | base64
eUdsRGNNdlNaUFg0UHlQMFE1YkhnQVlnaTFpeUVIdjIK=

resulting in the file:

apiVersion: v1beta3
kind: Secret
metadata:
  name: apiserver-secret
  namespace: mytunes
data:
  token: "eUdsRGNNdlNaUFg0UHlQMFE1YkhnQVlnaTFpeUVIdjIK="

which can be used to create the secret in your namespace:

kubectl create -f apiserver-secret.yaml --namespace=mytunes
secrets/apiserver-secret

Now you are ready to create the replication controller which will then create the pods:

$ kubectl create -f music-rc.yaml --namespace=mytunes
replicationcontrollers/music-db

It's also useful to have a service with an external load balancer for accessing the Elasticsearch cluster which can be found in the file music-service.yaml.

apiVersion: v1beta3
kind: Service
metadata:
  name: music-server
  namespace: mytunes
  labels:
    name: music-db
spec:
  selector:
    name: music-db
  ports:
  - name: db
    port: 9200
    targetPort: es
  createExternalLoadBalancer: true

Let's create the service with an external load balancer:

$ kubectl create -f music-service.yaml --namespace=mytunes
services/music-server

Let's see what we've got:

$ kubectl get pods,rc,services,secrets --namespace=mytunes

POD              IP            CONTAINER(S)   IMAGE(S)                       HOST                                     LABELS          STATUS    CREATED      MESSAGE
music-db-0fwsu   10.244.2.48                                                 kubernetes-minion-m49b/104.197.35.221    name=music-db   Running   6 minutes
                               es             kubernetes/elasticsearch:1.0                                                            Running   29 seconds
music-db-5pc2e   10.244.0.24                                                 kubernetes-minion-3c8c/146.148.41.184    name=music-db   Running   6 minutes
                               es             kubernetes/elasticsearch:1.0                                                            Running   6 minutes
music-db-bjqmv   10.244.3.31                                                 kubernetes-minion-zey5/104.154.59.10     name=music-db   Running   6 minutes
                               es             kubernetes/elasticsearch:1.0                                                            Running   19 seconds
music-db-swtrs   10.244.1.37                                                 kubernetes-minion-f9dw/130.211.159.230   name=music-db   Running   6 minutes
                               es             kubernetes/elasticsearch:1.0                                                            Running   6 minutes
CONTROLLER   CONTAINER(S)   IMAGE(S)                       SELECTOR        REPLICAS
music-db     es             kubernetes/elasticsearch:1.0   name=music-db   4
NAME           LABELS          SELECTOR        IP(S)            PORT(S)
music-server   name=music-db   name=music-db   10.0.138.61      9200/TCP
                                               104.197.12.157
NAME               TYPE      DATA
apiserver-secret   Opaque    2

This shows 4 instances of Elasticsearch running. After making sure that port 9200 is accessible for this cluster (e.g. using a firewall rule for GCE) we can make queries via the service which will be fielded by the matching Elasticsearch pods.

$ curl 104.197.12.157:9200
{
  "status" : 200,
  "name" : "Warpath",
  "cluster_name" : "mytunes-db",
  "version" : {
    "number" : "1.5.2",
    "build_hash" : "62ff9868b4c8a0c45860bebb259e21980778ab1c",
    "build_timestamp" : "2015-04-27T09:21:06Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}
$ curl 104.197.12.157:9200
{
  "status" : 200,
  "name" : "Callisto",
  "cluster_name" : "mytunes-db",
  "version" : {
    "number" : "1.5.2",
    "build_hash" : "62ff9868b4c8a0c45860bebb259e21980778ab1c",
    "build_timestamp" : "2015-04-27T09:21:06Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}

We can query the nodes to confirm that an Elasticsearch cluster has been formed.

$ curl 104.197.12.157:9200/_nodes?pretty=true
{
  "cluster_name" : "mytunes-db",
  "nodes" : {
    "u-KrvywFQmyaH5BulSclsA" : {
      "name" : "Jonas Harrow",
...
        "discovery" : {
          "zen" : {
            "ping" : {
              "unicast" : {
                "hosts" : [ "10.244.2.48", "10.244.0.24", "10.244.3.31", "10.244.1.37" ]
              },
...
      "name" : "Warpath",
...
        "discovery" : {
          "zen" : {
            "ping" : {
              "unicast" : {
                "hosts" : [ "10.244.2.48", "10.244.0.24", "10.244.3.31", "10.244.1.37" ]
              },
...
        "name" : "Callisto",
...
        "discovery" : {
          "zen" : {
            "ping" : {
              "unicast" : {
                "hosts" : [ "10.244.2.48", "10.244.0.24", "10.244.3.31", "10.244.1.37" ]
              },
...
      "name" : "Vapor",
...
        "discovery" : {
          "zen" : {
            "ping" : {
              "unicast" : {
                "hosts" : [ "10.244.2.48", "10.244.0.24", "10.244.3.31", "10.244.1.37" ]
...

Let's ramp up the number of Elasticsearch nodes from 4 to 10:

$ kubectl scale --replicas=10 replicationcontrollers music-db --namespace=mytunes
scaled
$ kubectl get pods --namespace=mytunes
POD              IP            CONTAINER(S)   IMAGE(S)                       HOST                                     LABELS          STATUS    CREATED      MESSAGE
music-db-0fwsu   10.244.2.48                                                 kubernetes-minion-m49b/104.197.35.221    name=music-db   Running   33 minutes
                               es             kubernetes/elasticsearch:1.0                                                            Running   26 minutes
music-db-2erje   10.244.2.50                                                 kubernetes-minion-m49b/104.197.35.221    name=music-db   Running   48 seconds
                               es             kubernetes/elasticsearch:1.0                                                            Running   46 seconds
music-db-5pc2e   10.244.0.24                                                 kubernetes-minion-3c8c/146.148.41.184    name=music-db   Running   33 minutes
                               es             kubernetes/elasticsearch:1.0                                                            Running   32 minutes
music-db-8rkvp   10.244.3.33                                                 kubernetes-minion-zey5/104.154.59.10     name=music-db   Running   48 seconds
                               es             kubernetes/elasticsearch:1.0                                                            Running   46 seconds
music-db-bjqmv   10.244.3.31                                                 kubernetes-minion-zey5/104.154.59.10     name=music-db   Running   33 minutes
                               es             kubernetes/elasticsearch:1.0                                                            Running   26 minutes
music-db-efc46   10.244.2.49                                                 kubernetes-minion-m49b/104.197.35.221    name=music-db   Running   48 seconds
                               es             kubernetes/elasticsearch:1.0                                                            Running   46 seconds
music-db-fhqyg   10.244.0.25                                                 kubernetes-minion-3c8c/146.148.41.184    name=music-db   Running   48 seconds
                               es             kubernetes/elasticsearch:1.0                                                            Running   47 seconds
music-db-guxe4   10.244.3.32                                                 kubernetes-minion-zey5/104.154.59.10     name=music-db   Running   48 seconds
                               es             kubernetes/elasticsearch:1.0                                                            Running   46 seconds
music-db-pbiq1   10.244.1.38                                                 kubernetes-minion-f9dw/130.211.159.230   name=music-db   Running   48 seconds
                               es             kubernetes/elasticsearch:1.0                                                            Running   47 seconds
music-db-swtrs   10.244.1.37                                                 kubernetes-minion-f9dw/130.211.159.230   name=music-db   Running   33 minutes
                               es             kubernetes/elasticsearch:1.0                                                            Running   32 minutes

Let's check to make sure that these 10 nodes are part of the same Elasticsearch cluster:

$ curl 104.197.12.157:9200/_nodes?pretty=true | grep name
"cluster_name" : "mytunes-db",
      "name" : "Killraven",
        "name" : "Killraven",
          "name" : "mytunes-db"
        "vm_name" : "OpenJDK 64-Bit Server VM",
          "name" : "eth0",
      "name" : "Tefral the Surveyor",
        "name" : "Tefral the Surveyor",
          "name" : "mytunes-db"
        "vm_name" : "OpenJDK 64-Bit Server VM",
          "name" : "eth0",
      "name" : "Jonas Harrow",
        "name" : "Jonas Harrow",
          "name" : "mytunes-db"
        "vm_name" : "OpenJDK 64-Bit Server VM",
          "name" : "eth0",
      "name" : "Warpath",
        "name" : "Warpath",
          "name" : "mytunes-db"
        "vm_name" : "OpenJDK 64-Bit Server VM",
          "name" : "eth0",
      "name" : "Brute I",
        "name" : "Brute I",
          "name" : "mytunes-db"
        "vm_name" : "OpenJDK 64-Bit Server VM",
          "name" : "eth0",
      "name" : "Callisto",
        "name" : "Callisto",
          "name" : "mytunes-db"
        "vm_name" : "OpenJDK 64-Bit Server VM",
          "name" : "eth0",
      "name" : "Vapor",
        "name" : "Vapor",
          "name" : "mytunes-db"
        "vm_name" : "OpenJDK 64-Bit Server VM",
          "name" : "eth0",
      "name" : "Timeslip",
        "name" : "Timeslip",
          "name" : "mytunes-db"
        "vm_name" : "OpenJDK 64-Bit Server VM",
          "name" : "eth0",
      "name" : "Magik",
        "name" : "Magik",
          "name" : "mytunes-db"
        "vm_name" : "OpenJDK 64-Bit Server VM",
          "name" : "eth0",
      "name" : "Brother Voodoo",
        "name" : "Brother Voodoo",
          "name" : "mytunes-db"
        "vm_name" : "OpenJDK 64-Bit Server VM",
          "name" : "eth0",

Analytics

Documentation

The Go Gopher

There is no documentation for this package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL