Deploying an Elasticsearch stack on a Kubernetes cluster

If Logstash is used to receive log input from Filebeat centrally, it is easy to cause performance bottleneck due to single machine; if Kafka is used to receive log input from Filebeat, the timeliness of logs is not guaranteed. Here, the logs collected by Filebeat are directly output to Elasticsearch.

1. Preparation

• Node planning

  Instead of distinguishing master, data, and client nodes, three nodes of a cluster are directly multiplexed here. If it is a very large Elasticsearch cluster, it is recommended to distinguish between control plane and data plane to improve stability.

• Ensure the cluster has available default storage

  1 2 3 4

  kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE local (default) openebs.io/local Delete WaitForFirstConsumer false 411d

• Generate secret key

  1 2 3 4 5 6 7 8 9 10 11 12

  export ELASTICSEARCH_IMAGE=k8simage/elasticsearch:7.17.5 docker rm -f elastic-helm-charts-certs || true rm -f elastic-certificates.p12 elastic-certificate.pem elastic-certificate.crt elastic-stack-ca.p12 || true docker run --name elastic-helm-charts-certs -i -w /tmp \ $ELASTICSEARCH_IMAGE \ /bin/sh -c " \ elasticsearch-certutil ca --out /tmp/elastic-stack-ca.p12 --pass '' && \ elasticsearch-certutil cert --name security-master --dns security-master --ca /tmp/elastic-stack-ca.p12 --pass '' --ca-pass '' --out /tmp/elastic-certificates.p12" && \ docker cp elastic-helm-charts-certs:/tmp/elastic-certificates.p12 ./ && \ docker rm -f elastic-helm-charts-certs && \ openssl pkcs12 -nodes -passin pass:'' -in elastic-certificates.p12 -out elastic-certificate.pem && \ openssl x509 -outform der -in elastic-certificate.pem -out elastic-certificate.crt

• Create Kubernetes Secret credentials

  1 2 3 4

  kubectl create ns elastic kubectl -n elastic create secret generic elastic-certificates --from-file=elastic-certificates.p12 kubectl -n elastic create secret generic elastic-certificate-pem --from-file=elastic-certificate.pem kubectl -n elastic create secret generic elastic-certificate-crt --from-file=elastic-certificate.crt

• Create login user and password

  1 2 3 4 5 6 7

  ELASTIC_USER=elastic ELASTIC_PASSWORD=XXXXXX kubectl create secret generic elastic-credentials -n elastic \ --from-literal=username="${ELASTIC_USER}" \ --from-literal=password="${ELASTIC_PASSWORD}" \ --dry-run=client -o yaml | kubectl apply -f -

• Add Elastic Helm Charts repository

  1	helm repo add elastic https://helm.elastic.co

2. Install Elasticsearch

• Create a values.yaml file and configure the installation parameters

  1	vim es-master-values.yaml

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39

  clusterName: "elasticsearch" nodeGroup: "master" roles: master: "true" ingest: "true" data: "true" imageTag: 7.17.5 image: k8simage/elasticsearch volumeClaimTemplate: accessModes: ["ReadWriteOnce"] resources: requests: storage: 200Gi replicas: 3 esConfig: elasticsearch.yml: | xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12 extraEnvs: - name: ELASTIC_PASSWORD valueFrom: secretKeyRef: name: elastic-credentials key: password secretMounts: - name: elastic-certificates secretName: elastic-certificates path: /usr/share/elasticsearch/config/certs antiAffinity: "soft"

• Start installing components

  1	helm install elasticsearch-master elastic/elasticsearch -n elastic --values es-master-values.yaml --set service.type=NodePort

3. Install Kibana

• Create a values.yaml file and configure the installation parameters

  1	vim kibana-values.yaml

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

  elasticsearchHosts: "http://elasticsearch-master.elastic.svc:9200" imageTag: 7.17.5 image: k8simage/kibana replicas: 1 extraEnvs: - name: "NODE_OPTIONS" value: "--max-old-space-size=1800" - name: "ELASTICSEARCH_USERNAME" valueFrom: secretKeyRef: name: elastic-credentials key: username - name: "ELASTICSEARCH_PASSWORD" valueFrom: secretKeyRef: name: elastic-credentials key: password

• Start installing components

  1	helm install kibana elastic/kibana -n elastic --values kibana-values.yaml --set service.type=NodePort

4. Install Filebeat

• Create a values.yaml file and configure the installation parameters

  1	vim filebeat-values.yaml

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

  image: k8simage/filebeat daemonset: extraEnvs: - name: "ELASTICSEARCH_USERNAME" valueFrom: secretKeyRef: name: elastic-credentials key: username - name: "ELASTICSEARCH_PASSWORD" valueFrom: secretKeyRef: name: elastic-credentials key: password filebeatConfig: filebeat.yml: | filebeat.inputs: - type: filestream id: varlog paths: - /var/log/*.log - type: container enabled: true paths: - /var/lib/docker/containers/*/*.log output.elasticsearch: host: "${NODE_NAME}" hosts: '["http://${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}"]' username: "${ELASTICSEARCH_USERNAME}" password: "${ELASTICSEARCH_PASSWORD}" protocol: http

  The filebeat.inputs in filebeat.yml determines which logs filebeat will collect.

• Start installing components

  1	helm install filebeat elastic/filebeat -n elastic --values filebeat-values.yaml

5. Check the service verification function

• Ensure all services are started properly

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

kubectl -n elastic get pod

NAME                             READY   STATUS    RESTARTS   AGE
elasticsearch-master-0           1/1     Running   0          10h
elasticsearch-master-1           1/1     Running   0          10h
elasticsearch-master-2           1/1     Running   0          10h
filebeat-filebeat-4l9jh          1/1     Running   0          9h
filebeat-filebeat-4m7r6          1/1     Running   0          9h
filebeat-filebeat-6x9pd          1/1     Running   0          9h
kibana-kibana-7976c5dc76-xfswq   1/1     Running   0          10h

• View the accessed ports

  1 2 3 4 5 6

  kubectl -n elastic get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE elasticsearch-master NodePort 10.233.35.209 <none> 9200:30093/TCP,9300:31743/TCP 10h elasticsearch-master-headless ClusterIP None <none> 9200/TCP,9300/TCP 10h kibana-kibana NodePort 10.233.53.16 <none> 5601:30454/TCP 11h

• Accessing Elasticsearch

  Elasticsearch can be accessed via HOST_IP:30093, as shown below. At this point, you need to enter the account password, elastic:XXXXXXX, which is the account information set in the preparation.

  

Once the credentials are entered correctly, Elasticsearch will return information about the cluster.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

{
  "name": "elasticsearch-master-1",
  "cluster_name": "elasticsearch",
  "cluster_uuid": "D6ugYWeTQSqi3kUAK-B36w",
  "version": {
    "number": "7.17.5",
    "build_flavor": "default",
    "build_type": "docker",
    "build_hash": "8d61b4f7ddf931f219e3745f295ed2bbc50c8e84",
    "build_date": "2022-06-23T21:57:28.736740635Z",
    "build_snapshot": false,
    "lucene_version": "8.11.1",
    "minimum_wire_compatibility_version": "6.8.0",
    "minimum_index_compatibility_version": "6.0.0-beta1"
  },
  "tagline": "You Know, for Search"
}

• Accessing Kibana

Kibana can be accessed via HOST_IP:30454, as shown below, with the same account as Elasticsearch.

6. Clean up the environment

• Uninstall Elasticsearch

  1	helm uninstall elasticsearch-master -n elastic

• Uninstall Kibana

  1	helm uninstall kibana -n elastic

• Uninstall Filebeat

  1	helm uninstall filebeat -n elastic

• Clear Storage

  1	kubectl -n elastic delete pvc --all

• Clean up environment variables

  1 2	unset ELASTIC_USER unset ELASTIC_PASSWORD