Running a highly available Kafka cluster on Kubernetes
Contents
Apache Kafka is the most popular distributed messaging publish-subscribe system available today. While Kafka is very powerful, it is equally complex and requires a highly available and robust platform to run. In a world where microservices are prevalent and most companies are adopting distributed computing, it is still very advantageous to use Kafka as a core messaging system.
If you run your microservices in a Kubernetes cluster, it makes sense to run a Kafka cluster in Kubernetes to take advantage of its built-in resiliency and high availability, and we can easily interact with Kafka Pods within the cluster using the built-in Kubernetes service discovery.
Here we will introduce how to build a distributed Kafka cluster on Kubernetes, here we will use Helm Chart and StatefulSet for deployment, of course if you want to dynamically generate persistent data volumes, you also need to configure a StorageClass resource in advance, for example based on Ceph RBD, if you don’t have dynamic volumes configured in your cluster, you need to create 3 unbound PVs for data persistence in advance.
The current Kafka deployment on Kubernetes based on Helm’s official repository chartincubator/kafka
, using the image confluentinc/cp-kafka:5.0.1
, that is, the deployment is the Kafka version provided by Confluent, Confluent Platform Kafka (CP Kafka for short) provides some advanced features that Apache Kafka does not, such as cross-data center backup, Schema registry, and cluster monitoring tools.
Installation
Installing with Helm Chart requires Helm to be installed, of course, and can be done directly with the latest version of Helm v3 as follows.
|
|
Then add Kafka’s Chart repository.
|
|
Next we can configure the Values file to be installed, either directly using the default values.yaml file, which can then be used to customize it, for example by specifying our own StorageClass.
|
|
Here I have used the default directly for installation.
|
|
If you don’t configure StorageClass or available PVs, kafka’s Pod will be in Pending state when you install it, so be sure to configure the data volumes in advance.
Normally, Kafka will be installed successfully in a few moments.
|
|
By default, 3 ZK Pods and 3 Kafka Pods are installed, which ensures high availability of the application, or you can see the information about the persistent volumes I configured.
|
|
If we configure a default StorageClass, it will dynamically request persistent volumes. If your cluster does not have dynamic volumes enabled, you can modify values.yaml to use static volumes.
Then look at the corresponding Service object.
|
|
You can see that there is a zookeeper service called kafka-zookeeper
and a Kafka service called kafka
. For the management of the Kafka cluster, we will interact with the kafka-zookeeper
service, and for the sending and receiving of cluster messages, we will use the kafka
service.
Client testing
Now that the Kafka cluster is set up, let’s install a Kafka client that will help us generate and retrieve topics messages.
Create the client directly with the following command.
|
|
Once the client pod is created successfully we can start doing some simple tests. First let’s create a topic named test1
with a partition and replication factor '1'
.
|
|
Then create a producer that will post messages to this topic.
|
|
Then retype a terminal page that allows us to open a consumer session so we can see the message we sent.
|
|
Now we send the message in the producer’s window and see the corresponding message in the consumer session window above:
Here the Kafka cluster is proven to work properly. For example, it should be noted that we did not persist the zk cluster, so if it is a production environment, you must remember to do the data persistence and customize it according to your needs in the values.yaml file, but of course for production environments it is recommended to use an Operator to build Kafka clusters, such as strimzi-kafka-operator
.