How to deploy Kafka and Zookeeper cluster on Kubernetes

This tutorial explains how to Deploy Kafka and zookeeper on Kubernetes.

Prerequisite:

• Setup a multi-node kubernetes cluster up and running with a functioning kubectl.

• The Kubernetes cluster needs to have at least 3 worker nodes with at least 4 GB RAM and 2 Core CPU on each worker node (t2.medium ec2 instances)

• If you don't have a kubernetes cluster, set it up by following the instructions in the link below:

• https://github.com/ashishrpandey/kubernetes-training/blob/master/00installation.md

• or  read here in more details https://www.zekelabs.com/blog/how-to-install-kubernetes-cluster-on-aws-ec2-instances/

Let's start

Log in to the master node. Clone this repository ..

git clone https://github.com/ashishrpandey/kafka-demo
cd kafka-demo/

Deploy zookeeper statefulset, zookeeper service and a headless service

kubectl apply -f zookeeper.yaml 

Deploy kafka statefulset and a headless service

kubectl apply -f kafka.yaml 

Create a topic

Run a pod, move into the pod and run kafka-topics.sh script to create the topic

kubectl run -ti --image=gcr.io/google_containers/kubernetes-kafka:1.0-10.2.1 createtopic --restart=Never --rm -- kafka-topics.sh --create --topic test --zookeeper zk-cs.default.svc.cluster.local:2181 --partitions 1 --replication-factor 3

Run the Consumer process

kubectl run -ti --image=gcr.io/google_containers/kubernetes-kafka:1.0-10.2.1 consume --restart=Never --rm -- kafka-console-consumer.sh --topic test --bootstrap-server kafka-0.kafka-hs.default.svc.cluster.local:9093

This will block the terminal. Keep the terminal open as it is.

In another terminal Run the Producer process

kubectl run -ti --image=gcr.io/google_containers/kubernetes-kafka:1.0-10.2.1 produce --restart=Never --rm -- kafka-console-producer.sh --topic test --broker-list kafka-0.kafka-hs.default.svc.cluster.local:9093,kafka-1.kafka-hs.default.svc.cluster.local:9093,kafka-2.kafka-hs.default.svc.cluster.local:9093 done;

Write some text in the Producer terminal. After you press "enter" after the text, you shall see the same message on the Consumer terminal.

Note:

In this tutorial, we are using non-persistent volume. This volume gets deleted with the deletion of the pod. So you should always run Kafka/Zookeeper statefulSets with the persistentVolumeClaimTemplate (See the commented code in yaml files) with appropriate persistent volumes.

Ashish Graduated from MNNIT Allahabad in Computer Science stream. With ~10 years of experience Corporate experience in companies like Aricent in telecom domain & Guavus in big data domain, Ashish brings a lot of insight in Tech Industry Space. Here goes his expertise: Cloud Technologies - AWS, Azure, Docker, Kubernetes, Terraform, DevOps Tools - Chef, Puppet, Ansible, Jenkins, Programming - Python, C, Java. He worked with Nokia Siemens Network regarding Infrastructure Management. Apart from Tech. industry, he is passionate about the social sector.

---

Keywords : cloud containers deployment DevOps docker kubernetes-training kubernetes training

Recommended Reading

How to build a Multi-Cloud and Hybrid-Cloud Infrastructure using Terraform?

In the era of cloud-wars, the CIOs often have a hard time adopting a single cloud. Putting all their infrastructure into one cloud is a risky proposition. So the best approach is to use the multi-cloud or hybrid cloud strategy. Having multiple cloud provide...

Get Kubernetes Certified

Kubernetes is increasingly becoming the de-facto standard for container-orchestration. It is used for deploying and managing microservice-based applications. It also helps in scaling and maintaining as well. It is open-source software that was initially rel...

What is Helm in Kubernetes?

In this article, we would be discussing what Helm is and how it is used for the simple deployment of applications in the Kubernetes network. Continue reading to learn more about Helm in Kubernetes.

Know more about Terraform

Terraform is a tool made by Hashicorp. It is also used as a tool for cloud-automation. It is an open-source software to implement “Infrastructure as Code (IaC)”. The language used to write the terraform script is known as Hashicorp Configuration Language (H...

How to install Kubernetes Clusters Using Terraform?

Kubernetes is a container orchestration platform that can be used to deploy and manage a containerized applications. Generally, Microservices-based applications are first converted into Docker (or other container runtimes) images and then these microservice...

How to install Kubernetes Cluster on AWS EC2 instances

When someone begins learning Kubernetes, the first challenge is to setup the kubernetes cluster. Most of the online tutorials take help of virtual boxes and minikubes, which are good to begin with but have a lot of limitations. This article will guide you t...

Container is the new process and Kubernetes is the new Unix.

Once a microservice is deployed in a container it shall be scheduled, scaled and managed independently. But when you are talking about hundreds of microservices doing that manually would be inefficient. Welcome Kubernetes, for doing container orchestration ...

What are Big Data, Hadoop & Spark ? What is the relationship among them ?

Big Data is a problem statement & what it means is the size of data under process has grown to 100's of petabytes ( 1 PB = 1000TB ). Yahoo mail generates some 40-50 PB of data every day. Yahoo has to read that 40-50 PB of data & filter out spans. E-commerce...