How to install Apache Kafka on Rocky Linux 8 or AlmaLinux

Here are the steps to install Apache Kafka on Rocky Linux or AlmaLinux 8 server, of course, using command terminal. 

Apache Kafka is open-source software that enables the storage and processing of data streams via a distributed streaming platform. In simple words, Apache Kafka is an event streaming platform that acts as a messaging system between the sender and the recipient with high fault tolerance and scalability capabilities because it is based on a distributed architecture that is optimized for the same.

Well, this system was originally developed by LinkedIn as a message queue, however, being a project of Apache Software Foundation, it is open source and a powerful streaming platform with a variety of functions. It offers interfaces to write data to Kafka clusters, to read data, or to import and export data to and from third-party systems. Due to low latency and high throughput, it can process real-time streams easily.

The interfaces also make it possible for users to load data streams from third-party systems or to export them to these systems.  This makes Apache Kafka suitable for large amounts of data and applications in the big data environment.

It can be used for a wide range of applications such as tracking website activity in real-time, monitoring distributed applications, aggregating log files from different sources, synchronization of data in distributed systems, training of models in real-time helps in machine learning, and more…

Apache Kafka offers these four main interfaces (APIs – Application Programming Interfaces). Know more about each API at the official documentation page:

  • Admin API
  • Producer API
  • Consumer API
  • Streams API
  • Connect API

What do you need to follow this tutorial:

  • Rocky or AlmaLinux 8 or any other RHEL based server, if possible then clean one.
  • A user with sudo access.

 

Steps to install Apache Kafka on Rocky Linux 8

The given guide is applicable for all the RHEL 8 based Linux systems including CentOS 8 and Oracle Linux 8 to install Kafka if required.

1. Update system

Well, before moving further run the system update command to ensure all the installed packages are up to date. For that run the below command, this will also refresh the repository cache.

sudo dnf update

 

2. Install Java

Apache Kafka needs Java to run, hence first we need to install that on our local environment and it must be equal or greater than Java 8. Well, we don’t need to add any third repository because the package to get JAVA is already there on the system base repo, hence, let’s use the given command.

For Java 11

sudo dnf install java-11-openjdk

Well, for the latest version such as 16 use the below commands:

sudo dnf install epel-release
sudo dnf install java-latest-openjdk

 

3. Download the latest Apache Kafka on Rocky Linux 8 or Almalinux

Apache Kafka is available as a tarball file on the official website. Hence, Go to the official website and download the latest version. You can also copy any mirror link and use the wget command to download using the command line, as we have done here:

sudo dnf install wget nano
wget https://dlcdn.apache.org/kafka/3.0.0/kafka_2.13-3.0.0.tgz

Extract the downloaded file 

tar -xf kafka_*tgz

To see the file:

ls

Move it to /usr/local/  just to ensure we won’t delete the Kafka folder accidentally.

sudo mv kafka_2.13-3.0.0/ /usr/local/kafka

 

4. Create System Service for Zookeeper and Kafka

Although just for the test you can run both Zookeeper and Kafka service script directly, manually, however, for the production server we have to run it in the background. Hence, create systemd units for both the scripts.

Create systems file Zookeeper

As per the official website in the future, Kafka will not need Zookeeper, however, while writing this article we need it. So, create a service file first for Zookeeper.

sudo nano /etc/systemd/system/zookeeper.service

Copy-Paste the below-given lines:

[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/usr/bin/bash /usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
ExecStop=/usr/bin/bash /usr/local/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Save and close the file by pressing Ctr+O, hit the Enter key, and then exit it using Ctrl+X.

Now, create Kafka systemd File

sudo nano /etc/systemd/system/kafka.service

Paste the following lines. Note – Change the Java_Home, in case you are using some other version. To find it you can use the command –  sudo find /usr/ -name *jdk

[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service

[Service]
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/jre-11-openjdk"
ExecStart=/usr/bin/bash /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
ExecStop=/usr/bin/bash /usr/local/kafka/bin/kafka-server-stop.sh

[Install]
WantedBy=multi-user.target

Save the file with Ctrl+O, press Enter key, and then use Ctrl+X to exit the same.

Reload daemon

To reflect the above-made changes into the system and use the service files, reload the system daemon once.

sudo systemctl daemon-reload

 

5. Start Zookeeper and Kafka Server on Rocky Linux

Now, let’s start and enable both server services to make sure they will also get active even after the system reboot.

sudo systemctl start zookeeper
sudo systemctl start kafka
sudo systemctl enable zookeeper
sudo systemctl enable kafka

Check services status:

sudo systemctl status zookeeper
Apache Zookeper Server service start

sudo systemctl status kafka

Apadceh Kafka Server service start

 

6. Create Test Topics on Kafka – Rocky or AlmaLinux

Kafka allows us to read, write, store, and process events across the various machines, however, to store these events we need someplace or folder and that called “Topics“. So on your server terminal create at least one topic using the following command, using the same later you can create as many Topics as you want.

Let’s say our first Topic name is – testevent. So to create the same run:

Go to your Kafka directory.

cd /usr/local/kafka/

And use the Topics script:

bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic testevent

Create Topics

After creating as many as topics you want, to list them all we can use the following command:

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

 

7. Write some event using Kafka Producer and read with consumer

Kafka offers two APIs- Producer and Consumer,  for both it offers a command-line client. The producer is responsible for creating events and the Consumer uses them to display or reads the data generated by the Producer.

Open Two terminal tabs or sessions to understand the event generator and reader setup in real-time.

#On one first terminal:

To test let’s create some events using the Producer script:

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic testevent

Type some text that you want to stream and display on the consumer end.

#On the other terminal

Run, the below-given command along with the topic name to check the messages or generated event data in real-time:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic testevent --from-beginning

Send Apache producer message to Consumer

 

Other Articles:

How to Install LAMP on Rocky Linux 8 Server
How to install phpMyAdmin on Rocky Linux 8 with Apache
Install Apache, MySQL, and PHP on Rocky or AlmaLinux 8
How To Install InfluxDB on AlmaLinux or Rocky 8

 

 

 

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.