Apache Kafka is a distributed streaming platform that can act as a message broker, as the heart of a stream processing pipeline, or even as the backbone of an enterprise data synchronization system. Kafka is not only a highly-available and fault-tolerant system; it also handles vastly higher throughput compared to other message brokers such as RabbitMQ or ActiveMQ.
In this tutorial, you will install Apache Kafka, run three brokers in a cluster, and learn how to produce and consume messages from your cluster. This tutorial assumes that you have no existing Kafka or ZooKeeper installation.
To install Apache Kafka on Mac, Java is the only prerequisite. First we shall look into the installation steps of Java and then we shall setup Apache Kafka and run it on the Mac.
Prerequisites
To complete this tutorial, you will need:
Note: Java 7 support was dropped in 2.0.0. Java 11 support was added in 2.1.0.
Installation
Download Apache Kafka and its related binaries from the Apache Kafka website. At the time of this article, the latest version is Apache Kafka 2.1.1. After downloading from the previous link, extract the
.tgz file from the location it was downloaded to.
System Architecture
Let's take a look at the architecture of a simple Apache Kafka setup.
Kafka utilizes ZooKeeper to manage and coordinate brokers within a cluster. Producers and consumers are the main components that interact with Kafka, which we'll take a look at once we have a running Kafka broker. In this tutorial, we'll be running three Kafka brokers and one ZooKeeper node.
The above diagram shows the architecture of the systems and tools used in this tutorial. It helps demonstrate how Kafka brokers utilize ZooKeeper, which components the command line tools we'll be using interact with, and shows the ports of the running services.
Starting Zookeeper
ZooKeeper is a centralized service that is used to maintain naming and configuration data as well as to provide flexible and robust synchronization within distributed systems. Kafka requires ZooKeeper, so we must start an instance of ZooKeeper before we start Kafka.
Conveniently, the download for Apache Kafka includes an easy way to run a ZooKeeper instance. Inside of the
bin directory, there is a file named zookeeper-server-start.sh . To start ZooKeeper, run the following command from the root directory of your download:
In your terminal, ZooKeeper logs will start flowing and you will shortly see a line that states ZooKeeper is running on port
2181 . This is ZooKeeper's default port, and can be changed in config/zookeeper.properties .
Note: The default directory where ZooKeeper stores its state is set to
Apache Kafka Download For Windows
Lastly, open a new terminal window and let ZooKeeper continue running in your original terminal. Ensure you
cd to the root directory of your extracted Kafka download.
Setting Up A Kafka Cluster
The official Kafka quick start guide only runs one broker – that's not really a distributed system or a cluster; so we're going to run three brokers! :)
Let's examine the configuration file for a Kafka broker located at
config/server.properties . You can view the configuration file from your new terminal window by running:
Download mendeley software for mac. There's quite a bit of configuration, but the main properties we care about are the following:
All three of these configuration properties must be unique per broker. By default, you can see the default broker id is
0 and the default Kafka port is 9092 . Since we're going to start 3 brokers, let's copy this file for each broker and leave server.properties as-is for reference. We can do this by running:
Next, we need to modify the properties listed above to be unique per broker. You'll want to ensure you uncomment the
listeners property. Modify the files using your favorite text editor, or via a CLI program such as vim . Make sure to only modify the lines below, and not to replace the whole file with them!
server-1.properties
Mac Install Kafka
server-2.properties
server-3.properties
Apache Kafka Install Windows
Yay! We now have a configuration file for each broker. Each broker has a unique id, listens on a unique port, and stores data in a unique location. Download netflix movies on mac reddit.
Note: As with ZooKeeper, the data is stored in the
Starting Kafka
In addition to your current terminal, open two more terminal windows and
cd to your Kafka download directory. You should have four terminals open at this point; one running ZooKeeper and three for running Kafka.
To start Kafka, you'll want to run the following commands, with each one in a separate terminal:
You'll start to see logs in each terminal for the brokers you started. If you look at your ZooKeeper terminal, you'll also see logs from the brokers connecting to ZooKeeper. Each terminal should end with a line similar to:
Congrats! You now have a Kafka cluster running, with a unique broker exposed on ports
9091 , 9092 , and 9093 .
Creating A Topic
Now that we have a Kafka cluster running, let's send some messages! To do this, we must first create a topic. Kafka includes some command line tools to do this, located in the
bin directory. Open a new terminal window and cd to the Kafka download directory.
Let's create a topic named
test . We can do this by utilizing the kafka-topics.sh script in the bin directory.
Let's analyze the arguments we're passing the script:
In the command above, we create a single partition topic. We also set the replication factor to
3 . This means that data will be replicated (copied for redundancy) to all of our brokers.
Note: The max replication factor for a topic is the number of brokers you have running. In this case, we have max replication factor of 3.
We can now
describe the topic to gain insight into our newly created topic:
This will output something similar to: https://Magic-Music-Editor-Serial-Key.peatix.com/.
This explains that our topic
test has one partition, a replication factor of three, and no non-default configurations set. It also shows for our one partition, partition 0 , that the leader is broker 2 and that we have 3 in-sync replicas. Your leader may be different than broker 2 , but you should have 3 in-sync replicas.
To learn more about what partitions, replicas, and in-sync replicas mean, go check out and read my post Apache Kafka: An Introduction.
Producing Messages
Now that we have a Kafka topic, let's send some messages to it! We can do this using the
kafka-console-producer.sh script in the bin directory. This is a handy tool for producing messages from the command line.
Run the console producer with the following command:
We pass the list of Kafka brokers with the
--broker-list argument and the name of the topic to produce to with the --topic argument. You should now have a terminal line starting with > . From here, you can type a message and hit enter to send it to Kafka. For example:
Once you've sent some messages, exit out of the console producer by using
cmd + c or ctrl + c .
Consuming Messages
We've successfully sent some messages to our Kafka topic, so the last thing we need to do is read those messages. Download headers only in mac mail. We can do this by using the
kafka-console-consumer.sh script in the bin directory. This is a handy tool for consuming messages from the command line.
Run the console consumer against our topic with the following command: https://yjxcuae.weebly.com/download-game-of-thrones-season-6-e-9.html.
We set the
--bootstrap-server argument to a comma-separated list of our brokers; this can be one or all of the brokers. I typically use all brokers for consistency. We also set the argument --topic to our topic name and pass the --from-beginning flag to read all messages in the topic. If you don't pass --from-beginning , you'll only see messages that have been produced since starting the consumer.
You should see the messages sent earlier appear in the output:
To exit the consumer, use
cmd + c or ctrl + c .
Conclusion
Congrats! You've successfully started a local Kafka cluster, created a topic, sent messages to it with a console producer, and read messages from it with a console consumer. For fun, you can start the console producer and console consumer in separate terminal windows and produce some more messages. You'd then be able to see messages get consumed and printed in real time! Eset remote admin. Sweet!
You can stop the Kafka brokers and ZooKeeper node by using
cmd + c or ctrl + c in their respective terminal windows. I hope this tutorial helped you in getting a local Kafka cluster set up, and now you should be ready to continue on in your Kafka journey!
Comments are closed.
|
AuthorWrite something about yourself. No need to be fancy, just an overview. Archives
December 2020
Categories |