noc19-cs33 Lec 22 Introduction to Kafka

Introduction

This tutorial provides a comprehensive overview of Apache Kafka, as introduced in the lecture from IIT Kanpur's CS33 course. Apache Kafka is a distributed event streaming platform used for building real-time data pipelines and streaming applications. Understanding Kafka is essential for modern data handling and processing tasks in various applications.

Step 1: Understanding the Basics of Kafka

• What is Kafka?

  • A distributed streaming platform that allows for the publishing and subscribing of streams of records.
  • Acts as a messaging system that can handle large volumes of data efficiently.

• Key Components of Kafka:

  • Producers: Applications that publish messages to topics.
  • Consumers: Applications that subscribe to topics to read messages.
  • Topics: Categories to which messages are published.
  • Brokers: Kafka servers that store and serve messages.

• Real-World Application:

  • Suitable for real-time analytics, monitoring, and data integration scenarios.

Step 2: Setting Up Kafka

• Installation Requirements:

  • Java (version 8 or higher)
  • Apache Kafka package (download from the official Kafka website)

• Installation Steps:

  1. Download the Kafka binaries from the official website.
  2. Extract the downloaded files to your desired location.
  3. Start Zookeeper (a service Kafka relies on):

     bin/zookeeper-server-start.sh config/zookeeper.properties
     

  4. Start the Kafka server:

     bin/kafka-server-start.sh config/server.properties
     

Step 3: Creating a Topic

• Definition of a Topic:

  • A topic is a stream of records that can be published and subscribed to.

• Steps to Create a Topic:

  1. Open a new terminal.
  2. Use the following command to create a topic named "test":

     bin/kafka-topics.sh --create --topic test --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
     

  3. Confirm the topic creation by listing all topics:

     bin/kafka-topics.sh --list --bootstrap-server localhost:9092
     

Step 4: Producing Messages to a Topic

• Steps to Send Messages:
  1. Open a terminal for the producer.
  2. Use the following command to start producing messages to the "test" topic:

     bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:9092
     

  3. Type messages in the terminal. Press Enter to send each message.

Step 5: Consuming Messages from a Topic

• Steps to Read Messages:
  1. Open another terminal for the consumer.
  2. Use the following command to read messages from the "test" topic:

     bin/kafka-console-consumer.sh --topic test --from-beginning --bootstrap-server localhost:9092
     

  3. You will see the messages that were produced in real-time.

Conclusion

In this tutorial, we covered the fundamental concepts of Apache Kafka, including its components, setup process, topic management, and how to produce and consume messages. This foundational knowledge is crucial for working with real-time data streaming applications. As a next step, consider exploring Kafka's advanced features such as partitioning, replication, and stream processing to enhance your skills further.