Code python

Python - Read Messages in Kafka Topic

visibility 1,757 comment 0 access_time 10 months ago language English

from kafka import KafkaConsumer
from kafka.structs import TopicPartition

topic = 'kontext-kafka-5'
bootstrap_servers = 'localhost:9092'
consumer = KafkaConsumer(
    topic, bootstrap_servers=bootstrap_servers, auto_offset_reset='earliest')

partitions = consumer.partitions_for_topic(topic)
for p in partitions:
    topic_partition = TopicPartition(topic, p)
    # Seek offset 0
    consumer.seek(partition=topic_partition, offset=0)
    for msg in consumer:
        print(msg.value.decode("utf-8"))

copyright This page is subject to Site terms.

comment Comments

No comments yet.

Please log in or register to comment.

account_circle Log in person_add Register

Log in with external accounts

thumb_up 1 fork_right

more_vert

Code description

This code snippet provides example of reading messages from Kafka topics. In common practices, one Kafka consumer is used to read from one partition for a topic; if there are multiple partitions for the same topic, each consumer can run in different servers or containers. Those consumers for different partitions are also configured within one consumer group.

In this example, all partitions are assigned to the same consumer. This code snippet utilize Python package kafka-python. It can be installed via the following command if you have pip installed:

pip install kafka-python

Or we can also use built-in pip in Python 3:

python -m pip install kafka-python

References

Apache Kafka Python Producer and Consumer Clients Introduction

Install and Run Kafka 3.2.0 On WSL

Install and Run Kafka 2.6.0 On Windows 10

python kafka