from kafka import KafkaConsumer from kafka.structs import TopicPartition topic = 'kontext-kafka-5' bootstrap_servers = 'localhost:9092' consumer = KafkaConsumer( topic, bootstrap_servers=bootstrap_servers, auto_offset_reset='earliest') partitions = consumer.partitions_for_topic(topic) for p in partitions: topic_partition = TopicPartition(topic, p) # Seek offset 0 consumer.seek(partition=topic_partition, offset=0) for msg in consumer: print(msg.value.decode("utf-8"))
codePython - Read Messages in Kafka Topic
This code snippet provides example of reading messages from Kafka topics. In common practices, one Kafka consumer is used to read from one partition for a topic; if there are multiple partitions for the same topic, each consumer can run in different servers or containers. Those consumers for different partitions are also configured within one consumer group.
In this example, all partitions are assigned to the same consumer. This code snippet utilize Python package kafka-python. It can be installed via the following command if you have
pip install kafka-python
Or we can also use built-in pip in Python 3:
python -m pip install kafka-python