Kontext Kontext | Code Snippets & Tips

Subscribe to Multiple Topics in Kafka Consumer

event 2022-08-23 visibility 2,624 comment 0 insights
more_vert
insights Stats

Code description

When fetching records from Kafka bootstrap servers, we can specify one consumer to subscribe to more than one topic. This can be done in the init function by passing a list of topics:

topics = ['kontext-events', 'other-events']
    bootstrap_servers = 'localhost:9092'
    consumer = KafkaConsumer(
        *topics,
        client_id='local-test',
        bootstrap_servers=bootstrap_servers,
        auto_offset_reset='earliest'
    )

Alternatively, we can use subscribe() function to subscribe to multiple topics. We can also regular expressions to match topics via patterns:

subscribe(topics=(), pattern=None, listener=None)

Code snippet

from kafka import KafkaConsumer


def main():
    """
    Kafka example
    """
    topics = ['kontext-events', 'other-events']
    bootstrap_servers = 'localhost:9092'
    consumer = KafkaConsumer(
        client_id='local-test',
        bootstrap_servers=bootstrap_servers,
        auto_offset_reset='earliest'
    )

    consumer.subscribe(topics=topics)

    while True:
        print('polling...')
        records = consumer.poll(timeout_ms=1000)
        for _, consumer_records in records.items():
            # Parse records
            for consumer_record in consumer_records:
                print(str(consumer_record.value.decode('utf-8')))
            continue


if __name__ == '__main__':
    main()
More from Kontext
comment Comments
No comments yet.

Please log in or register to comment.

account_circle Log in person_add Register

Log in with external accounts