I am using kafka consumer eachBatch: async ({ })
to fetch messages from kafka because our system has to process millions of messages per day and sync the same to Big Query
How can I increase batch size to 100 messages?
Asked
Active
Viewed 1,797 times
0
OneCricketeer
- 179,855
- 19
- 132
- 245
Rayappan A
- 1
- 1
-
Why 100? `max.poll.records` Kafka config defaults to 500 records already – OneCricketeer Jun 02 '22 at 16:37
-
to change max.poll.records in which configuration file? please tell me the location of configuration file. – Rayappan A Jun 03 '22 at 04:09
-
There's no file. It's a consumer client property set around the same place as bootstrap servers and group id, for example – OneCricketeer Jun 03 '22 at 13:56
-
Maybe KafkaJS client doesn't support this though https://github.com/tulios/kafkajs/issues/420 but node-librdkafka would – OneCricketeer Jun 03 '22 at 14:02