0

Helo,

"kafka: error while consuming TestTopic/0: kafka server: Request exceeded the user-specified time limit in the request."

I am getting the above error intermittently while consuming from kafka broker(i.e kafka) OR GCP Pub Sub(i.e gcp_pubsub) pubsub on topic "TestTopic" via Benthos kafka input configuration as below(for input kafka):

input:
  broker:
    inputs:
      - kafka:
          addresses:
            - ${KAFKA_BOOTSTRAP_SERVERS:localhost:9092}
          topics: 
            - TestTopic
          client_id: clientIdTest
          consumer_group: consumerGroupTest
          checkpoint_limit: 2000
          batching:
            count: 1000
            byte_size: 10485760
            period: "1s"
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
mohammad zaheen
  • 1
  • 1
  • 1
    Have you tried using https://www.benthos.dev/docs/components/inputs/gcp_pubsub as input? I’m not very familiar with GCP PubSub, but if it’s fully compatible with the Kafka API, then maybe you need to tweak some settings, like `group.session_timeout`. See the Advanced tab [here](https://www.benthos.dev/docs/components/inputs/kafka). Lastly, try the [kafka_franz](https://www.benthos.dev/docs/components/inputs/kafka_franz) input, which works just like `kafka`, but it uses the [franz-go](https://github.com/twmb/franz-go) library instead of Shopify’s [sarama](https://github.com/Shopify/sarama). – Mihai Todor Jun 30 '22 at 08:24
  • Thanks for your reply Mihai, Yes, it is happening for both inputs that consume from kafka broker(i.e kafka) and GCP Pub Sub(i.e gcp_pubsub). Ok, i will try to increase the group.session_timeout from 10s(default) to some prefeable say 40s and will let you know. Regarding kafka_franz, i saw that it is in experimental state and they said "This component is experimental and therefore subject to change or removal outside of major version releases.", so i donot thing that it is a good idea to use it in production, correct me if i am wrong. – mohammad zaheen Jun 30 '22 at 11:07
  • I’m a contributor to the project and I don’t think there’s any reason to remove kafka_franz. If you still experience issues, then please provide some means of replicating the issue. Also feel free to reach out to the maintainer via https://benthos.dev/community – Mihai Todor Jun 30 '22 at 15:43
  • Hi Mihai, We tweak the group.session_timeout from 10s(default) to 100s but still we are getting the same issue, FYI: we are publishing 350 messages/second for 30 min on TestTopic having 8 partitions, and error is happening on each partitions on exact same time. Any suggestions...............? – mohammad zaheen Jul 01 '22 at 10:41
  • Bummer. Try joining the community as I mentioned above and provide some means of reproducing the issue. – Mihai Todor Jul 01 '22 at 14:11

0 Answers0