0

I am using Rest proxy in my application developed in Go and the steps for consuming topics are:

  1. Create a Consumer instance
  2. Subscribe to a topic
  3. get the records
  4. and delete the consumer instance

Now the question I have is I want to have a consumer running always in a goroutine polling for topics, do I need to follow each of the steps always or can I do step1 and 2 first and keep cycling on step 3 and not delete the consumer instance?

AR7
  • 366
  • 1
  • 16
  • Seems fine to me. What's not working with that? Do you have to use the REST proxy, though, instead of a direct go Kafka client? – OneCricketeer Nov 09 '21 at 13:00
  • I want to continuously poll the topic for new messages, that's what is not working. is it possible to create the consumer instance only once and not delete and use it poll the topic? I have to use REST proxy because the client's on prem network does not allow any other port apart from 443 – AR7 Nov 09 '21 at 13:47
  • Yes, you can keep calling `GET /consumers/.../records`. There is no need to delete anything until you are actually done with the consumer instance – OneCricketeer Nov 09 '21 at 15:35
  • Got that part and have implemented it. Is there some implicit delay in the REST Proxy Get Records API? because if a i send consecutive messages, it does not pick them up immediately. – AR7 Nov 10 '21 at 06:39
  • Not sure I understand the question, but the producer process isn't immediate, no – OneCricketeer Nov 10 '21 at 13:36
  • Let me try to elaborate, I have a producer pushing data to a topic and Kafka REST proxy consumer which is polling the topic continuously. What I have observed is that the proxy consumer does not pick up all the messages pushed to the topic, it's like there is a message loss. Not sure why that is happening. – AR7 Nov 10 '21 at 15:28
  • Are you sure it isn't the producer batch not getting completely flushed? – OneCricketeer Nov 10 '21 at 16:56
  • Yes, I am sure on that, because the moment I produce the message, I can see it on the Control center, but the Proxy consumer does not pick the message. – AR7 Nov 11 '21 at 03:48

0 Answers0