2

I'm using Kafka 0.10 REST API. I just used an HTTP object in Java to invoke the Kafka REST API (such as the curl commands). I need to indicate the consumer offset when I consume the message, otherwise it reads from the beginning or the latest, but I could not find the parameter to indicate the offset.

And is there a full REST proxy document to describe every parameter please.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Jack
  • 5,540
  • 13
  • 65
  • 113
  • Just to make sure that I got your question right: You want to change the message offset of a given consumer-group for a given topic and partition? – TobiSH Apr 01 '18 at 20:05
  • yes,I mean how to seek the specific partition and offset, when I start to consume? – Jack Apr 02 '18 at 14:48

2 Answers2

3

Assuming you mean the Confluent Kafka REST Proxy since Apache Kafka does not have a REST API for consuming messages.

Full docs are on the Confluent web site here

https://docs.confluent.io/current/kafka-rest/docs/api.html

Version 0.10 is the version of Apache Kafka but not the version of the Confluent REST Proxy. The Confluent release that includes Apache Kafka 0.10.0 is Confluent 3.0.0. There are many enhancements to the REST Proxy since this release several years ago so suggest that you upgrade to 4.0 or 4.1 and use the v2 REST API.

In the newer versions you can POST a list of offsets like this:

POST /consumers/testgroup/instances/my_consumer/offsets HTTP/1.1
Host: proxy-instance.kafkaproxy.example.com
Content-Type: application/vnd.kafka.v2+json

{
  "offsets": [
    {
      "topic": "test",
      "partition": 0,
      "offset": 20
    },
    {
      "topic": "test",
      "partition": 1,
      "offset": 30
    }
  ]
}
Hans Jespersen
  • 8,024
  • 1
  • 24
  • 31
  • I mean how to seek the specific partition and offset, when I start to consume? – Jack Apr 02 '18 at 14:48
  • btw, could you please recommend a Kafka Rest query tool(like web), so I don't have to do query in my command line, just as: curl..... – Jack Apr 02 '18 at 14:50
  • Commit the offsets you want and then just consume as normal and it will start from the last committed offset – Hans Jespersen Apr 02 '18 at 22:54
  • but we never commit, the team only store the offset in a database, so I have to seek to the last offset and then consume, otherwise, there might duplicated data consumed. – Jack Apr 02 '18 at 22:56
  • Then you are not doing Kafka consumption correctly. You have to commit your offsets in Kafka periodically or your consumer will fall out of the consumer group and reset back to the beginning. – Hans Jespersen Apr 02 '18 at 23:02
1

From https://docs.confluent.io/current/kafka-rest/docs/api.html

GET /topics/(string: topic_name)/partitions/(int: partition_id)/messages?offset=(int)[&count=(int)]

Alexandru Tetelea
  • 19
  • 1