how to access latest offset of topic in confluent kafka rest proxy to calculate lag

Question

In confluent kafka rest proxy we can get the last committed offset of particular consumer group but how can we get the latest offset of topic to calculate the lag.

Answer 1

You can use Kafka REST Proxy to fetch the latest offset committed for a particular partition. According to the Confluent Docs,

GET /consumers/(string: group_name)/instances/(string: instance)/offsets

Get the last committed offsets for the given partitions (whether the commit happened by this process or another).

Note that this request must be made to the specific REST proxy instance holding the consumer instance.

Parameters:

• group_name (string) -- The name of the consumer group

• instance (string) -- The ID of the consumer instance Request JSON

Array of Objects:

• partitions -- A list of partitions to find the last committed offsets for
• partitions[i].topic (string) -- Name of the topic
• partitions[i].partition (int) -- Partition ID

Response JSON Array of Objects:

• offsets -- A list of committed offsets
• offsets[i].topic (string) -- Name of the topic for which an offset was committed
• offsets[i].partition (int) -- Partition ID for which an offset was committed
• offsets[i].offset (int) -- Committed offset
• offsets[i].metadata (string) -- Metadata for the committed offset

Status Codes:

• 404 Not Found --
• Error code 40402 -- Partition not found
• Error code 40403 -- Consumer instance not found

Example Request:

GET /consumers/testgroup/instances/my_consumer/offsets HTTP/1.1
Host: proxy-instance.kafkaproxy.example.com
Accept: application/vnd.kafka.v2+json, application/vnd.kafka+json, application/json

{
  "partitions": [
    {
      "topic": "test",
      "partition": 0
    },
    {
      "topic": "test",
      "partition": 1
    }

  ]
}

Example Response:

HTTP/1.1 200 OK
Content-Type: application/vnd.kafka.v2+json

{"offsets":
 [
  {
    "topic": "test",
    "partition": 0,
    "offset": 21,
    "metadata":""
  },
  {
    "topic": "test",
    "partition": 1,
    "offset": 31,
    "metadata":""
  }
 ]
}

Answer 2

Looks like there is an early access feature for this: https://docs.confluent.io/platform/current/kafka-rest/api.html#get--clusters-cluster_id-consumer-groups-consumer_group_id-lags