How to acknowledge consume message in kafka using php-rdkafka?

Question

I am using php-rdkafka as php kafka client. I successfully product my test message by using test group.and consume the message by using below code,

$kafkaConsumer = new RdKafka\Consumer();
$kafkaConsumer->addBrokers("127.0.0.1:9292");
$topic = $kafkaConsumer->newTopic("test");
$topic->consumeStart(0, RD_KAFKA_OFFSET_BEGINNING);

while (true) {
    $msg = $topic->consume(0, 1000);
    if($msg){
    if ($msg->err) {
        echo $msg->errstr(), "\n";
        break;
    } else {
        echo $msg->payload, "\n";
    }
  }
}

But when I try to again set message in test group and trying to consume message for test group then I am getting old message as well as new message. So I just want to how can I acknowledge old message so I can get only new message not old one ? Can someone put some shine on this ?

My kafka version is 0.11.0.1

Use RD_KAFKA_OFFSET_STORED in place of RD_KAFKA_OFFSET_BEGINNING to start consuming from committed offset. — Nipun Talukdar, Oct 15 '17 at 20:05
I tried that but I am getting error like `Invalid argument or configuration` @NipunTalukdar — Keyur Shah, Oct 16 '17 at 07:05

Mickael Maison · Accepted Answer · 2017-10-22T15:30:35.787

6

The method to acknowledge consumed messages in Kafka is to commit its offset. That way when restarting your consumer it can retrieve the last committed offset and restart where it left off.

As suggested in the comments, you need to use RD_KAFKA_OFFSET_STORED to instruct the consumer to retrieve the stored offset.

But you also need to provide a group name by setting the group.id config:

<?php

$conf = new RdKafka\Conf();

// Set the group id. This is required when storing offsets on the broker
$conf->set('group.id', 'myConsumerGroup');

$rk = new RdKafka\Consumer($conf);
$rk->addBrokers("127.0.0.1:9292");

$topicConf = new RdKafka\TopicConf();
$topicConf->set('auto.commit.interval.ms', 100);

// Set where to start consuming messages when there is no initial offset in
// offset store or the desired offset is out of range.
// 'smallest': start from the beginning
$topicConf->set('auto.offset.reset', 'smallest');

$topic = $rk->newTopic("test", $topicConf);

// Start consuming partition 0
$topic->consumeStart(0, RD_KAFKA_OFFSET_STORED);

while (true) {
    $message = $topic->consume(0, 120*10000);
    switch ($message->err) {
        case RD_KAFKA_RESP_ERR_NO_ERROR:
            var_dump($message);
            break;
        case RD_KAFKA_RESP_ERR__PARTITION_EOF:
            echo "No more messages; will wait for more\n";
            break;
        case RD_KAFKA_RESP_ERR__TIMED_OUT:
            echo "Timed out\n";
            break;
        default:
            throw new \Exception($message->errstr(), $message->err);
            break;
    }
}
?>

edited Oct 22 '17 at 15:30

answered Oct 22 '17 at 14:15

Mickael Maison

25,067
7
71
68

Thanks for the answer , what is the value of `consumer-group-name` ? do I also need to set this in producer ? – Keyur Shah Oct 22 '17 at 14:23
I am getting the error like `No such configuration property: "mygroup"` . Could you please give me example ? – Keyur Shah Oct 22 '17 at 14:28
1

You need to keep `group.id`, it's the configuration name ! But you can put any string you like instead of `consumer-group-name`. But in order to be able to retrieve your offset, you need to make sure you restart your consumer with the same value. – Mickael Maison Oct 22 '17 at 14:52
In producer also ? I set the both the value but getting same error. Sorry If it is minor I am newbie in kafka so.. – Keyur Shah Oct 22 '17 at 15:02
You only need to set the `group.id` config for the consumer. I've updated my answer to include the full code sample. This is based on the low lelvel consumer php-rdkafka example: https://arnaud-lb.github.io/php-rdkafka/phpdoc/rdkafka.examples-low-level-consumer-basic.html – Mickael Maison Oct 22 '17 at 15:31
From which place I can set the `group.id` ? at code level or another place ? and I am getting the error like `Fatal error: Uncaught RdKafka\Exception: No such configuration property: "mygroup" in /var/www/html//consumer.php on line 8` – Keyur Shah Oct 22 '17 at 19:37
This line: `$conf->set('group.id', 'myConsumerGroup');` sets the `group.id`. You can change `myConsumerGroup` to any string you like. – Mickael Maison Oct 22 '17 at 20:20

How to acknowledge consume message in kafka using php-rdkafka?

1 Answers1