0

in librdkafka,

The data has been put in the transmission queue via following function.

virtual ErrorCode   produce (Topic *topic, int32_t partition, int msgflags, void *payload, size_t len, const void *key, size_t key_len, void *msg_opaque)

The result of the transfer can be found in the dr_cb() registered as a callback through the poll(0).

class ExampleDeliveryReportCb : public RdKafka::DeliveryReportCb {
 public:
  void dr_cb (RdKafka::Message &message) {
    // I want to know the order of produce.
  }
};

If I produce 10 times, how can I know if the third produce has been successful? I can see the actual messages or the number of messages left in the transmission queue, but the result of third produce is unknown.

I want to synchronize the sequence number of the message produced and the sequence number of the message reported to be complete via dr_cb. What should I do?

김현우
  • 23
  • 7

1 Answers1

2

Ordering is only a thing per partition in Kafka, if you produce to multiple partitions there is no ordering inbetween those partitions.

As for messages produced to the same partition they will be produced in their original order, unless there is an error which warrants a retry, in which case reordering is possible.

You can either set max.in.flight=1 to avoid reordering, which unfortunately also decreases throughput and increases latency, or use the Idempotent Producer with enable.idempotence=true to get guaranteed ordered delivery at very little throughput cost.

You can use the message opaque to attach a pointer to a message in produce() which you'll get back as msg_opaque in the delivery report, allowing you to map delivery reports to the original object you produced.

Edenhill
  • 2,897
  • 22
  • 35
  • thanks! and There are a few additional questions. I use version 0.11.4. 1. Is the socket.max.fails option not related to TCP 3-way handshake? If I set this value to 10 and produce to a broker that does not have the kafka service enabled, it will take more than 10 times to send a syn packet and receive an rst packet. – 김현우 Nov 19 '18 at 23:46
  • 2. Is the request.required.acks option and message timeout irrelevant? If I set this value to 0 and then produce, the document states that no ack is received from the broker. That is, outq_len () should be decremented regardless of the success of the transmission, but outq_len () does not decrease, but rather causes a timeout. What am I missing? – 김현우 Nov 19 '18 at 23:50
  • socket.max.fails operates on the application layer, namely it is a threshold of the number of failed Kafka protocol requests before the connection is torn down. Leave it at its default value. – Edenhill Nov 20 '18 at 09:23
  • If a connection to the broker can't be made the produced messages will eventually time out (message.timeout.ms). The producer can't even attempt to produce a message if the broker connection is down, regardless if acks=0. – Edenhill Nov 20 '18 at 09:24