9

One way of achieving it could be by setting the properties parameter
max.in.flight.requests.per.connection = 1.

But I want to know if there is an even direct or alternate way of sending messages synchronously in kafka, something like producer.syncSend(...).

Gerardo Cauich
  • 559
  • 5
  • 20
  • Can you explain why you want that? If this is about message-ordering guarantees, this may be a bit more complicated (and sending them synchronously does not really change things there). – Thilo Aug 06 '17 at 06:47
  • Messages sent to the same partition in the same topic by the same producer will retain that order. There are no ordering guarantees across multiple partitions, topics or producers. You have to arrange the messages on the consumer-side in application code if you need that (for example by looking at message timestamps). – Thilo Aug 07 '17 at 01:37
  • @Thilo, this leads me to another question. Are batches per partition or per topic or per producer? Or some combination of these. ? –  Aug 07 '17 at 03:12

6 Answers6

14

The producer API returns a Future from send. You can call Future#get to block until the sending has completed.

See this example from the Javadocs:

If you want to simulate a simple blocking call you can call the get() method immediately:

 byte[] key = "key".getBytes();
 byte[] value = "value".getBytes();
 ProducerRecord<byte[],byte[]> record = 
     new ProducerRecord<byte[],byte[]>("my-topic", key, value)
 producer.send(record).get();
Thilo
  • 257,207
  • 101
  • 511
  • 656
  • 3
    Calling get() as suggested in this answer will only tell you that the message has been fully sent. This does nothing in terms of providing a solution to simulating a synchronous call. A real solution involving kafka is going to involve a completion message of some kind being consumed by a completion consumer which would free the blocked request thread. – Rodney P. Barbati May 09 '18 at 21:04
  • 3
    If by "synchronous call" you mean that you want to wait until all consumers you are interested in have completed whatever action you want them to take as a result of the message, then yes, the answer does not provide a "synchronous RPC mechanism". But it does address OP's request for a `producer.sendSync`. – Thilo May 10 '18 at 02:24
  • @Thilo I think we would need to flush the message also just after the get() method to ensure its delivery to the broker. Please correct me if I'm wrong. – Ankit Singodia Dec 21 '19 at 05:29
  • 1
    @AnkitSingodia Blocking until the Future returned from the producer completes will (by default) include delivery confirmation from the broker. This can be configured via the `acks` producer parameter. You can set it to 0 to disable confirmations, or to a value higher than 1 to also confirm succesful replication to a quota. – Thilo Jan 01 '20 at 13:12
6

As Thilo suggested, you can call Future#get to block until the sending has completed. However you might have some performance issue, since the producer starts sending when the producer queue has batch.size elements, when the buffer of size buffer.memory is full or after max.block.ms milliseconds.

If you have a limited number of threads pushing to kafka, you will have to wait max.block.ms each time for your message to be sent. So in some cases, you will prefer using :

// send message to producer queue
Future<RecordMetadata> future = producer.send(new ProducerRecord<>(topic, key, message));
// flush producer queue to spare queuing time
producer.flush();
// throw error when kafka is unreachable
future.get(10, TimeUnit.SECONDS);
Camille Vienot
  • 727
  • 8
  • 6
1

The Thilo proposed answer is the way to go. In general, your suggestion about using max.in.flight.requests.per.connection = 1 is used for having still retries enabled but without losing messages ordering. It's not so used for having a sync producer.

ppatierno
  • 9,431
  • 1
  • 30
  • 45
1

When max.in.flight.requests.per.connection = 1, it just means the ordering of messages is guaranteed within a partition it has nothing to do with synchronization.

Python code in-case. For a synchronous send, make sure to block on the future with a good time-out.

from kafka import KafkaProducer
from kafka.errors import KafkaError

#by default ack = 1, if ack = 'all' --> waits for acks from replicas 
producer = KafkaProducer(bootstrap_servers=['brokerIP:9092'], ack= 'all')


key = b'key'
value = b'value'

future = producer.send("my-topic", key=key, value=value)

# block on this future for sync sends
try:
    record_metadata = future.get(timeout=10)
except KafkaError:
    log.exception()
    pass

print (record_metadata.topic)
print (record_metadata.partition)
print (record_metadata.offset)

producer.flush()
producer.close()
Smalltalkguy
  • 319
  • 1
  • 4
  • 13
0

From my adventures with Kafka :-) order of message production can only be guaranteed if you have one Producer thread and set max.in.flight.requests.per.connection = 1 (or turn of retries, i.e. retries= 0 or both).

If you what to scale to more than one Producer, then you have to "make sure" that messages that will be stored to the same partition will be produced by the same Producer instance.

Vassilis
  • 914
  • 8
  • 23
0

If you are not looking for an enterprise solution see this: https://dzone.com/articles/synchronous-kafka-using-spring-request-reply-1

Mohammad Reza Esmaeilzadeh
  • 1,313
  • 18
  • 31