0

Below is the config of job in etl.xml

<batch:job id="procuerJob">

<batch:step id="Produce">

    <batch:partition partitioner="partitioner">

        <batch:handler grid-size="${ partitioner.limit}"></batch:handler>

        <batch:step>

            <batch:tasklet>

                <batch:chunk reader="Reader" writer="kafkaProducer"
                             commit-interval="20000">

                </batch:chunk>

                <batch:listeners>

                    <batch:listener ref="producingListener" />

                    
                </batch:listeners>

            </batch:tasklet>

        </batch:step>

    </batch:partition>

</batch:step>

</batch:job>

below is the code used to send messaged to the topic.

ListenableFuture<SendResult<String, message>> listenableFuture = kafkaTemplate.send(message);

listenableFuture.addCallback(new ListenableFutureCallback<SendResult<String, message >>() {

@Override
public void onSuccess(SendResult<String, message > result) {
    log.info("marking as SUCCESS");
    manager.updateStatus(“someTable”, KafkaResponse.SUCCESS);
}

@Override
public void onFailure(Throwable ex) {
    log.info("marking as FAILURE");
    manager.updateKafkaStatus(someTable, KafkaResponse.FAILURE);
}

}

Once the kafkaTemplate.send(message)is executed , the listener is called and the job completes. I see the onSuccess(), onFailure() are called post the job is completed. How can I chnage the config of job so that listener is called after receiving the acknowledgement from kafka topic?

swetha
  • 91
  • 2
  • 11
  • Are you using a custom Kafka `ItemWriter`? – Mahmoud Ben Hassine Nov 02 '20 at 09:41
  • yes. which implements spring batch ItemWriter interface – swetha Nov 02 '20 at 09:50
  • The call to `kafkaTemplate.send` is asynchronous, so the registered callbacks might be called afterwards (ie after the job is finished). You should block waiting for the future to return and inspect its result if you want the write operation to finish before the job proceeds with subsequent chunks (or complete if the current chunk is the last one). – Mahmoud Ben Hassine Nov 02 '20 at 13:58
  • Is there a way that I can make use of CompletableFuture class ? – swetha Nov 02 '20 at 14:18
  • Would you like to some code example of what you suggested to block waiting for future. That might be of help. – swetha Nov 02 '20 at 14:23
  • I added an answer with a code example. The idea is to replace usage of callbacks which might be called after the job is finished with a blocking call to `future.get`. – Mahmoud Ben Hassine Nov 02 '20 at 15:33

1 Answers1

1

Would you like to some code example of what you suggested to block waiting for future. That might be of help.

I did not try the following but here is the idea:

ListenableFuture<SendResult<String, Message>> future = kafkaTemplate.send(message);
try {
    SendResult<String, Message> sendResult = future.get();
    // inspect sendResult
    log.info("marking as SUCCESS");
    manager.updateStatus(“someTable”, KafkaResponse.SUCCESS);
} catch (Exception e) {
     log.info("marking as FAILURE");
     manager.updateKafkaStatus(someTable, KafkaResponse.FAILURE);
     // do something with e
}
Mahmoud Ben Hassine
  • 28,519
  • 3
  • 32
  • 50
  • future.get() gives the message I sent to the topic. It doesn't tell me whether the message was sent to kafka successfully or failed for any reason. Here I want to get aknowledgement from kafka and based on that I want to call updateKafkaStatus(). – swetha Nov 03 '20 at 10:02
  • I used your suggestion to make use of future.get() and modified the code as below which worked the way I wanted. – swetha Nov 03 '20 at 10:16