Open Search BulkProcessor retries config not working

Question

What is the right way to configure the retries of bulk api? I am having some of the batches fail with 429 and 504 which could be retried.

Here is how the bulk processor is setup

    public BulkProcessor createBulkProcessor() {

        BiConsumer<BulkRequest, ActionListener<BulkResponse>> bulkAsyncRunner =
                (request, bulkListener) -> {

                    request.timeout(TimeValue.timeValueSeconds(bulkApiConfig.timeOutSecs));
                    client.bulkAsync(request, RequestOptions.DEFAULT, bulkListener);
                };

        BulkProcessor.Builder builder = BulkProcessor.builder(bulkAsyncRunner, new BulkProcessorListener());

                if(bulkApiConfig.payloadSizeMb!=null){
                    builder.setBulkSize(new ByteSizeValue(bulkApiConfig.payloadSizeMb, ByteSizeUnit.MB));
                }else{
                    builder.setBulkActions(bulkApiConfig.batchSize);
                }
                builder.setConcurrentRequests(bulkApiConfig.concurrentRequests)
                .setBackoffPolicy(
                        BackoffPolicy.exponentialBackoff(TimeValue.timeValueMillis(bulkApiConfig.retryInitialDelayMs),
                                bulkApiConfig.retries));
        return builder.build();
    }

public class BulkProcessorListener implements BulkProcessor.Listener {

    @Override
    public void afterBulk(long executionId, BulkRequest request, Throwable failure) {
       //log error
   }
}

However afterBulk() logs just once and not as many times as configured in backOffPolicy

Using

   implementation group: 'org.opensearch.client', name: 'opensearch-rest-high-level-client', version: '2.7.0'