4

I have a Spring Boot app using Reactor Kafka Consumer. The consumed messages must be sent to a webservice. Each HTTP request should contain x messages. After the webservice sends a HTTP response, I want to send the next request with the next x messages.

I am using Flux.window(int size) to take x messages and send the using Flux.flatMap(). In the flatMap() I use Spring WebClient.

Currently the windows are overlapping. This results in sending too many requests which is to much for the webservice.

This is my code:

@Service
public class KafkaConsumerService {
    private final ReceiverOptions<String, String> receiverOptions;
    @Value("${baseurl}")
    private String baseUrl;

    @Value("${windowSize}")
    private int windowSize;

    private static final Logger LOGGER = LoggerFactory.getLogger(KafkaConsumerService.class);
    private final WebClient webClient;

    private Disposable stream;

    @Autowired
    public KafkaConsumerService(ReceiverOptions<String, String> receiverOptions) {
        this.receiverOptions = receiverOptions;
        this.webClient = WebClient
                .builder()
                .defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_STREAM_JSON_VALUE)
                .build();
    }

    @PostConstruct
    public void startPipeline() {
        KafkaReceiver.create(receiverOptions)
                .receiveAutoAck()
                .concatMap(r -> r)
                .map(ConsumerRecord::value)
                .window(windowSize)
                .flatMap(this::sendToWebservice)
                .subscribe();
    }

    private Publisher<String> sendToWebservice(Flux<String> windowFlux) {
        LOGGER.info("===== sending...");
        return webClient
                .post()
                .uri(URI.create(baseUrl + "items"))
                .contentType(MediaType.APPLICATION_STREAM_JSON)
                .body(windowFlux, String.class)
                .retrieve()
                .bodyToMono(String.class)
                .retryWhen(Retry.any()
                        .exponentialBackoff(Duration.ofSeconds(1), Duration.ofSeconds(10))
                        .retryMax(15))
                .doOnError((error) -> LOGGER.error("===== ERROR in sending request"))
                .doOnSuccess((x) -> LOGGER.info("===== Success"));
    }
}

The current output looks like, but this isn't the desired result: 

===== sending...
===== sending...
===== sending...
===== sending...
===== sending...
===== sending...
===== sending...
===== sending...
===== sending...
===== sending...
===== Success
===== Success
===== Success
===== Success
===== Success
===== Success
===== Success
===== Success
===== Success
===== Success

How can I make a window wait for the previous one to "complete"? Shouldn't be there a backpressure from subscribing the webclient-response, that slows down the publisher (=kafka receiver)? Is window the correct operator or is there something else what I should use?

Raman
  • 548
  • 1
  • 7
  • 17
  • Hi, I have a similar use case do you have a clean way to throttle those webservice requests? – Saveriu CIANELLI Apr 21 '21 at 16:07
  • Hi @SaveriuCIANELLI, I did not find any solution. I switched back to Spring MVC for this. – Raman Apr 28 '21 at 18:51
  • Hi again! I found a way to "batch" kafka messages if you're interrested: https://stackoverflow.com/questions/67210355/slow-down-kafkareceiver-webclient-calls-on-reactor-flux?noredirect=1#comment118818685_67210355 – Saveriu CIANELLI Apr 28 '21 at 20:26

0 Answers0