1

I have created a Flux that uses subscribeOn(Subscribers.boundedElastic). It is wrapping an Iterator I have created. What I would like is for the Flux to request the next element in a worker thread while the Subscriber is working on the current result. Ideally the Flux would call hasNext() -> next() -> hasNext() THEN send the result of the next() call to the subscriber. I tried doing this with publish(2) thinking the pre-fetch of 2 would do the trick but didn't seem to do what I wanted. Here is the code for creating the Flux.


  @Override
  public Flux<SearchResponse> createDeepQueryFlux(@NonNull PITSearchInput input) {
    validator.validate(input);
    return Flux.<SearchResponse, PointInTimeIterator>generate(
            () -> new PointInTimeIterator(forwardingRestHighLevelClient, input),
            (deepQueryIterator, sink) -> {
                log.info("Generate called");
              if (deepQueryIterator.hasNext()) {
                sink.next(deepQueryIterator.next());
              } else {
                sink.complete();
              }
              return deepQueryIterator;
            },
            (deepQueryIterator) -> deepQueryIterator.shutdown())
        .subscribeOn(Schedulers.boundedElastic());
  }
lkatiforis
  • 5,703
  • 2
  • 16
  • 35
John B
  • 32,493
  • 6
  • 77
  • 98
  • Spring Data provides a reactive client for ES. Why do you use the blocking client? – lkatiforis Dec 06 '21 at 13:14
  • @lkatiforis The issue with that has been that when using the reactive client, it makes N calls before the subscriber has a chance to finish the first. I asked this question specifically to fix that. That said, this question is general and not elasticsearch specific. https://stackoverflow.com/questions/70204206/how-do-i-create-a-flux-using-generate-wrapping-calls-that-return-a-mono – John B Dec 06 '21 at 13:25
  • @lkatiforis Event with the reactive client, I would ideally like to make the second search request while the first is being processed by the subscriber so that the next SearchResponse is ready when the subscriber is ready for it rather than waiting for the subscriber to be ready before calling elastic. Does that make sense or am I thinking of it incorrectly? – John B Dec 07 '21 at 13:44
  • If I understood correctly, you are using the ES Search After API with PIT. I believe you need the response(sort value, PIT) from the previous call in order to make the next one right? – lkatiforis Dec 07 '21 at 13:56
  • @lkatiforis Correct. Ideally the flux would 1) query ES and wait for the response 2) once response is received release it to the subscriber but ALSO make the next search request. My thought is that the producer would always be one query ahead of the subscriber so that the subscriber does not have to wait for the next query after processing the first. However, I don't want to use create that would pull N responses. I only want it to be exactly one query ahead of the subscriber. – John B Dec 07 '21 at 16:20

0 Answers0