How to convert a Flux to a Mono with multiple counters

Question

We are trying to count different things on a Flux. We currently use AtomicIntegers to do the counting, but how can we get these values once the Flux completes? I tried .then, .collectList().flatMap and other things, but it seems the Mono I am returning is always created when the AtomicIntegers are still 0. It works if I just subscribe() on the flux and then return a new Mono, but it doesn't make sense to do so. I even tried it with Mono.defer but no luck. Is there a way I am missing?

Not working:

var successCount = new AtomicInteger();
var errorCount = new AtomicInteger();
var filteredRecords = new HashMap<String, List<String>>();
try (var reader = new CSVReader(new InputStreamReader(new ZipInputStream(is)))) {
  return Flux.fromIterable(reader)
      .map(this::toRecord)
      .filter(record -> filterRecords(record, filteredRecords))// add filtered records to a List
      .map(this::processRecord)
      .doOnNext(result -> successCount.getAndIncrement())// count successfully processed records
      .onErrorContinue((ex, record) -> {
        log.error(ERROR_RECORD, StructuredArguments.v("record", record), ex);
        errorCount.getAndIncrement();
      }).collectList().flatMap(resultList -> Mono.defer(() -> checkErrorsAndLog(successCount.get(), errorCount.get(), filteredRecords, fileName)));
      //}).then(checkErrorsAndLog(successCount.get(), errorCount.get(), filteredRecords, fileName));// alternate approach
} catch (CsvValidationException | IOException exception) {
  log.error(ERROR_LOG_MSG, StructuredArguments.value(FILE_NAME, fileName),
      StructuredArguments.value(RECORD_COUNT, 0), exception);
  return Mono.error(new ProcessingException(String.format(ERROR_PROCESSING_FAILED, fileName), exception));
}

Only this works:

}).subscribe();
return checkErrorsAndLog(successCount.get(), errorCount.get(), filteredRecords, fileName);

score 0 · Answer 1 · answered Apr 18 '23 at 18:52

0

If you just create a Flux (or any other Publisher) it doesn't start processing the data automatically, but you have to subscribe to it so it starts consuming data.
But usually you just return a publisher (i.e. the Flux/Mono you establish here). You don't subscribe to it until the very last step when you have the final flow that produces the final result of the program.
The processing is not running in the same thread as you start it (unless you configured for it specifically). So event if you do subscribe it immediately continues with the next line of code, which closes the reader without giving enough time to consume it.
It seem you mix synchronous code with asynchronous, and it's not going to work correctly. If you consume a resource, i.e., open-read-close something, you have to do in as a part of the Publisher. See Flux.using(...) method.
If you want to return just counters ignoring the data itself, you can create a new Mono after you main Flux.

___ Like:

.collectList() // I'm not sure you need it, because you're ignoring them
.then(Mono.fromCallable {
     Tuples.of(successCount.get(), errorCount.get())
})

answered Apr 18 '23 at 18:52

Igor Artamonov

35,450
10
82
113

Yes, the caller of this method does the subscribe, even though it returns a `Mono`. I still think the Mono in the `then` is evaluated even before the counters were incremented. `fromCallable` also doesn't work as the method returns a `Mono` – Thomas Apr 19 '23 at 06:44
`Mono` in `then` is subscribed at the last step so it should contain a valid number. I'm not sure what you mean by the second statement. – Igor Artamonov Apr 19 '23 at 14:43
My investigation has shown, that there is either a problem with the `Flux.fromIterable(reader)` or the `CSVReader` itself. I managed to have a simple `Flux.just` and then the rest works, but also not with `then` as this Mono is evaluated before the flux completes. I have to use `.collectList().flatMap()` – Thomas Apr 19 '23 at 14:50
For `CSVReader` I suggested `Flux.using(...)`. Callable in `then` should be execute at the last step, unless your code have something for an early subscription – Igor Artamonov Apr 19 '23 at 14:54
Also tried it with `Flux.using` - not sure if I use it correctly though - but same behavior. Will ask a separate question for that. – Thomas Apr 19 '23 at 15:18

How to convert a Flux to a Mono with multiple counters

1 Answers1

Linked