Is it possible to add new CompletableFutures to CompletableFuture.allOf() dynamically?

Question

I have a database with 3 tables:

Table A which contains data of A objects
Table B which contains data of B objects
Table C which contains data of C objects

A objects can have 0 or 1 B objects
B objects can have 0 or 1 C objects
(I know, these could be in just one table, but its just for the example)

I want to make a csv file from the whole database: each line should contain exactly one A object, optionally its B object, and optionally its C object.

For each table there is an asynchronous repository, that gives back a CompletionStage. So when I fetch A objects from the repositoryA, I get back a CompletionStage<List<A>>. When it completes, I make a Map for every A object, fill it with the data of A and I call the repositoryB.getB(A.id) , which gives back a CompletionStage<Optional<B>>. If the B value is not present, I append a new line to my CSV file with the data inside of the map. If the B is present, I add its values to the map, and call repositoryC.getC(B.id) which returns a CompletionStage<Optional<C>>. If the C is present, I add its values to the Map, and add a new line to the CSV file, if it is not, then I just add the new line.

The creation of the CSV is done, when all CompletionStages are completed. I tried to use the CompletableFuture.allOf(), but since at the beginning i don't know how many CompletionStages there will be, I can't add all of them to the allOf method, so I think i would need to add the Completionstages dynamically somehow. Is it possible?

Currently I have a working solution, but it blocks after every B and C fetch, so I want to make the whole code nonblocking.

This is my nonblocking attempt, but its not working well, as some of the B and C futures aren't added to the list of futures, so the code does not wait for their completion:

CompletableFuture<List<CompletableFuture>> genereteCSV = repositoryA.getAs().thenApplyAsync(listA-> {
            List<CompletableFuture> futures = new ArrayList<>();
            for (A a : listA) {
                Map<String, String> values = new Map<>();
                addAvaluesToMap(values, A);

                CompletableFuture Bfuture = repositoryB.getB(A.id).thenAcceptAsync((optionalB -> {
                    if (optionalB.isPresent()) {
                        addValuesToMap(values, B);

                        CompletableFuture Cfuture = repositoryC.getC(B.id).thenAcceptAsync(optionalC-> {
                            if (optionalC.isPresent()) {
                                addAvaluesToMap(values, C);
                            } 
                            addMapValuesToCSV(values);
                        });
                        futures.add(Cfuture);

                    } else {
                        addMapValuesToCSV(values);
                    }
                }));

                futures.add(Bfuture);
            }
            return futures;
        });

        geerateCSV.thenApplyAsync(futureList-> CompletableFuture.allOf(futureList.toArray(new CompletableFuture<?>[0])))
        .thenAccept(dummy->{System.out.println("CsV generation done");});

Oleksii Zghurskyi · Answer 1 · 2019-09-24T19:30:09.097

Basically, this is one possible plan to achieve non-blocking processing:

Create CompletableFuture for each object A (possibly populated with B and C objects)
Asynchronously gather A objects from CompletableFutures
Write created A objects to CSV file

Notes: In the example below I've used addAvaluesToMap and addMapValuesToCSV assuming, that you have them working. Also, I assume, that your usage of CompletableFutures is justified by your goals.

This will be an implementation of approach described above:

public void generateCSV() {
    repositoryA.getAs().thenAccept(listA -> {
        List<CompletableFuture<A>> futures = listA.stream()
                .map(a -> repositoryB.getB(a.id).thenComposeAsync(optionalB ->
                        optionalB.map(b -> repositoryC.getC(b.id).thenComposeAsync(optionalC -> {
                                    a.setB(b);
                                    return optionalC.map(c -> {
                                        b.setC(c);
                                        return CompletableFuture.completedFuture(a);
                                    }).orElse(CompletableFuture.completedFuture(a));
                                })
                        ).orElse(CompletableFuture.completedFuture(a)))
                ).collect(Collectors.toList());

        CompletableFuture.allOf(futures.toArray(new CompletableFuture<?>[0]))
                .thenAccept(v -> futures.stream()
                        .map(CompletableFuture::join)
                        .forEach(a -> {
                            Map<String, String> values = new HashMap<>();
                            addAvaluesToMap(values, a);
                            addMapValuesToCSV(values);
                        })
                )
                .exceptionally(throwable -> {
                    System.out.println("Failed generating CSV. Error: " + throwable);
                    return null;
                });
    }).exceptionally(throwable -> {
        System.out.println("Failed to get list of As. Error: " + throwable);
        return null;
    });
}

robjwilkins · Answer 2 · 2019-09-26T07:11:55.180

You are using a relational database. It should be easier and more performant to write a database query to return the data you want in the format you need than to write this in java. An SQL query will allow you to join three tables together very easily and provide the results in a format which can easily be extracted in csv format. Databases can perform these operations much more effectively than by writing your own implementation.

Is it possible to add new CompletableFutures to CompletableFuture.allOf() dynamically?

2 Answers2