How to create buffers of latest message for each id of group-by with project reactor?

Question

I have a stream in which I'm grouping it by id (to be able to process different ids in parallel but within the same id, by order). I'd like to write to MongoDB with batches that contain each message per id only once (to make sure that each batch operation will update with only one document at a time), and after every write, I'd like to create the next batch for the latest message of each group. I've created an image to demonstrate what I mean - the orange circles are the intended batches. Each rectangle represents a thread that holds GroupedFlux<?>.

enter example image here

I would like to know what is the operation that will enable doing this.

score 0 · Answer 1 · answered Apr 15 '22 at 11:03

Each rectangle represents a thread that holds GroupedFlux<?>

How do You group by #1 #2 and run groups on separate thread? I think there is no guarantee that group #1 will always run on the same thread?

Anyway I think this could be a solution to Your problem, You have to make parallel execution than merge groups and put it into mongo in reactive way and run other subscribers in separate threads.

public class Test1 {
    public static void main(String[] args) throws InterruptedException {

        Flux<TestData> cache = Flux.just(
                        TestData.of("#1", "a"), TestData.of("#1", "b"), TestData.of("#1", "c"), TestData.of("#1", "d"),
                        TestData.of("#2", "a"), TestData.of("#2", "b"), TestData.of("#2", "c"), TestData.of("#2", "d"),
                        TestData.of("#3", "a"), TestData.of("#3", "b"), TestData.of("#3", "c"), TestData.of("#3", "d"),
                        TestData.of("#4", "a"), TestData.of("#4", "b"), TestData.of("#4", "c"), TestData.of("#4", "d")
                )
                .delayElements(Duration.ofMillis(100))
                .groupBy(TestData::getX)
                .parallel(4)
                .flatMap(g -> g)
                .sequential()
                .groupBy(TestData::getY)
                .map(g -> g.cache(1))
                .cache().flatMap(g -> g);


        cache
                .subscribeOn(Schedulers.parallel())
                .subscribe(next -> System.out.printf("All %s%n", next)); //DO WHATEVER YOU WANT

        Thread.sleep(1000);
        System.out.println("-------------------- 1 second passed ------------------------------");

        cache
                .subscribeOn(Schedulers.parallel())
                .subscribe(next -> System.out.printf("Latest after 1 sec %s%n", next));

        Thread.sleep(1000);
        System.out.println("--------------- 2 second passed -----------------------");

        cache
                .subscribe(next -> System.out.printf("Unique 1 sec %s%n", next));
    }

    @Value(staticConstructor = "of")
    public static class TestData {

        private final String x;
        private final String y;
    }
}

Outputs:

All Test1.TestData(x=#1, y=a)
All Test1.TestData(x=#1, y=b)
All Test1.TestData(x=#1, y=c)
All Test1.TestData(x=#1, y=d)
All Test1.TestData(x=#2, y=a)
All Test1.TestData(x=#2, y=b)
All Test1.TestData(x=#2, y=c)
All Test1.TestData(x=#2, y=d)
All Test1.TestData(x=#3, y=a)
-------------------- 1 second passed ------------------------------
Latest after 1 sec passed Test1.TestData(x=#3, y=a)
Latest after 1 sec passed Test1.TestData(x=#2, y=b)
Latest after 1 sec passed Test1.TestData(x=#2, y=c)
Latest after 1 sec passed Test1.TestData(x=#2, y=d)
All Test1.TestData(x=#3, y=b)
Latest after 1 sec passed Test1.TestData(x=#3, y=b)
All Test1.TestData(x=#3, y=c)
Latest after 1 sec passed Test1.TestData(x=#3, y=c)
All Test1.TestData(x=#3, y=d)
Latest after 1 sec passed Test1.TestData(x=#3, y=d)
All Test1.TestData(x=#4, y=a)
Latest after 1 sec passed Test1.TestData(x=#4, y=a)
All Test1.TestData(x=#4, y=b)
Latest after 1 sec passed Test1.TestData(x=#4, y=b)
All Test1.TestData(x=#4, y=c)
Latest after 1 sec passed Test1.TestData(x=#4, y=c)
All Test1.TestData(x=#4, y=d)
Latest after 1 sec passed Test1.TestData(x=#4, y=d)
--------------- 2 second passed -----------------------
Unique 2 sec passed Test1.TestData(x=#4, y=a)
Unique 2 sec passed Test1.TestData(x=#4, y=b)
Unique 2 sec passed Test1.TestData(x=#4, y=c)
Unique 2 sec passed Test1.TestData(x=#4, y=d)

How to create buffers of latest message for each id of group-by with project reactor?

1 Answers1