I've been trying to build an app using the Java Flow API. While the idea of being able to perform backpressure between the publisher and subscriber in an event when their speeds are different, I'm not sure how it really helps since both the publisher and consumer usually reside within the same application; at least, that's what almost all examples online look like.
For example, in my application, there is a publisher producing messages retrieved from a RabbitMQ and a subscriber processing those messages. So, the message is submitted to the publisher in the RabbitMQ listener like so:
@RabbitListener(queues = "rabbit_queue")
public void rabbitHandler(MessageObject message) {
// do some stuff to the message and then submit it to the publisher
publisher.submit(message);
}
// Then the message will be processed in the subscriber
In an event if a publisher is producing faster than the subscriber can process, the subscriber can call a small n value on subscription.request(n). But, there are 2 things that I am not sure if my understanding of how the request(n) is going to help is correct:
- Since both the publisher and subscriber are in the same Spring application in this case, they pretty much share and are limited by the same amount of resources. If the subscriber is going to run out of memory or resources because there were too many elements being sent to it, we are supposed to be able to reduce the
nvalue inrequest(n). But this will then mean the buffer size in publisher will be full quickly. I can increase the buffer size in the publisher but I'm also limited by the same amount of resources the subscriber was facing because both the publisher and subscriber are in the same application using the same set of resources. Then what's the point of having all of those extra complexity of having a publisher and thatrequest()methods? - It seems to me that the publisher is usually receiving its elements from some sources. Sometimes, not all of these sources can be throttled. In my case, I have a RabbitMQ listener sending the messages to the publisher. But the rate at which the publisher is going to send out those messages to the subscription is largely dependent on the rate at which the
rabbitHandleris going to receive the messages from the RabbitMQ queue. If the RabbmitMQ is sending messages faster than the publisher's subscriber can process, the buffering of the messages are still done between the publisher and subscriber within the application and the problem in the above point will occur.
I'm pretty sure there is somewhere wrong in my understanding of the process because it feels like a catch-22 situation to me. It's like I can only hold so many balls in my 2 hands and I'm just passing the balls around between my 2 hands and calling it backpressure. Since both the publisher and subscriber are limited by the same amount of resource as they are both in the same application, what's the benefits of having that extra complexity when I could simply just pass the message on to another handler and be limited by the same amount of resources too, like this:
public class RabbitMqListener {
@RabbitListener(queues = "rabbit_queue")
public void rabbitHandler(MessageObject message) {
// do some stuff to the message and then submit it to the publisher
MessageProcessor.process(message);
}
}
public class MessageProcessor {
public static void process(MessageObject message) {
System.out.println("processing message...");
}
}
It will be great if somebody can help me to correct my understanding.
