0

I was wondering, because I can not find anything on symfony or other resources, if php's symfony/messenger can handle messages in "bulk" with any async transport.

For example. Grab 20 messages from the bus, handle those 20 messages, and ack or reject any of the messages.

I know RabbitMQ has a feature to grab n-amount of messages from the queue, and process all of them in a single run. In some cases this will have a better performance over scaling the async workers.

Does anybody have any leads, resources or experience with it? Or am I trying to resolve something by going against the idea of symfony/messenger?

[update]

I'm aware that bulk messages are not part of the (async) messaging concept. That each message should be processed individually. But some message brokers have implemented a feature to "grab" X-amount of messages from a queue and process them (either by sending an acknowledge or rejection, or otherwise). I know handling multiple messages in a single iteration increases complexity of any consumers, but in some cases it will improve performance.

I've used this concept of consuming multiple messages in a single iteration many times, but never with php's symfony/messenger.

Leroy
  • 1,600
  • 13
  • 23

2 Answers2

3

This was not natively possible prior to symfony 5.4.

They added a BatchHandlerInterface which will allow you to batch (and choose the size of the batch) your messages.

You can find more info here :

Elorfin
  • 2,487
  • 1
  • 27
  • 50
  • I haven't got any use-case anymore at this time. But I'm marking this as the answer. Thanks! – Leroy Feb 03 '22 at 12:12
0

First I think you have the wrong concept. There is no such thing as “messages in bulk” in the queue world.

The idea of the queue is that one message is received in the consumer and the consumer is responsible of letting know the queue that the message was acknowledged, so it can be deleted. If this does not happen in X time the message is again visible for other messages.

If the messenger get 20 messages from the queue it still process them one by one and after he finish processing just acknowledge every message. These 20 messages are “hidden” to other consumers for some time /it depends of the configuration of the queue/. This answer also the question of multiple consumers.

Alexander Dimitrov
  • 944
  • 1
  • 6
  • 17
  • Yes, I'm aware of this concept, I've applied it a few times in some use cases. I also understand bulk processing of messages as a concept, that it's not really a bulk. I was only wondering if this is even possible with the symfony/messenger php package, because I know some message busses have this feature available, just to make things just a bit more performant (however, it will cost more complexity). – Leroy May 20 '20 at 21:43
  • I have I legit use case for processing in bulk - I think: I work with a an API that allows me to batch PUT requests to make batch changes. My code dispatches each change as a single message. I want my asynchronous message handler to 1) accept a batch of 100 messages, 2) construct a single HTTP request, 3) accept all messages if success, or fail all messages on error. I didn't want to create one message containing the data for all of the 100 changes. If nothing else, I wanted to maintain handler-side flexibility about the batch size. Am I doing it wrong? – Jan Klan Apr 20 '22 at 01:00
  • Here is a link to a gist with my message handler FYI https://gist.github.com/janklan/6979d87124e2287ce6b3878bebf837b5 – Jan Klan Apr 20 '22 at 01:00