0

I'm working on putting a message passing system w/ Kombu together, but I ran into the following problem. Say I have messages that are being routed to routing keys 'x' and 'y'. This works great in situations where there are no dependencies between 'x' and 'y'.

However, consider another situation where I am sending data to routing keys 'a', 'b', and 'c', and a single queue is grabbing messages from those routing keys. If I require data from a, b, and c together to process a single callback, is there any way to aggregate these messages into a single worker drain, or is this a complete bastardization of the message passing paradigm?

I know that I could cache the message elsewhere (e.g., Redis) and only process when I have the requisite data, but I am wondering if Kombu could do this without having to cache the data and wake up the worker each time. Thanks for any suggestions; I can give some code examples if it's helpful.

Shookit
  • 1,162
  • 2
  • 13
  • 29
  • Of course you could store the messages on the client side somewhere and then process them. But why would you send messages with different routing keys if they depend on each other? Also can't you aggregate them on publishing side? – cantSleepNow May 25 '16 at 21:15
  • The actual dependency graph is more complex than my example. a, b, and c are generated from separate sources and/or separate machines. Furthermore, the (a, b, c) data dependency is not the only one; there are also independent a and b dependencies (and maybe even an (a, b) one). This lets the producer and consumer be much less directly coupled. – Shookit May 25 '16 at 21:45

0 Answers0