0

I have a REST server on heroku. It will have N-dynos for the REST service and N-dynos for workers.

Essentially, I have some long running rest requests. When these come in I want to delegate them to one of the workers and give the client a redirect to poll the operation and eventually return the result of the operation.

I am going to use JEDIS/REDIS from RedisToGo for this. As far as I can tell there are two ways I can do this.

  1. I can use the PUB/SUB functionality. Have the publisher create unique identities for the work results and return these in a redirect URI to the REST client.
  2. Essentially the same thing but instead of PUB/SUB use RPUSH/BLPOP.

I'm not sure what the advantage is to #1. For example, if I have a task called LongMathOperation it seems like I can simply have a list for this. The list elements are JSON objects that have the math operation arguments as well as a UUID generated by the REST server for where the results should be placed. Then all the worker dynos will just have blocking BLPOP calls and the first one there will get the job, process it, and put the results in REDIS using the key of the UUID.

Make sense? So my question is "why would using PUB/SUB be better than this?" What does PUB/SUB bring to the table here that I am missing?

Thanks!

robert_difalco
  • 4,821
  • 4
  • 36
  • 58

1 Answers1

0

I would also use lists because pubsub messages are not persistent. If you have no subscribers then the messages are lost. In other words, if for whatever reason you do not have any workers listening then the client won't get served properly. Lists are persistent on the other hand. But pubsub does not take as much memory as lists obviously for the same reason: there is nothing to store.

akonsu
  • 28,824
  • 33
  • 119
  • 194