3

I read in forum that while implementing any application using AMQP it is necessary to use fewer queues. So would I be completely wrong to assume that if I were cloning twitter I would have a unique and durable queue for each user signing up? It just seems the most natural approach and if not assign a unique queue for each user how would one design something like that.

What is the most used approach for web messaging. I see RabbitHUb and Rabbit WebHooks but Webhooks doesn't seem to be a scalable solution. i am working with Rails and my AMQP server as running as a Daemon.

scvalex
  • 14,931
  • 2
  • 34
  • 43
Sid
  • 6,134
  • 9
  • 34
  • 57
  • 1
    If you have a queue that is dedicated per user you are technically wasting resources when they are not using it, Can you provide a use case for your application? – Devin M Aug 12 '11 at 05:58
  • I was thinking of something like twitter or a simple chat application. The chat application that I built used one queue for each room. Also, i am curious to know how would one decide how many queues if its not one queue per user, since tweets and chats need to be constantly stored and even when the user is away. – Sid Aug 12 '11 at 06:33

2 Answers2

6

In RabbitMQ, queues are quite cheap. They're effectively lightweight Erlang processes, and you can run tens to hundreds of thousands of queues on a single commodity machine (i.e. my laptop). Of course, each will consume a bit of RAM, but unused-recently queues will hibernate, so they'll consume as little memory as possible. In addition, if Rabbit runs low on memory for messages, it will page old messages to disk.

The above only applies to a single machine. RabbitMQ supports a form of lightweight clustering. When you join several Rabbit nodes into a cluster, each can see the queues and exchanges on the other nodes but each runs only its own queues. So, you'll be able to have even more queues! (to the limit of Erlang clusters, which is usually a few hundred nodes) So, a cluster forms a logical broker distributed over several machines; clients connect to it and use it transparently through any of the nodes.

That said, having a single durable queue for each user seems a bit strange: in AMQP, you cannot browse messages while they're on the queue; you may only get/consume messages which takes them off the queue and publish which adds the to the end of the queue. So, you can use AMQP as a message router, but you can't use it as a sort of message database.

scvalex
  • 14,931
  • 2
  • 34
  • 43
  • Thanks scvalex that actually the kind of answer I was looking for. Any idea how I would approach the twitter/chat application scenario if I was not using a unique durable queue for each user. I am unable to visualize a design for applications that behave like twitter and any help would be invaluable. Thanks – Sid Aug 14 '11 at 15:00
  • I'm not quite sure what to say. I don't know how twitter works, but I don't think their core is something like a messaging system. I'd store the tweets in a database; same for the followers and the most recent N tweets for a user. If you do some caching in front of the webserver and between the server and database, I'd expect it to run fairly well for a moderate number of users (as opposed to the ridiculously many users Twitter has). Messaging is usually used (at least by myself) to coordinate different machines. – scvalex Aug 15 '11 at 09:06
  • 1
    That said, if you want to scale beyond some thousand users, you'll probably need to use smarter databases (distributing SQL over different machines is a bit problematic). For starters, let's say you expect a lot more reads than writes. Caching will do a lot of good, but you can do better by separating "write" databases from "read" databases. That is, you write only to one database (or a certain number of them) and you have separate read databases that mirror the write database and synchronize occasionally (they don't have to be in perfect sync; they just have to eventually consistent). – scvalex Aug 15 '11 at 09:12
  • CouchDB, another Erlang, project could handle this fairly nicely. It also includes a MapReduce framework for processing parallel data. This is all armchair programming, btw; I have never tried to do anything similar. – scvalex Aug 15 '11 at 09:13
0

Here is a thread that just talks about that: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2009-February/003041.html

Enno Shioji
  • 26,542
  • 13
  • 70
  • 109
  • Note that things have changed a bit since then: queue bindings need to be held in memory; topic bindings will take up quite a bit of memory, though, since the introduction of Fast Topic Routing, just how much memory depends on the complexity of the matches; direct bindings should be fine, even for huge numbers of queues. – scvalex Aug 12 '11 at 13:32