16

I am a newbie to real-time application development and am trying to wrap my head around the myriad options out there. I have read as many blog posts, notes and essays out there that people have been kind enough to share. Yet, a simple problem seems unanswered in my tiny brain. I thought a number of other people might have the same issues, so I might as well sign up and post here on SO. Here goes:

I am building a tiny real-time app which is asynchronous chat + another fun feature. I boiled my choices down to the following two options:

  1. LAMP + RabbitMQ
  2. Node.JS + Redis + Pub-Sub

I believe that I get the basics to start learning and building this out. However, my (seriously n00b) questions are:

  • How do I communicate with the end-user -> Client to/from Server in both of those? Would that be simple Javascript long/infinite polling?
  • Of the two, which might more efficient to build out and manage from a single Slice (assuming 100 - 1,000 users)?
  • Should I just build everything out with jQuery in the 'old school' paradigm and then identify which stack might make more sense? Just so that I can get the product fleshed out as a prototype and then 'optimize' it. Or is writing in one over the other more than mere optimization? ( I feel so, but I am not 100% on this personally )

I hope this isn't a crazy question and won't get flamed right away. Would love some constructive feedback, love this community!

Thank you.

iUsable
  • 361
  • 3
  • 13

3 Answers3

21

Architecturally, both of your choices are the same as storing data in an Oracle database server for another application to retrieve.

Both the RabbitMQ and the Redis solution require your apps to connect to an intermediary server that handles the data communications. Redis is most like Oracle, because it can be used simply as a persistent database with a network API. But RabbitMQ is a little different because the MQ Broker is not really responsible for persisting data. If you configure it right and use the right options when publishing a message, then RabbitMQ will actually persist the data for you but you can't get the data out except as part of the normal message queueing process. In other words, RabbitMQ is for communicating messages and only offers persistence as a way of recovering from network problems or system crashes.

I would suggest using RabbitMQ and whatever programming languages you are already familiar with. Since the M in LAMP is usually interpreted as MySQL, this means that you would either not use MySQL at all, or only use it for long term storage of data, not for the realtime communications.

The RabbitMQ site has a huge amount of documentation about building apps with AMQP. I suggest that after you install RabbitMQ, you read through the docs for rabbitmqctl and then create a vhost to experiment in. That way it is easy to clean up your experiments without resetting everything. I also suggest using only topic exchanges because you can emulate the behavior of direct and fanout exchanges by using wildcards in the routing_key. Remember, you only publish messages to exchanges, and you only receive messages from queues. The exchange is responsible for pattern matching the message's routing_key to the queue's binding_key to determine which queues should receive a copy of the message. It is worthwhile learning the whole AMQP model even if you only plan to send messages to one queue with the same name as the routing_key.

If you are building your client in the browser, and you want to build a prototype, then you should consider just using XHR today, and then move to something like Kamaloka-js which is a pure Javascript implementation of AMQP (the AMQ Protocol) which is the standard protocol used to communicate to a RabbitMQ message broker. In other words, build it with what you know today, and then speed it up later which something (AMQP) that has a long term future in your toolbox.

Michael Dillon
  • 31,973
  • 6
  • 70
  • 106
  • This was super useful! Thank you! Sorry can't vote it up just yet :) – iUsable May 30 '11 at 07:57
  • This is what I am thinking - 1. Build out on LAMP and jQuery for myself, just to prototype the entire thing. 2. Setup Mongo or Redis to handle the real-time data in-memory or virtualized, test everything still works. 3. Use Pub/Sub or RabbitMQ to optimize the transport layer, test. 4. Do I need anything for the client-server side, like cometd or do I just use long-polling? I don't want to use websockets with their limited adoption currently. Thank you. – iUsable May 30 '11 at 08:01
  • 1
    use a client library like socket.io to avoid the slow adoption of websockets -- it falls back on long polling/flash socket etc. Modules exist for both client and server. – Josh May 31 '11 at 01:21
  • Using socket.io, that makes sense :) – iUsable Jun 13 '11 at 14:45
2

Should I just build everything out with jQuery in the 'old school' paradigm and then identify which stack might make more sense? Just so that I can get the product fleshed out as a prototype and then 'optimize' it. Or is writing in one over the other more than mere optimization? ( I feel so, but I am not 100% on this personally )

This is usually called RAD (rapid application design/development) and it is what I would recommend right now. This lets you build the proof of concept that you can use to work off of later to get what you want to happen.

As for how to talk to the clients from the server, and vice versa, have you read at all on websockets?

Given the choice between LAMP or event based programming, for what you're suggesting, I would tell you to go with the event based programming, so nodejs. But that's just one man's opinion.

jcolebrand
  • 15,889
  • 12
  • 75
  • 121
  • Thank you jcolebrand. I have been reading a lot on WebSockets (and socket.io and pusherapp.com) but the problem is that it just doesn't have ubiquitous or near ubiquitous acceptance yet. Especially on mobile browsers (even upto gingerbread afaik). – iUsable May 29 '11 at 19:47
  • The other concern I have is do I end up building the classic sql way and redo all of the db stuff later in the k-v model for Redis or just go all in now? – iUsable May 29 '11 at 19:48
  • You have to pick your battles. I didn't see your requirements for mobile up above, so I presumed you wanted a RIA with the full support of a desktop app. You're just going to have to go for what you can get. – jcolebrand May 29 '11 at 19:49
  • do the RAD the way it works best for you, so long as those things are clearly defined in comments and intent, then converting later will be easy enough. – jcolebrand May 29 '11 at 19:49
  • I hear you on this, thank you. Any ideas on 'How do I communicate with the end-user -> Client to/from Server in both of those? Would that be simple Javascript long/infinite polling?' – iUsable May 29 '11 at 19:54
  • websockets replaces javascript longpolling. I suggest you read on that before we continue this discussion, as it were ;) Also, see http://chat.stackoverflow.com/rooms/17/javascript for more targeted javascript questions that you don't have a clear question about. – jcolebrand May 29 '11 at 20:43
0

Well,

LAMP - Apache create new process for every request. RabbitMQ can be useful with many features. Node.js - Uses single process to handle all request asynchronously with help of event looping. So, no extra overhead process creation like apache. For asynchronous chat application, socket.io + Node.js + redis pub-sup is best stack. I have already implemented real-time notification using above stack.

Hemant Thorat
  • 2,386
  • 20
  • 14