7

I am interested in the Pub/Sub paradigm in order to provide a notifications system (ie : like Facebook), especially in a web application which has publishers (in several web applications on the same web server IIS) and one or more subscribers, in charge to display on the web the notifications for the front user.

I found out Redis, it seems to be a great server which provides interesting features : Caching (like Memcached) , Pub/Sub, queue.

Unfortunately, I didn't find any examples in a web context (ASP.NET, with Ajax/jQuery), except WebSockets and NodeJS but I don't want to use those ones (too early). I guess I need a process (subscriber) which receives messages from the publishers but I don't see how to do that in a web application (pub/sub works fine with unit tests).

EDIT : we currently use .NET (ASP.NET Forms) and try out ServiceStack.Redis library (http://www.servicestack.net/)

Olivier DUVAL
  • 121
  • 3
  • 8
  • Not sure I understand. Forgetting about the details, why don't you add a web service (using for example WCF/REST with JSON, or whatever) to your web site (which is already configured to scale) and query this web service using ajax technology (JQuery, ASP.NET Ajax, again whatever) in the pages? What's missing in this picture, in terms of architecture? – Simon Mourier Jun 24 '11 at 07:24

3 Answers3

5

For this application, I would strongly suggest using SignalR, which is a .Net framework that enables real-time push to connected clients.

ElHaix
  • 12,846
  • 27
  • 115
  • 203
5

Actually Redis Pub/Sub handles this scenario quite well, as Redis is an async non-blocking server it can hold many connections cheaply and it scales well.

Salvatore (aka Mr Redis :) describes the O(1) time complexity of Publish and Subscribe operations:

You can consider the work of subscribing/unsubscribing as a constant time operation, O(1) for both subscribing and unsubscribing (actually PSUBSCRIBE does more work than this if you are subscribed already to many patterns with the same client).

...

About memory, it is similar or smaller than the one used by a key, so you should not have problems to subscribe to millions of channels even in a small server.

So Redis is more than capable and designed for this scenario, but the problem as Tom pointed out in order to maintain a persistent connection users will need long-running connections (aka http-push / long-poll) and each active user will take its own thread. Holding a thread isn't great for scalability and technologically you would be better off using a non-blocking http server like Manos de Mono or node.js which are both async and non-blocking and can handle this scenario. Note: WebSockets is more efficient for real-time notifications over HTTP, so ideally you would use that if the users browser supports it and fallback to regular HTTP if they don't (or fallback to use Flash for WebSockets on the client).

So it's not the Redis or its Pub/Sub that doesn't scale here, it's the number of concurrent connections that a threaded HTTP server like IIS or Apache that is the limit, with that said you can still support a fair amount of concurrent users with IIS (this post suggests 3000) and since IIS is the bottleneck and not Redis you can easily just add an extra IIS server into the mix and distribute the load.

Community
  • 1
  • 1
mythz
  • 141,670
  • 29
  • 246
  • 390
  • Thanks for your response mythz. It sounds not good using Pub/Sub in a web context (under IIS) :( May be, the solution would be to develop a **windows service** (or a "daemon") which would be as subscriber. In this way, publishers (web app) could send their notifications toward this service. – Olivier DUVAL Jun 23 '11 at 09:07
3

Redis publish/subscribe is not designed for this scenario - it requires a persistent connection to redis, which you have if you are writing a worker process but not when you are working with stateless web requests.

A publish/subscribe system that works for end users over http takes a little more work, but not too much - the simplest approach is to use a sorted set for each channel and record the time a user last got notifications. You could also do it with a list recording subscribers for each channel and write to the inbox list of each of those users whenever a notification is added.

With either of those methods a user can retrieve their new notifications very quickly. It will be a form of polling rather than true push notifications, but you aren't really going to get away from that due to the nature of http.

Technically you could use redis pub/sub with long-running http connections, but if every user needs their own thread with active redis and http connections, scalability won't be very good.

Tom Clarkson
  • 16,074
  • 2
  • 43
  • 51