3

I'm looking for a way of throttling various processes on a cluster-wide basis. This requires some kind of centralised control that can cope with an arbitrary number of event consumers. The thought that I had involves a read-only queue that generates tokens at a certain rate with no backlog (so missed events are just discarded). For example, say I have some web API that needs to be throttled to 10,000 messages per hour, but that can be called from any number of servers in a cluster. I would configure a queue to generate tokens at 10k messages/hour, and all servers connect to that queue and retrieve a token before proceeding. This would introduce an element of latency (of 3600/10000 sec after the first request), but would be smooth and predictable regardless of consumer count. I don't want to have a backlog because I don't want to have a rush after a quiet period - the aim is not just to limit to a total number of events per hour, but to spread them evenly across it.

My main app is PHP and it's running on linux. At the moment I'm very happy with beanstalkd for normal queuing, but it doesn't support this mode of operation. I've used RabbitMQ in the past but found it heavy and fragile in comparison. It would be nice if this could be done by the queue manager itself since it needs no external input after configuration.

In the absence of specific support for something like this, I could try using an ordinary queue with a process pushing tokens into it with very short expiry, though that seems very inelegant. Any better ideas?

Synchro
  • 35,538
  • 15
  • 81
  • 104

1 Answers1

0

You can use one of this approaches:

  1. Use topic exchange with regular queues but set message ttl to your needs. The pro of this method is that you can have small backlog, say, for last 5 seconds, which allow your application to recover after short-time issues, like network lose during some maintenance. No cons.

  2. You can publish messages to fanout exchange and declare queues with auto-delete flag and then consume message from them. The biggest con of this way is that messages get duplicated through queues. Actually, it may be pro if you need such behavior, but you can also achieve it with topic exchange easy with additional queues with the same binding.

pinepain
  • 12,453
  • 3
  • 60
  • 65
  • Duplicates of messages don't matter - all that matters is the availability of a token; it doesn't need to have any content or meaning at all - I don't actually need any message input at all if the queue can create its own tokens. I really don't want to use Rabbit as I've had so many issues with it in the past. – Synchro Jan 22 '14 at 14:35