15

Would have loved to use Amazon SQS if it provided some semblance of FIFO access, but the sequence seems to completely random.

Is there something that would provide me FIFO queuing as-a-cloud-service with the high availability of SQS?

If that is asking for too much - what would be the easiest way of putting together something with the above requirements in EC2? Or maybe in other words, what's the easiest highly available queuing solution that works in EC2?

Thanks for your insights!

Steffen Opel
  • 63,899
  • 11
  • 192
  • 211
user393144
  • 1,575
  • 3
  • 14
  • 21

4 Answers4

17

Update 2016-11-19

Amazon SQS has just gained FIFO Queues with Exactly-Once Processing & Deduplication:

Today we are making SQS even more powerful and flexible with support for FIFO (first-in, first-out) queues. We are rolling out this new type of queue in two regions now, and plan to make it available in many others in early 2017.

These queues are designed to guarantee that messages are processed exactly once, in the order that they are sent, and without duplicates. [...]

[emphasis mine]

As emphasized, these new FIFO SQS queues will cover the use case at hand, but are not yet available in all SQS regions [initially only in US East (Ohio) and US West (Oregon)]. Also, the SQS FAQ for FIFO queues outlines notable differences between standard and FIFO queues that should be considered upfront, for example a throughput limit of 300 transactions per second.

Initial Answer

Would have loved to use Amazon SQS if it provided some semblance of FIFO access, but the sequence seems to completely random.

While I haven't experienced completely random message ordering yet (this likely depends on the use case and especially the message volume though), there is no FIFO guarantee indeed, see the respective FAQ Does Amazon SQS provide first-in-first-out (FIFO) access to messages?:

No, Amazon SQS does not guarantee FIFO access to messages in Amazon SQS queues, mainly because of the distributed nature of the Amazon SQS. If you require specific message ordering, you should design your application to handle it.

Given you expressed interest in hosted RabbitMQ as well as StormMQ, I might as well point you to other commercial offerings:

CloudAMQP

CloudAMQP is RabbitMQ as a Service, thus exactly what you have been asking for, given RabbitMQ supports the desired true FIFO message ordering (see Amazon SQS vs. RabbitMQ for a nice comparison).

According to their Plans & Prices, it is apparently only offered as an addon to first class platform providers though, hence you'll have to look into these in turn:

IronMQ

IronMQ offers developers ready-to-use messaging with highly reliable delivery and cloud-optimized performance. It is complying to nowadays expectations on a Software as a service (SaaS) product, not the least regarding an easy to understand and especially published pricing model As rightfully critizised by Sleavely, Iron.io has seemingly dropped its former exemplary pricing model - see Alex Payne's How Not To Sell Software in 2012 for a nice rant and advise regarding this.

I've only tested it shortly myself so far, but have been pretty pleased with the offered features and language integrations (see Client Libraries and Beanstalkd Support) - given their competitive price tag and especially the free tier makes it a good candidate for exploring a message queuing as a service solution, not the least in combination with their second product IronWorker (An easy-to-use and massively scalable task queue [...]), which provides functionality not even available from AWS as of today.

FIFO message ordering

Unfortunately I haven't been able to figure out whether true FIFO is supported by IronMQ directly, consequently I actually doubt it - accordingly, you'd need to file a support request to verify this.

Evan Shaw from Iron.io confirmed that IronMQ provides FIFO message ordering in fact (thanks much).

Community
  • 1
  • 1
Steffen Opel
  • 63,899
  • 11
  • 192
  • 211
  • 6
    I work at Iron.io and can confirm that IronMQ orders messages FIFO. – Evan Shaw May 02 '12 at 20:02
  • 1
    @EvanShaw: Thanks much, proactive support like this is highly appreciated :) – Steffen Opel May 02 '12 at 20:17
  • Thanks a bunch. I will go research these options and post back. – user393144 May 03 '12 at 21:46
  • @EvanShaw: is it only FIFO or is EOIO (Exactly Once In Order)? – Remus Rusanu Jan 06 '13 at 10:32
  • Rabbitmq: From 2.7.0 the relative order of re-queued messages from a single consumer is preserved. Therefore, if another consumer receives them later, they will be consumed in the same order they originally appeared. Of course, if two or more consumers on the same queue fail, there is no guarantee that messages re-queued by distinct consumers will retain their relative order. But in the majority of cases where order matters, this guarantee should be enough – TlmaK0 Jul 04 '13 at 15:04
  • 1
    The only way to access IronMQ's pricing is through Archive.org (anno 2014 or 2015); unfortunately it seems IronMQ doesn't qualify for "How to sell software in 2017". – Sleavely Jan 19 '17 at 14:50
0

Check out RabitMQ and StormMQ.

Jeff
  • 6,646
  • 5
  • 27
  • 33
  • StormMQ looks interesting, but it seems hard to find information about their pricing from their website. Also they seem to be in a closed BETA. On RabbitMQ I didn't find any hosted solution on their website - is there one that you can point me to? – user393144 Apr 30 '12 at 16:57
0

ActiveMQ is a widely used message broker that offers FIFO queues. Have a look at http://www.cinovo.de/activemq-as-a-service if you are looking for a high availability and hosted version for use with AWS.

Andreas
  • 844
  • 4
  • 10
-1

I think IronMQ would be the way to go for you

user3662429
  • 71
  • 1
  • 8