Questions tagged [message-queue]

"Message queue" is a design pattern or software engineering component that defines discipline or API for communication between two or more interrelated processes or systems. The message queue enforces asynchronous processing and loose coupling. Depending on the implementation they may also provide delivery and order guarantees of the messages. Message processing guarantees are explicitly excluded from the design.

Message queue design pattern introduces an intermediary between the producer of messages and its consumer(s). This intermediary - the queue - accepts the messages and stores them until a consumer retrieves them or until some other event (e.g. expiration time). In general the producer does not know or care where, how, and when the messages in the queue will be consumed.

Software components such as MSMQ or IBM MQ or Tibco (usually called Queue Managers) provide queue storage and features for manipulation of messages in queues. These manipulations include: expiration, persistence, notification of expired messages (dead-letter queues), message cloning, queue security, distribution (between multiple Queue Managers).

This tag is best used for cross-platform message queuing questions. For questions about specific products, use or include the tag for that product:

3929 questions
86
votes
4 answers

RabbitMQ - Message order of delivery

I need to choose a new Queue broker for my new project. This time I need a scalable queue that supports pub/sub, and keeping message ordering is a must. I read Alexis comment: He writes: "Indeed, we think RabbitMQ provides stronger ordering…
Bick
  • 17,833
  • 52
  • 146
  • 251
85
votes
7 answers

Performance comparison between ZeroMQ, RabbitMQ and Apache Qpid

I need a high performance message bus for my application so I am evaluating performance of ZeroMQ, RabbitMQ and Apache Qpid. To measure the performance, I am running a test program that publishes say 10,000 messages using one of the message queue…
ahsankhan
  • 951
  • 1
  • 7
  • 5
81
votes
2 answers

NServiceBus and Rabbit MQ or Kafka

I am trying to learn messaging system. I have found that RabbitMq and NServiceBus are using together in few places. My questions are If I am using the RabbitMQ then why do i need NServiceBus? and vice versa What NServiceBus can do but RabbitMQ or…
MJK
  • 3,434
  • 3
  • 32
  • 55
79
votes
5 answers

Are message queues obsolete in linux?

I've been playing with message queues (System V, but POSIX should be ok too) in Linux recently and they seem perfect for my application, but after reading The Art of Unix Programming I'm not sure if they are really a good…
Purple Tentacle
  • 1,606
  • 1
  • 17
  • 19
75
votes
1 answer

Long running REST API with queues

We are implementing a REST API, which will kick off multiple long running backend tasks. I have been reading the RESTful Web Services Cookbook and the recommendation is to return HTTP 202 / Accepted with a Content-Location header pointing to the…
73
votes
5 answers

Is Apache Kafka appropriate for use as an unordered task queue?

Kafka splits incoming messages up into partitions, according to the partition assigned by the producer. Messages from partitions then get consumed by consumers in different consumer groups. This architecture makes me wary of using Kafka as a…
morsecoder
  • 1,479
  • 2
  • 15
  • 24
73
votes
5 answers

Effective strategy to avoid duplicate messages in apache kafka consumer

I have been studying apache kafka for a month now. I am however, stuck at a point now. My use case is, I have two or more consumer processes running on different machines. I ran a few tests in which I published 10,000 messages in kafka server. Then…
Shades88
  • 7,934
  • 22
  • 88
  • 130
72
votes
2 answers

RabbitMQ - How many queues can RabbitMQ handle on a single server?

What's the maximum number of queues that RabbitMQ can handle on a single server? Does it depend on RAM? Does it depends on erlang processes?
Raj Sf
  • 1,408
  • 1
  • 17
  • 26
72
votes
2 answers

Rabbitmq or Gearman - choosing a jobs queue

At work, we need to build a jobs server for things like sending emails, building PDFs, crunching some data, etc. Obviously, we'd like to build on some sort of generic queueing system. I'm familiar with Gearman, and that this is exact problem that…
brianz
  • 7,268
  • 4
  • 37
  • 44
71
votes
4 answers

ActiveMQ vs Apollo vs Kafka

I don't have any previous experience with *MQs and I'm looking to build knowledge on JMS and message queues in general. That way, I wonder whether I should start with ActiveMQ or just "ignore" it altogether and start by teaching myself Apollo. Is…
Martin
  • 1,317
  • 3
  • 13
  • 18
69
votes
1 answer

Comparison between RabbitMQ and MSMQ

Can I get the comparison between RabbitMQ and MSMQ. It will be helpful performance information on different factors are available.
N K
  • 3,217
  • 5
  • 23
  • 38
67
votes
1 answer

difference between pub-sub and push-pull pattern in zeroMq

This two images are from http://zguide.zeromq.org/page:all. What is the difference between this two pattern if we ignore sink in push-pull pattern ? Is there a difference in how a message gets transfer, if yes what is the difference ?
Bhuvan
  • 4,028
  • 6
  • 42
  • 84
64
votes
3 answers

Is there a performance difference between pooling connections or channels in rabbitmq?

I'm a newbie with Rabbitmq(and programming) so sorry in advance if this is obvious. I am creating a pool to share between threads that are working on a queue but I'm not sure if I should use connections or channels in the pool. I know I need…
Lostsoul
  • 25,013
  • 48
  • 144
  • 239
62
votes
5 answers

Topic Exchange vs Direct Exchange in RabbitMQ

We've got an application which will be using RabbitMQ and have several different queues for passing messages between tiers. Initially, I was planning to use multiple direct exchanges, with one for each message type, but it looks like having a single…
Mun
  • 14,098
  • 11
  • 59
  • 83
61
votes
9 answers

Check RabbitMQ queue size from client

Does anyone know if there's a way to check the number of messages in a RabbitMQ queue from a client application? I'm using the .NET client library.
Pablote
  • 4,745
  • 9
  • 39
  • 46