Questions tagged [message-queue]

"Message queue" is a design pattern or software engineering component that defines discipline or API for communication between two or more interrelated processes or systems. The message queue enforces asynchronous processing and loose coupling. Depending on the implementation they may also provide delivery and order guarantees of the messages. Message processing guarantees are explicitly excluded from the design.

Message queue design pattern introduces an intermediary between the producer of messages and its consumer(s). This intermediary - the queue - accepts the messages and stores them until a consumer retrieves them or until some other event (e.g. expiration time). In general the producer does not know or care where, how, and when the messages in the queue will be consumed.

Software components such as MSMQ or IBM MQ or Tibco (usually called Queue Managers) provide queue storage and features for manipulation of messages in queues. These manipulations include: expiration, persistence, notification of expired messages (dead-letter queues), message cloning, queue security, distribution (between multiple Queue Managers).

This tag is best used for cross-platform message queuing questions. For questions about specific products, use or include the tag for that product:

3929 questions
26
votes
7 answers

Error "unknown delivery tag" occurs when i try ack messages to RabbitMQ using pika (python)

I want process messages in few threads but i'm getting error during execute this code: from __future__ import with_statement import pika import sys from pika.adapters.blocking_connection import BlockingConnection from pika import connection,…
solo117
  • 1,069
  • 2
  • 11
  • 12
26
votes
3 answers

Advantages & Disadvantages of Brokered vs non-brokered Messaging Systems

I'm trying to design a real-time monitoring & control system that's modular, so it can distributed, and expanded/reconfigured for different hardware & networks. I've quickly come to the conclusion I'll need some kind of distributed enterprise…
Joe
  • 6,773
  • 2
  • 47
  • 81
26
votes
3 answers

What is the difference between a message channel and the message queue itself?

What is the difference between a message channel and a message queue itself? They're different things. The queue actually holds messages which will be processed (pushed to the listener) in FIFO manner. A channel is a medium through which…
alokraop
  • 853
  • 1
  • 11
  • 29
25
votes
2 answers

JMS - Going from one to multiple consumers

I have a JMS client which is producing messages and sending over a JMS queue to its unique consumer. What I want is more than one consumer getting those messages. The first thing that comes to my mind is converting the queue to a topic, so current…
25
votes
8 answers

Stats/Monitor/Inspector for beanstalkd

Does anyone know of an app that can monitor a beanstalkd queue? I'm looking for something that shows stats on tubes and jobs, and allows you to inspect the details. I'm not really picky about language/platform, just want to know if there's…
Tim Lytle
  • 17,549
  • 10
  • 60
  • 91
24
votes
1 answer

RabbitMQ / ActiveMQ or Redis for over 250,000 msg/s

Eventhough redis and message queueing software are usually used for different purposes, I would like to ask pros and cons of using redis for the following use case: group of event collectors write incoming messages as key/value . consumers fetch…
anonymous
  • 243
  • 1
  • 2
  • 4
24
votes
5 answers

In Kubernetes, how do I autoscale based on the size of a queue?

Suppose I have a RabbitMQ instance and a set of pods that pick messages from RabbitMQ and process them. How do I make Kubernetes increase the number of pods as the queue size increases? (I'm mentioning RabbitMQ, but that's just an example. Pick your…
Likk
  • 747
  • 3
  • 7
  • 8
24
votes
8 answers

Publish multiple messages to RabbitMQ from a file

Publishing single messages to a RabbitMQ queue can be easily done with the UI, by simply putting the message in the UI and clicking the "Publish Message" button. How do you publish a batch of messages? I have a file with messages to be sent to…
summerbulb
  • 5,709
  • 8
  • 37
  • 83
24
votes
4 answers

How to do error handling with EasyNetQ / RabbitMQ

I'm using RabbitMQ in C# with the EasyNetQ library. I'm using a pub/sub pattern here. I still have a few issues that I hope anyone can help me with: When there's an error while consuming a message, it's automatically moved to an error queue. How…
Leon Cullens
  • 12,276
  • 10
  • 51
  • 85
24
votes
2 answers

Are there any MQ servers that can run embedded in a Java process?

I'm researching queuing solutions for one of my team's apps. Ideally we would like something that can be configured both as a lightweight, in-process broker (for low-throughput messaging between threads) and as an external broker. Is there an MQ…
Evan Haas
  • 2,524
  • 2
  • 22
  • 34
23
votes
3 answers

Are there any good alternatives to RabbitMQ for low-end setups?

I've been using RabbitMQ in a few setups now, and I can't shake the feeling that there must be something that's more easily set up. Despite it's conveniences it's hard to justify MQ for a solution that only processes a couple of thousand messages a…
lyschoening
  • 18,170
  • 11
  • 44
  • 54
23
votes
2 answers

Why is `multiprocessing.Queue.get` so slow?

I need help in understanding multiprocessing.Queue. The problem I'm facing is that getting results from queue.get(...) are hilariously behind compared to calls to queue.put(...) and the queue's buffer (the deque). This leaking abstraction led me to…
JBSnorro
  • 6,048
  • 3
  • 41
  • 62
23
votes
7 answers

Resubmitting a message from dead letter queue - Azure Service Bus

I have created a service bus queue in Azure and it works well. And if the message is not getting delivered within default try (10 times), it is correctly moving the message to the dead letter queue. Now, I would like to resubmit this message from…
Libin Joseph
  • 7,070
  • 5
  • 29
  • 52
23
votes
3 answers

Why BroadCastEvent are queued in Laravel? How to stop that?

I am working on the project which need to broadcast latitude and longitude on realtime I have something like below namespace App\Events; use App\Events\Event; use Illuminate\Queue\SerializesModels; use…
sumit
  • 15,003
  • 12
  • 69
  • 110
23
votes
11 answers

MSMQ v Database Table

An existing process changes the status field of a booking record in a table, in response to user input. I have another process to write, that will run asynchronously for records with a particular status. It will read the table record, perform some…
David White
  • 3,014
  • 1
  • 32
  • 35