Questions tagged [message-queue]

"Message queue" is a design pattern or software engineering component that defines discipline or API for communication between two or more interrelated processes or systems. The message queue enforces asynchronous processing and loose coupling. Depending on the implementation they may also provide delivery and order guarantees of the messages. Message processing guarantees are explicitly excluded from the design.

Message queue design pattern introduces an intermediary between the producer of messages and its consumer(s). This intermediary - the queue - accepts the messages and stores them until a consumer retrieves them or until some other event (e.g. expiration time). In general the producer does not know or care where, how, and when the messages in the queue will be consumed.

Software components such as MSMQ or IBM MQ or Tibco (usually called Queue Managers) provide queue storage and features for manipulation of messages in queues. These manipulations include: expiration, persistence, notification of expired messages (dead-letter queues), message cloning, queue security, distribution (between multiple Queue Managers).

This tag is best used for cross-platform message queuing questions. For questions about specific products, use or include the tag for that product:

3929 questions
1
vote
0 answers

Preserve messages during unprecedented JVM crash in a high throughput system

I am building a high volume system that will be processing up to a hundred million messages everyday. I have a microservice that is reading from a Kafka topic and doing some basic processing on them before forwarding them to the next…
1
vote
1 answer

parallel job processing with Azure Service Bus Queues

I have an Azure Service Bus queue with Rest API requests for Database ingestion jobs. For this, I have a requirement to create task workers to complete each job asynchronously as given in the below diagram. I want to process these messages in…
1
vote
2 answers

MSMQ Private queue automatically remove unread message beyond 24 hours

I use MSMQ through WCF to store messages in private queue. (private queue hosted by Windows Server 2003). Messages are stored only for 24 hours and automatically removed from queue, somebody know why?
fred
  • 21
  • 1
  • 4
1
vote
1 answer

Is having a type property in a queue message an indication of a bad design in RabbitMQ?

I’m trying to better understand the nuances of designing a distributed system using queues in general and RabbitMQ in particular. Let say I have messages like { "id": 12, "name": “John” "role": “Employee” } and { "id": 13, "name":…
Vahid
  • 1,625
  • 1
  • 18
  • 33
1
vote
0 answers

Central place for RabbitMQ queue binding management in a microservice-based app

I'm working on a microservice-based application. It is built of ca. 30 microservices and as usually some of them communicate with each other over RabbitMQ. Such services have names of exchanges, queues and routing keys in their configuration file…
ka3ak
  • 2,435
  • 2
  • 30
  • 57
1
vote
1 answer

Why were the terminologies "producer" and "consumer" adopted in Kafka?

As far as I know, Kafka has been implemented based on the architecture of "Publish-Subscribe Pattern". I took a look at this concept, and found out that it's quite similar to "Message Queueing Pattern(Producer-Consumer Pattern)" but simultaneously…
1
vote
0 answers

Best way for message structure for Kafka Events for CUD operations to a table

I am creating three different event types for Kafka or any queue for an insert, update or delete to a table in SQL server. I am thinking of how the message should be structured. What is the best way to structure these messages in Kafka or any…
Learn AspNet
  • 1,192
  • 3
  • 34
  • 74
1
vote
1 answer

MSMQ Thread Safety

I'm working with message queues in c#. I'm sending messages to a queue from code called by a timer (set up in the main thread), and also from a background worker thread. If I have two instances of the message queue, one for the timer and one for the…
macleojw
  • 4,113
  • 10
  • 43
  • 63
1
vote
1 answer

Redis-cli - list of running queues command?

I want to see the List of queues in the Redis server using Redis-cli. I am using this command to just monitor the queue. redis-cli MONITOR | grep queuename Please tell me if we have the any cli comamnd which meet my requirement.
Ojer_Dev
  • 108
  • 1
  • 12
1
vote
2 answers

Messages behavior after fork()

I'm learning interprocess communication through messages. In the following code snippet, I apply fork() and send messages between the parent and the child processes. I expect "1 - 2 - 3 - 4" console output. However, I got "1 - 2", and after this,…
pigeon_gcc
  • 13
  • 6
1
vote
1 answer

How safe is casting a struct to uint8_t * or char * and accessing it via bytestream?

The following logic works fine but I'm uncertain of the caveats with what the standard says and whether it's totally safe to cast a struct to uint8_t * or char * to send to a message queue (which itself takes in a pointer to the buffer as well) or…
xyf
  • 664
  • 1
  • 6
  • 16
1
vote
2 answers

C# Object Passing over Network Stream with High Redundancy

I need to send a binary payload with metadata over an unconventional slow, low-bandwidth, jittery connection which can basically be treated like any other normal connection at and above Layer 3. I expect the binary payload to be no more than 512kb…
1
vote
1 answer

How to close/clear an IPC message queue?

I have this call in the cleanup of the main thread of process_B which receives messages on an IPC message queue: if (msgctl(qId, IPC_RMID, NULL) < 0) { perror("msgctl"); } And when reached, reports the following: msgctl : Invalid…
Ender
  • 1,652
  • 2
  • 25
  • 50
1
vote
0 answers

Boost interprocess message_queue and COM

I'm using boost::interprocess::message_queue for communication between processes. In one of them i use ::CoInitializeEx( 0, COINIT_MULTITHREADED ); to set up for COM calls. I cannot access any queues created after this call from the other process,…
erik
  • 111
  • 1
  • 6
1
vote
0 answers

Capacity Planning for Task/Message Queue

There is a celery task in Django setup which uses Redis as a backend. I want to improve throughput, responsiveness, and predict task completion time for capacity planning. The task takes the data from the internal system and syncs it to an external…
Kracekumar
  • 19,457
  • 10
  • 47
  • 56