Questions tagged [message-queue]

"Message queue" is a design pattern or software engineering component that defines discipline or API for communication between two or more interrelated processes or systems. The message queue enforces asynchronous processing and loose coupling. Depending on the implementation they may also provide delivery and order guarantees of the messages. Message processing guarantees are explicitly excluded from the design.

Message queue design pattern introduces an intermediary between the producer of messages and its consumer(s). This intermediary - the queue - accepts the messages and stores them until a consumer retrieves them or until some other event (e.g. expiration time). In general the producer does not know or care where, how, and when the messages in the queue will be consumed.

Software components such as MSMQ or IBM MQ or Tibco (usually called Queue Managers) provide queue storage and features for manipulation of messages in queues. These manipulations include: expiration, persistence, notification of expired messages (dead-letter queues), message cloning, queue security, distribution (between multiple Queue Managers).

This tag is best used for cross-platform message queuing questions. For questions about specific products, use or include the tag for that product:

3929 questions
1
vote
1 answer

Kafka multiple consumers consuming from one partition - can the message be read twice?

I have a system with Kafka that looks like this (all consumers are within a single consumer group): Producer ---[ 1 topic, 1 partition] ---> Consumer1 |--> Consumer2 ... …
1
vote
2 answers

Message validation for async messaging systems

I'm looking for the best approach as to how I can go about doing validation of a message as its enqueued in async messaging based systems. Scenario: Let's say we have a two services A and B where they need to interact with each other asynchronously.…
1
vote
0 answers

Can a PostMessage to the same window be done from multiple threads simultaneously?

I'm using a dispatching mechanism in my Win32 application where non-main threads can post a piece of work wrapped in an std::function for later execution on the main thread. This is realized by creating a hidden window from the main thread, and…
1
vote
2 answers

How to Handle Long Running Consumption for SQS Application

I have a service which integrate with AWS SQS. Some of our messages take longer than 12 hour(maximum visibility timeout) to process. In this case, other consumers will pick up the same message after visibility timeout while the message is still…
1
vote
1 answer

Idempotency with side effects

Imagine I have a message queue on to which I place messages saying that I want to send a user an email. If I’m using a message broker that provides “at least once” delivery guarantees, then from all the resources I’ve been reading, they say “you…
whilefalse
  • 38
  • 3
1
vote
1 answer

Configuring socket timeout on amqplib connect in nodejs

If the RabbitMQ instance if found error then it takes about 120 seconds to timeout before trying to the error Here is my code used for connecting: async function connectAmqp() { try { // Create amqp connection…
Armenia
  • 95
  • 1
  • 8
1
vote
2 answers

Message queue IPC implementation on Linux

I have a multi-processed perl system that picks up a lot of files ( may be in 10 Millions ) and works on the them one by one Since each process must pick up exclusive files I need to have one seperate process that does the file-listing and sends the…
Ram
  • 21
  • 2
1
vote
1 answer

fastAPI + APScheduler not working asyncronously

I am trying to set up a fastAPI app doing the following: Accept messages as post requests and put them in a queue; A background job is, from time to time, pulling messages (up to a certain batch size) from the queue, processing them in a batch, and…
Marco Spinaci
  • 1,750
  • 15
  • 22
1
vote
0 answers

CamelCorrelationID/JMS CorrelationID implementation

I am confused how to implement CamelCorrelationID/JMSCorrelationID in a way where each application listen to its own response from server using messaging system. The queues are shared in this scenario. I have implemented my own CorrelationID to…
MFA
  • 23
  • 5
1
vote
0 answers

Intermittent librdkafka consumer crash with `malloc(): mismatching next->prev_size (unsorted)`

I am using single node Kafka as a message broker between a C application (model) and a Python application (controller). Both sides - i.e. model and controller produce and consume, from separate topics. The workflow is as below: C app (Model) waits…
Chintan Pathak
  • 302
  • 2
  • 5
  • 20
1
vote
2 answers

Azure Service Bus, using filters to assemble a large message broken up into smaller messages

I'm trying to find a solution for receiving large messages on Azure Service Bus. The essential pattern I was thinking is to publish a large messages in parts -- along with a correlation id, a page, and an "of". So if I have a four-part message, they…
BenW
  • 737
  • 10
  • 41
1
vote
1 answer

How to handle an Azure Function rerunning when using message queue binding?

I have a v1 Azure Function that is triggered by a message being written to the Azure Storage message queue. The Azure Function needs to perform multiple updates to SharePoint Online. Occasionally these operations fail. This results in the message…
Ivan Wilson
  • 403
  • 3
  • 12
1
vote
0 answers

IBM MQ correlation id length limit issue

I am using the .net client to send and receive IBM MQ messages. I am setting the correlation id property under MQMessage to send and receive. I am able to send a message to the queue when the correlation id length is small(Accurately 24 characters)…
1
vote
0 answers

Using RabbitMQ for Background Jobs?

I already have a group of background jobs in my web API handled by Hangfire, the client asks to use rabbitMQ for these background jobs however I think this will be no sense to use rabbitMQ for this purpose because my jobs didn't call any external…
1
vote
1 answer

Can JMS message-queue be used in this context for my use case?

I'm new to message queue system and read a bit about JMS in particular. This question was also helpful in understanding better about real world use case of JMS. Ours is a web-based application and am trying to find out whether a particular flow in…
Gnanam
  • 10,613
  • 19
  • 54
  • 72