Questions tagged [producer-consumer]

The Producer-Consumer Problem (also known as the bounded-buffer problem) is a classical example of a multi-process synchronization problem. The problem describes two processes, the producer and the consumer, who share a common, fixed-size buffer used as a queue.

The Producer-Consumer Problem (also known as the bounded-buffer problem) is a classical example of a multi-process synchronization problem. The problem describes two processes, the producer and the consumer, who share a common, fixed-size buffer used as a queue. The producer's job is to generate a piece of data, put it into the buffer and start again. At the same time the consumer is consuming the data (i.e., removing it from the buffer) one piece at a time. The problem is to make sure that the producer won't try to add data into the buffer if it's full and that the consumer won't try to remove data from an empty buffer.

The solution for the producer is to either go to sleep or discard data if the buffer is full. The next time the consumer removes an item from the buffer, it notifies the producer who starts to fill the buffer again. In the same way, the consumer can go to sleep if it finds the buffer to be empty. The next time the producer puts data into the buffer, it wakes up the sleeping consumer. The solution can be reached by means of inter-process communication, typically using semaphores. An inadequate solution could result in a deadlock where both processes are waiting to be awakened. The problem can also be generalized to have multiple producers and consumers.

References

1564 questions
0
votes
1 answer

Worker blocked on consume - RabbitMQ - Node.js

I've got a problem with RabbitMQ in Node.js . I'm trying to implement a Pub/Sub connector in which each user has its own queue to poll to get the messages. When I publish a message via Postma and the user consumes it, I have no problem (I get the…
JackLametta
  • 400
  • 1
  • 2
  • 21
0
votes
2 answers

What is the significance of condition.notify() in python's threading module?

So I'm using python's Condition from the threading module: from threading import Thread, Condition condition = Condition() I have a Producer class (subclass of Thread) that essentially in a for loop adds items to a queue until the queue is full…
0
votes
1 answer

How to correct index in circular array insertion

I am trying to find a formula for insert front in a dynamic circular array. One problem I have is when try to run display in which case the inx after the first two insertion will be off because there is an unoccupied index. This is assuming that I…
mokarab
  • 13
  • 1
  • 2
  • 4
0
votes
2 answers

Is this implementation of inter-process Producer Consumer correct and safe against process crash?

I am developing a message queue between two processes on Windows. I would like to support multiple producers and one consumer. The queue must not be corrupted by the crash of one of the processes, that is, the other processes are not effected by the…
David Sackstein
  • 500
  • 1
  • 5
  • 19
0
votes
1 answer

How to maintain order of execution of consumer threads

I am running code for one producer and multiple consumers. I want to prioritize execution of consumer threads. i.e. if I have consThread1, consThread2, consThread3. my question is how to restrict consThread3 to consume before consThread1 and…
Abhishek Nehe
  • 45
  • 1
  • 8
0
votes
1 answer

subprocess and shared self object in a call back

I have a typical multiple producers with a single consumer using python subprocess and one queue. The consumer has a callback to another object. Although the object is shared with all the subprocesses, when the subprocesses are finished, the changes…
RicLeal
  • 923
  • 9
  • 23
0
votes
1 answer

Does the IsCompleted property in the BlockingCollection actually block?

I am trying to implement a producer/consumer pattern for a stream of data that I am reading off of a controller asynchronously. I would like to use the BlockingCollection in order to do so, but want to make sure I get the desired results. My…
Snoop
  • 1,046
  • 1
  • 13
  • 33
0
votes
0 answers

backpressure .net reactive - fast obeservable

I am working to build a producer..consumer pattern using .NET reactive. Producer reads messages from Kafka message bus. Once the message is read, it needs to be handed over to the consumer to process the message. I was able to do that using .NET…
0
votes
1 answer

Java Threads | Producer Consumer : What's wrong with the code?

Trying to learn multithreading and inter-process communication of threads. Have implemented a typical producer-consumer problem. However, the output am getting is pretty sequential which should not be the case while working with threads ideally. Ok,…
roger_that
  • 9,493
  • 18
  • 66
  • 102
0
votes
0 answers

VB.NET Event Handler as the producer of a multithreaded producer-consumer pattern

The pattern below smells for a few reasons, but for the purposes of this question, let's focus on the global variables and the event handler: Dim objectQueue As New BlockingCollection(Of Object) Dim objectQueueCancel As New…
0
votes
1 answer

Producer consumer - Need to process queue when queue is full or after certain elapsed time

I am using Java as my programming interface. Here is my work flow. I have a producer who has a file name with path and adds it to queue. And then consumer who consumes it. After dequeue I add it to arraylist and then check if it has reached a count…
0
votes
2 answers

RabbitMQ: Are multiple consumers on one queue using a non-polling strategy possible?

we use RabbitMQ to send jobs from a producer on one machine, to a small group of consumers distributed across several machines. The producer generates jobs and places them on the queue, and the consumers check the queue every 10ms to see if there…
jkndrkn
  • 4,012
  • 4
  • 36
  • 41
0
votes
2 answers

Fastest way to synchronize work between a generating and consuming process in C#?

Update 1: It seems like BlockingCollection suggested below is the perfect fit, and I will try to apply that first thing tomorrow. Thank you for the replies. Update 2: Indeed it seems to perform admirably. Thanks again for all the help. __ I have…
knut
  • 689
  • 5
  • 13
0
votes
1 answer

Produce 1 million message/seconds with kafka

I have a requirement to prove kafka producer can produce 1 million message/second to Kafka cluster and then evaluate its performance. How can I achieve produce 1 million message/sec? public static void main(String args[]){ Random rnd = new…
rania triki
  • 191
  • 2
  • 11
0
votes
0 answers

Is it possible to block on wait on a semaphore without using the data when it's available?

The software I'm working on is a data analyzer with a sliding window. I have 2 threads, one producer and one consumer, that use a circular buffer. The consumer must process data only if the first element in the buffer is old enough, therefore there…
rookie coder
  • 141
  • 1
  • 8