Questions tagged [producer-consumer]

The Producer-Consumer Problem (also known as the bounded-buffer problem) is a classical example of a multi-process synchronization problem. The problem describes two processes, the producer and the consumer, who share a common, fixed-size buffer used as a queue.

The Producer-Consumer Problem (also known as the bounded-buffer problem) is a classical example of a multi-process synchronization problem. The problem describes two processes, the producer and the consumer, who share a common, fixed-size buffer used as a queue. The producer's job is to generate a piece of data, put it into the buffer and start again. At the same time the consumer is consuming the data (i.e., removing it from the buffer) one piece at a time. The problem is to make sure that the producer won't try to add data into the buffer if it's full and that the consumer won't try to remove data from an empty buffer.

The solution for the producer is to either go to sleep or discard data if the buffer is full. The next time the consumer removes an item from the buffer, it notifies the producer who starts to fill the buffer again. In the same way, the consumer can go to sleep if it finds the buffer to be empty. The next time the producer puts data into the buffer, it wakes up the sleeping consumer. The solution can be reached by means of inter-process communication, typically using semaphores. An inadequate solution could result in a deadlock where both processes are waiting to be awakened. The problem can also be generalized to have multiple producers and consumers.

References

1564 questions
8
votes
1 answer

Does Producer/Consumer model equal Actor?

So lately I've been reading a lot of article about how concurrent programming is hard, and how concurrent programming with shared state is near impossible. So languages like Erlang (I think this is on, if not the question still makes sense) use the…
Gandalf
  • 9,648
  • 8
  • 53
  • 88
8
votes
1 answer

Looking for "consumer that returns value" abstraction in Java

Is there a built-in or robust third-party abstraction for consumer that returns value in Java 8+? P.S. For deferred execution it may return Future as well. Update. Function interface has a perfect syntactic match, but there is a consideration around…
Denis Kulagin
  • 8,472
  • 17
  • 60
  • 129
8
votes
1 answer

Semaphore in Ada

This is an assignment and I have been asked to implement a Semaphore in Ada as the description below. However I have implemented the Semaphore.adb and called this Semaphore in the producerconsumer_sem.adb which I created. I get some output which is…
Adam
  • 856
  • 2
  • 9
  • 18
8
votes
3 answers

does Monitor.Wait Needs synchronization?

I have developed a generic producer-consumer queue which pulses by Monitor in the following way: the enqueue : public void EnqueueTask(T task) { _workerQueue.Enqueue(task); Monitor.Pulse(_locker); } the dequeue: private…
user437631
  • 1,112
  • 4
  • 12
  • 28
8
votes
2 answers

Producer work consistently hashing to consumers via a message queue?

I have a producer that I want to distribute work consistently across consumers by consistent hashing. For example, with consumer nodes X and Y, tasks A, B, C should always go to consumer X, and D, E, F to consumer Y. But that may shift a little if Z…
Bluu
  • 5,226
  • 4
  • 29
  • 34
8
votes
4 answers

Efficient consumer thread with multiple producers

I am trying to make a producer/consumer thread situation more efficient by skipping expensive event operations if necessary with something like: //cas(variable, compare, set) is atomic compare and swap //queue is already lock free running =…
iam
  • 1,623
  • 1
  • 14
  • 28
8
votes
1 answer

LMAX Disruptor - what determines the batch size?

I have been recently learning about the LMAX Disruptor and been doing some experimentation. One thing that is puzzling me is the endOfBatch parameter of the onEvent handler method of the EventHandler. Consider my following code. First, the dummy…
Asif Iqbal
  • 4,562
  • 5
  • 27
  • 31
8
votes
2 answers

Asynchronously consuming pipe with bash

I have a bash script like this data_generator_that_never_guits | while read data do an_expensive_process_with data done The first process continuously generates events (at irregular intervals) which needs to be processed as they become available.…
MrMobster
  • 1,851
  • 16
  • 25
8
votes
2 answers

Buffered Background InputStream Implementations

I've written background InputStream (and OutputStream) implementations that wrap other streams, and read ahead on a background thread, primarily allowing for decompression/compression to happen in different threads from the processing of the…
Stephen Denne
  • 36,219
  • 10
  • 45
  • 60
8
votes
3 answers

How to implement consumer-producer with multiple consumers and multiple queues

Assume there are 1 producer P and 2 consumers C1 and C2. And there are 2 queues Q1 and Q2, both with a specific capacity. P will produce items and put it into Q1 and Q2 alternately. Item is produced for specific consumer and cannot be consumed by…
8
votes
2 answers

Looking for the right ring buffer implementation in C

I am looking for a ring buffer implementation (or pseudocode) in C with the following characteristics: multiple producer single consumer pattern (MPSC) consumer blocks on empty producers block on full lock-free (I expect high contention) So far…
ziu
  • 2,634
  • 2
  • 24
  • 39
8
votes
1 answer

Thread count growth when using Task Parallel Library

I'm using C# TPL and I'm having a problem with a producer/consumer code... for some reason, TPL doesn't reuse threads and keeps creating new ones without stopping I made a simple example to demonstrate this behavior: class Program { static…
8
votes
1 answer

How do I use a BlockingCollection in the Producer/Consumer pattern when the producers are also the consumers - How do I end?

I have a recursive problem where the consumers do some work at each level of a tree, then need to recurse down the tree and perform that same work at the next level. I want to use ConcurrentBag/BlockingCollection etc to run this in parallel. In this…
Jason Coyne
  • 6,509
  • 8
  • 40
  • 70
8
votes
3 answers

How to wrap ConcurrentDictionary in BlockingCollection?

I try to implement a ConcurrentDictionary by wrapping it in a BlockingCollection but did not seem to be successful. I understand that one variable declarations work with BlockingCollection such as ConcurrentBag, ConcurrentQueue, etc. So, to…
7
votes
1 answer

RabbitMQ transfer rates speed up?

I look for ideas how to speed up message transfers through RabbitMQ. I installed the latest version on Windows 64 bit, running a server on my local machine on which I also publish and consume to/from through a C# implementation. I initially maxed…
Matt
  • 7,004
  • 11
  • 71
  • 117