Questions tagged [producer-consumer]

The Producer-Consumer Problem (also known as the bounded-buffer problem) is a classical example of a multi-process synchronization problem. The problem describes two processes, the producer and the consumer, who share a common, fixed-size buffer used as a queue.

The Producer-Consumer Problem (also known as the bounded-buffer problem) is a classical example of a multi-process synchronization problem. The problem describes two processes, the producer and the consumer, who share a common, fixed-size buffer used as a queue. The producer's job is to generate a piece of data, put it into the buffer and start again. At the same time the consumer is consuming the data (i.e., removing it from the buffer) one piece at a time. The problem is to make sure that the producer won't try to add data into the buffer if it's full and that the consumer won't try to remove data from an empty buffer.

The solution for the producer is to either go to sleep or discard data if the buffer is full. The next time the consumer removes an item from the buffer, it notifies the producer who starts to fill the buffer again. In the same way, the consumer can go to sleep if it finds the buffer to be empty. The next time the producer puts data into the buffer, it wakes up the sleeping consumer. The solution can be reached by means of inter-process communication, typically using semaphores. An inadequate solution could result in a deadlock where both processes are waiting to be awakened. The problem can also be generalized to have multiple producers and consumers.

References

1564 questions
18
votes
4 answers

How to make worker threads quit after work is finished in a multithreaded producer-consumer pattern?

I am trying to implement a multithreaded producer-consumer pattern using Queue.Queue in Python 2.7. I am trying to figure out how to make the consumers, i.e. the worker threads, stop once all required work is done. See the second comment by Martin…
Lone Learner
  • 18,088
  • 20
  • 102
  • 200
18
votes
4 answers

Go: One producer many consumers

So I have seen a lot of ways of implementing one consumer and many producers in Go - the classic fanIn function from the Concurrency in Go talk. What I want is a fanOut function. It takes as a parameter a channel it reads a value from and returns a…
Carl
  • 43,122
  • 10
  • 80
  • 104
17
votes
2 answers

Find out if I'm on the unity thread

How can I check if the thread I'm on is the Unity thread? I tried capturing the threadId at constructor time, but somewhere along the lifetime of the program, the threadId moves. In my project, some secondary thread processes need access to a newly…
Boris Callens
  • 90,659
  • 85
  • 207
  • 305
16
votes
1 answer

Apple doc's GCD Producer-Consumer solution wrong?

In the Migrating Away from Threads section of Apple's Concurrency Programming Guide, there is Changing Producer-Consumer Implementations, which claims that the typical multistep pthread mutex + condition variable implementation can be simplified…
16
votes
3 answers

How to consume a BlockingCollection in batches

I've come up with some code to consume all wating items from a queue. Rather than processing the items 1 by 1, it makes sense to process all waiting items as a set. I've declared my queue like this. private BlockingCollection items = new…
Jodrell
  • 34,946
  • 5
  • 87
  • 124
15
votes
2 answers

Python - simple reading lines from a pipe

I'm trying to read lines from a pipe and process them, but I'm doing something silly and I can't figure out what. The producer is going to keep producing lines indefinitely, like this: producer.py import time while True: print 'Data' …
Kiv
  • 31,940
  • 6
  • 44
  • 59
15
votes
3 answers

Multiple producers, single consumer

I have to develop a multithreaded application, where there will be multiple threads, each thread generates custom event log which need to be saved in queue (not Microsoft MSMQ). There will be another thread which reads log data from queue and…
Vikram Ranabhatt
  • 7,268
  • 15
  • 70
  • 133
15
votes
2 answers

Difference between SynchronousQueue vs TransferQueue

What is the difference between these two implementations? In which cases should be used one over another?
fashuser
  • 2,152
  • 3
  • 29
  • 51
14
votes
3 answers

Error: Could not find or load main class config.zookeeper.properties

I am trying to execute a sample producer consumer application using Apache Kafka. I downloaded it from https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.0.0/kafka-0.10.0.0-src.tgz . Then I started following the steps given in…
Sanjay
  • 443
  • 1
  • 7
  • 22
14
votes
5 answers

boost c++ lock-free queue vs shared queue

I'm quite new in multithreading programming, I just know the most common Producer-Consumer-Queue. I'm using the boost c++ libraries and I don't know if is better use boost::lockfree::queue or a wrapper class around std::queue that is using `mutex`…
Elvis Dukaj
  • 7,142
  • 12
  • 43
  • 85
13
votes
1 answer

Calling Dispose on an BlockingCollection

I've reused the example producer consumer queue from the C# in a Nutshell book of Albahari (http://www.albahari.com/threading/part5.aspx#_BlockingCollectionT) and a colleague remarked: "Why isn't the Dispose called on the BlockingCollection in the…
Sander
  • 616
  • 1
  • 6
  • 17
13
votes
1 answer

Implementing the Producer/Consumer Pattern in C#

How can I implement the Producer/Consumer patterns in C# using Events and Delegates? What do I need to keep an eye out for when it comes to resources when using these design patterns? Are there any edge cases I need to be aware of?
tush1r
  • 19,443
  • 14
  • 36
  • 35
13
votes
4 answers

Producer-Consumer Queue in AngularJS

I know python and databases since several years ago. But I want to improve my limited JavaScript knowledge. For my toy project I want to use an asynchronous queue in the web browser and use AngularJS for this. In python there is a nice class called…
guettli
  • 25,042
  • 81
  • 346
  • 663
12
votes
3 answers

C++ Producer consumer queue with (very) fast and reliable handover

Hi I am looking into having thread handover using a fast and reliable producer consumer queue. I am working on Windows with VC++. I based my design on Anthony Williams queue, that is, basically a boost::mutex with a boost::condition_variable. Now…
Cookie
  • 12,004
  • 13
  • 54
  • 83
12
votes
1 answer

Why would I place a synchronized block within a single-threaded method?

I stumbled upon this article on IBM - developerworks, and the code they posted had me raise some questions: Why is the building of the local variable Map wrapped within a synchronized block? Note that they implicitly say there is only one producer…
payloc91
  • 3,724
  • 1
  • 17
  • 45