Questions tagged [producer-consumer]

The Producer-Consumer Problem (also known as the bounded-buffer problem) is a classical example of a multi-process synchronization problem. The problem describes two processes, the producer and the consumer, who share a common, fixed-size buffer used as a queue.

The Producer-Consumer Problem (also known as the bounded-buffer problem) is a classical example of a multi-process synchronization problem. The problem describes two processes, the producer and the consumer, who share a common, fixed-size buffer used as a queue. The producer's job is to generate a piece of data, put it into the buffer and start again. At the same time the consumer is consuming the data (i.e., removing it from the buffer) one piece at a time. The problem is to make sure that the producer won't try to add data into the buffer if it's full and that the consumer won't try to remove data from an empty buffer.

The solution for the producer is to either go to sleep or discard data if the buffer is full. The next time the consumer removes an item from the buffer, it notifies the producer who starts to fill the buffer again. In the same way, the consumer can go to sleep if it finds the buffer to be empty. The next time the producer puts data into the buffer, it wakes up the sleeping consumer. The solution can be reached by means of inter-process communication, typically using semaphores. An inadequate solution could result in a deadlock where both processes are waiting to be awakened. The problem can also be generalized to have multiple producers and consumers.

References

1564 questions
12
votes
3 answers

BlockingCollection that discards old data

I have a BlockingCollection. Producer tasks add items to it, and consumer tasks remove items. Now I want to limit the number of items in the collection, automatically discarding old data if more items are added. The collection should never contain…
HugoRune
  • 13,157
  • 7
  • 69
  • 144
12
votes
2 answers

Is this a job for TPL Dataflow?

I run a pretty typical producer/consumer model on different tasks. Task1: Reads batches of byte[] from binary files and kicks off a new task for each collection of byte arrays. (the operation is batched for memory management purposes). Task 2-n:…
Matt
  • 7,004
  • 11
  • 71
  • 117
11
votes
3 answers

producer-consumer problem with pthreads

I'm attempting to solve the producer-consumer problem using pthreads and semaphores, but it looks like the producer threads aren't producing, and the consumer threads aren't consuming. It appears that the threads are being created: /* Do actual…
Mike
  • 662
  • 3
  • 13
  • 27
11
votes
6 answers

Using pthread condition variable with rwlock

I'm looking for a way to use pthread rwlock structure with conditions routines in C++. I have two questions: First: How is it possible and if we can't, why ? Second: Why current POSIX pthread have not implemented this behaviour ? To understand my…
Doomsday
  • 2,650
  • 25
  • 33
11
votes
3 answers

producer/consumer work queues

I'm wrestling with the best way to implement my processing pipeline. My producers feed work to a BlockingQueue. On the consumer side, I poll the queue, wrap what I get in a Runnable task, and submit it to an ExecutorService. while…
10
votes
2 answers

Java thread wait and notify

I have two threads. Thread A is pulling some elements from queue and thread B is adding some elements to the queue. I want thread A to go to sleep when the queue is empty. When thread B adds some element to the queue it should make sure that thread…
mgamer
  • 13,580
  • 25
  • 87
  • 145
10
votes
1 answer

"The usage of semaphores is subtly wrong"

This past semester I was taking an OS practicum in C, in which the first project involved making a threads package, then writing a multiple producer-consumer program to demonstrate the functionality. However, after getting grading feedback, I lost…
Gautam Kamath
  • 203
  • 1
  • 5
10
votes
3 answers

RabbitMQ C# API Event based Message Consumption

while (true) { BasicDeliverEventArgs e = (BasicDeliverEventArgs)Consumer.Queue.Dequeue(); IBasicProperties properties = e.BasicProperties; byte[] body = e.Body; Console.WriteLine("Recieved Message : " +…
Jigar Sheth
  • 586
  • 2
  • 5
  • 21
10
votes
3 answers

C++11 non-blocking producer/consumer

I have a C++11 application with a high-priority thread that's producing data, and a low-priority thread that's consuming it (in my case, writing it to disk). I'd like to make sure the high-priority producer thread is never blocked, i.e. it uses only…
nonagon
  • 3,271
  • 1
  • 29
  • 42
10
votes
5 answers

What is the best way to communicate a kernel module with a user space program?

This question seems to be simple, but I want to send an event to notify my user space program that the module buffer is ready to be read. For example, I have a buffer in my kernel module and its data will be consumed by the user space program. If…
jcfaracco
  • 853
  • 2
  • 6
  • 21
9
votes
3 answers

Using a named mutex to lock a file

I'm using a named mutex to lock access to a file (with path 'strFilePath') in a construction like this: private void DoSomethingsWithAFile(string strFilePath) { Mutex mutex = new Mutex(false,strFilePath.Replace("\\","")); try { …
Jurgen
  • 99
  • 1
  • 2
9
votes
2 answers

How to subscribe exactly once to an element from AsyncSubject (consumer pattern)

In rxjs5, I have an AsyncSubject and want to subscribe to it multiple times, but only ONE subscriber should ever receive the next() event. All others (if they are not yet unsubscribed) should immediately get the complete() event without…
Dynalon
  • 6,577
  • 10
  • 54
  • 84
8
votes
3 answers

Generic .Net Producer/Consumer

I'm toying with the idea of implementing a generic Producer/Consumer pair + processing queue in C# for fun. The idea is you can just create objects that implement appropriate IProducer and IConsumer interfaces (default implementations supplied),…
Joel Coehoorn
  • 399,467
  • 113
  • 570
  • 794
8
votes
8 answers

Multi-threading to speed up an email-sending application

I have built an application to send email mailers for a website through Amazon SES. It is coded in C#. Each email takes .3 seconds to send via the Amazon SES API. That means, using a single-threaded application, I can only send 3 emails per…
8
votes
2 answers

Producer-Consumer waiting when queue is empty?

I have a list of work items that need to be processed in order. Sometimes the list will be empty, sometimes it will have a thousand items. Only one can be processed at a time and in order. Currently I am doing the following which to me looks stupid…
Peter
  • 1,685
  • 3
  • 16
  • 22