Questions tagged [producer-consumer]

The Producer-Consumer Problem (also known as the bounded-buffer problem) is a classical example of a multi-process synchronization problem. The problem describes two processes, the producer and the consumer, who share a common, fixed-size buffer used as a queue.

The Producer-Consumer Problem (also known as the bounded-buffer problem) is a classical example of a multi-process synchronization problem. The problem describes two processes, the producer and the consumer, who share a common, fixed-size buffer used as a queue. The producer's job is to generate a piece of data, put it into the buffer and start again. At the same time the consumer is consuming the data (i.e., removing it from the buffer) one piece at a time. The problem is to make sure that the producer won't try to add data into the buffer if it's full and that the consumer won't try to remove data from an empty buffer.

The solution for the producer is to either go to sleep or discard data if the buffer is full. The next time the consumer removes an item from the buffer, it notifies the producer who starts to fill the buffer again. In the same way, the consumer can go to sleep if it finds the buffer to be empty. The next time the producer puts data into the buffer, it wakes up the sleeping consumer. The solution can be reached by means of inter-process communication, typically using semaphores. An inadequate solution could result in a deadlock where both processes are waiting to be awakened. The problem can also be generalized to have multiple producers and consumers.

References

1564 questions
6
votes
3 answers

Java: Producer = Consumer, how to know when to stop?

I have several workers, that use ArrayBlockingQueue. Every worker takes one object from queue, process it, and in result can get several objects, that will be put into queue for further processing. So, worker = producer + consumer. Worker: public…
Oleg Golovanov
  • 905
  • 1
  • 14
  • 24
6
votes
5 answers

Java LinkedBlockingQueue with ability to signal when done?

I have a situation of a single producer and single consumer working with a queue of objects. There are two situations when the queue might be empty: The consumer handled the objects quicker than the producer was capable of generating new objects…
Yon
  • 1,023
  • 4
  • 13
  • 23
6
votes
3 answers

Producer-consumer inter-thread communication

having trouble with inter-thread communication and "solved" it by using "dummy messages" all over the place. Is this a bad idea? What are possible solutions? Example Problem i have. main thread starts a thread for processing and inserting records…
beginner_
  • 7,230
  • 18
  • 70
  • 127
6
votes
6 answers

Is this Python producer-consumer lockless approach thread-safe?

I recently wrote a program that used a simple producer/consumer pattern. It initially had a bug related to improper use of threading.Lock that I eventually fixed. But it made me think whether it's possible to implement producer/consumer pattern in a…
Jasiu
  • 2,674
  • 2
  • 27
  • 27
6
votes
1 answer

Golang: How to tell whether producer or consumer is slower when communicating via buffered channels?

I have an app in Golang where I have a pipeline setup where each component performs some work, then pass along its results to another component via a buffered channel, then that component performs some work on its input then pass along its results…
Thomas Nguyen
  • 464
  • 1
  • 5
  • 16
6
votes
1 answer

Kafka consumer manual offset commit

In one of my use-case consist of consuming the data, do some operations and produce it to new topic. I'm using https://www.npmjs.com/package/kafkajs npm library. I would like to commit the offset manually after successful operations to avoid any…
Parveen Kumar
  • 397
  • 1
  • 5
  • 12
6
votes
2 answers

Producer/Consumer in NServiceBus command handler

Because NServiceBus doesn't seem to support adding a priority mechanism to the message queue, I want to implement this myself. Command handler (Producer): public void Handle(DoAnAction message) { _messageManager.Insert(message); } Single…
andy
  • 531
  • 5
  • 16
6
votes
1 answer

Why condition_variable is waiting for the lock in producer-consumer? C++

See the following classical producer-consumer code: int main() { std::queue produced_nums; std::mutex m; std::condition_variable cond_var; bool done = false; bool notified = false; std::thread producer([&]() { …
Eduard Rostomyan
  • 7,050
  • 2
  • 37
  • 76
6
votes
1 answer

How to update a running system using protobuf from int32 to 64?

We have a running/production system, where an int32 protobuf field is being communicated to and fro between components. We need to update it to int64 on the fly without getting the current system down. And deployment would also take a lot of time so…
6
votes
1 answer

Are SQL Server sequences thread safe?

Title is too broad but I couldn't find a more specific one, please feel free to change with better one. I have a table which is working with sequences instead identity. I have three producer applications which are concurrently insert into table, and…
ibubi
  • 2,469
  • 3
  • 29
  • 50
6
votes
2 answers

C# producer/consumer / observer?

I have a producer / consumer queue, except that there are specific types of objects. So not just any consumer can consume an added object. I don't want to make a specific queue for each type, as there are too many. (It sort of stretches the…
Xodarap
  • 11,581
  • 11
  • 56
  • 94
6
votes
4 answers

Java implementation of Producer Consumer throws java.lang.IllegalMonitorStateException

import java.util.LinkedList; import java.util.Queue; class Producer extends PubSub implements Runnable{ @Override public void run() { synchronized(queue){ if (queue.size() == 99){ try { …
station
  • 6,715
  • 14
  • 55
  • 89
6
votes
2 answers

Producer/Consumer in Java. Why do we need two conditions?

I've read about the problem on https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/locks/Condition.html. This is what the documentation says about the two conditions: We would like to keep waiting put threads and take threads in…
6
votes
3 answers

C# Await Multiple Events in Producer/Consumer

I'm implementing a data link layer using a producer/consumer pattern. The data link layer has its own thread and state machine to communicate the data link protocol over the wire (Ethernet, RS-232...). The interface to the physical layer is…
Brian Heilig
  • 592
  • 4
  • 16
6
votes
2 answers

Consumer(s)-Producer issue in webserver streaming an array of data

Producer-Consumer blog post states that: "2) Producer doesn't need to know about who is consumer or how many consumers are there. Same is true with Consumer." My problem is that I have an array of data that I need to get from the Webserver to…
smuggledPancakes
  • 9,881
  • 20
  • 74
  • 113