Questions tagged [producer-consumer]

The Producer-Consumer Problem (also known as the bounded-buffer problem) is a classical example of a multi-process synchronization problem. The problem describes two processes, the producer and the consumer, who share a common, fixed-size buffer used as a queue.

The Producer-Consumer Problem (also known as the bounded-buffer problem) is a classical example of a multi-process synchronization problem. The problem describes two processes, the producer and the consumer, who share a common, fixed-size buffer used as a queue. The producer's job is to generate a piece of data, put it into the buffer and start again. At the same time the consumer is consuming the data (i.e., removing it from the buffer) one piece at a time. The problem is to make sure that the producer won't try to add data into the buffer if it's full and that the consumer won't try to remove data from an empty buffer.

The solution for the producer is to either go to sleep or discard data if the buffer is full. The next time the consumer removes an item from the buffer, it notifies the producer who starts to fill the buffer again. In the same way, the consumer can go to sleep if it finds the buffer to be empty. The next time the producer puts data into the buffer, it wakes up the sleeping consumer. The solution can be reached by means of inter-process communication, typically using semaphores. An inadequate solution could result in a deadlock where both processes are waiting to be awakened. The problem can also be generalized to have multiple producers and consumers.

References

1564 questions
5
votes
2 answers

Two std::unique_lock used on same mutex causes deadlock ?

I found this code on code review stack exchange which implements a producer-consumer problem. I am posting a section of code here. In the given code, let's consider a scenario when producer produces a value by calling void add(int num), it acquires…
Sujith
  • 107
  • 6
5
votes
2 answers

TPL Dataflow - very fast producer, not so fast consumers OutOfMemory exception

I'm experimenting with TPL Dataflow before porting it into my production code. The production code is a classical producer/consumer system - producer(s) produce messages (related to financial domain), consumers process those messages. What I'm…
Michael
  • 2,961
  • 2
  • 28
  • 54
5
votes
1 answer

ActiveMQ Transport: tcp: Thread RUNNABLE state - too many threads hanging

Below ActiveMQ implementation is present in code. Sometimes, system stops working and become very slow. When I checked thread dump using JavaMelody - I have seen too many threads are on Runnable state for long time and is not being…
user1800979
  • 121
  • 6
5
votes
1 answer

Producer-Consumer model - binary semaphore or mutex?

This is mainly about the understanding of the concept, which confuses me. Mutex means that one thread takes the control of the access of shared resource, performs operations and unlocks it, then only other thread can gain access to lock while…
daydreamer
  • 87,243
  • 191
  • 450
  • 722
5
votes
1 answer

How to apply a concurrent solution to a Producer-Consumer like situation

I have a XML file with a sequence of nodes. Each node represents an element that I need to parse and add in a sorted list (the order must be the same of the nodes found in the file). At the moment I am using a sequential solution: struct Graphic { …
gliderkite
  • 8,828
  • 6
  • 44
  • 80
5
votes
3 answers

Best way to sequentially pass list values to single value consumer?

I'm toying with Java8's streams and CompletableFutures. My pre-existing code has a class that takes a single URL and downloads it: public class FileDownloader implements Runnable { private URL target; public FileDownloader(String target) { …
mabi
  • 5,279
  • 2
  • 43
  • 78
5
votes
1 answer

Kafka suddenly reset the consumer Offset

I'm working with Kafka 0.8 & zookeeper 3.3.5. Actually, we have a dozen of topic we are consuming without any issue. Recently, we started to feed and consume a new topic that has a weird behavior. The consumed offset was suddenly reset. It respects…
5
votes
1 answer

rabbitmq amqp - listening to ack messages from consumer

I have a producer and broker on the same machine. The producer sends messages like so: channel = connection.createChannel(); //Create a durable queue (if not already present) channel.queueDeclare(merchantId, true, false, false, null); //Publish…
arahant
  • 2,203
  • 7
  • 38
  • 62
5
votes
6 answers

kafka node, consumer got always old messages

im using the module kafka-node https://github.com/SOHU-Co/kafka-node and every time when i restart consumer they got all old messages, im using round robin system (load balancing) have you any idea how can i declare to server that i consumed a…
fadaytak
  • 83
  • 1
  • 2
  • 8
5
votes
1 answer

Modelling producer-consumer semantics with typeclasses?

If some entities in a system can function as producers of data or events, and other entities can function as consumers, does it make sense to externalize these "orthogonal concerns" into Producer and Consumer typeclasses? I can see that the Haskell…
James McCabe
  • 1,879
  • 2
  • 15
  • 22
5
votes
2 answers

What's the best way to asynchronously handle low-speed consumer (database) in high performance Java application

One EventHandler(DatabaseConsumer) of the Disruptor calls stored procedures in database, which is so slow that it blocks the Disruptor for some time. Since I need the Disruptor keep running without blocking. I am thinking adding an extra queue so…
5
votes
6 answers

How to implement blocking read using POSIX threads

I would like to implement a producer/consumer scenario that obeys interfaces that are roughly: class Consumer { private: vector read(size_t n) { // If the internal buffer has `n` elements, then dequeue them // Otherwise…
Frank Krueger
  • 69,552
  • 46
  • 163
  • 208
5
votes
3 answers

High throughput non-blocking server design: Alternatives to busy wait

I have been building a high-throughput server application for multimedia messaging, language of implementation is C++. Each server can be used in stand-alone mode or many servers can be joined together to create a DHT-based overlay network; the…
user2856296
5
votes
9 answers

producer - consumer multithreading in Java

I want to write program using multithreading wait and notify methods in Java. This program has a stack (max-length = 5). Producer generate number forever and put it in the stack, and consumer pick it from stack. When stack is full producer must wait…
Saeed Mirzaee
  • 51
  • 1
  • 1
  • 3
5
votes
1 answer

Java producer-consumer with stopping condition

I have N workers that share a queue of elements to compute. At each iteration, each worker removes an element from the queue and can produce more elements to compute, that are going to be put in the same queue. Basically, each producer is also a…
marcorossi
  • 1,941
  • 2
  • 21
  • 34