Questions tagged [producer-consumer]

The Producer-Consumer Problem (also known as the bounded-buffer problem) is a classical example of a multi-process synchronization problem. The problem describes two processes, the producer and the consumer, who share a common, fixed-size buffer used as a queue.

The Producer-Consumer Problem (also known as the bounded-buffer problem) is a classical example of a multi-process synchronization problem. The problem describes two processes, the producer and the consumer, who share a common, fixed-size buffer used as a queue. The producer's job is to generate a piece of data, put it into the buffer and start again. At the same time the consumer is consuming the data (i.e., removing it from the buffer) one piece at a time. The problem is to make sure that the producer won't try to add data into the buffer if it's full and that the consumer won't try to remove data from an empty buffer.

The solution for the producer is to either go to sleep or discard data if the buffer is full. The next time the consumer removes an item from the buffer, it notifies the producer who starts to fill the buffer again. In the same way, the consumer can go to sleep if it finds the buffer to be empty. The next time the producer puts data into the buffer, it wakes up the sleeping consumer. The solution can be reached by means of inter-process communication, typically using semaphores. An inadequate solution could result in a deadlock where both processes are waiting to be awakened. The problem can also be generalized to have multiple producers and consumers.

References

1564 questions
0
votes
1 answer

Pentaho - How to run the job as stream

I am new to Pentaho. I have a requirement that i need to create a ETL job to read the message from rabbitMQ and transform it and store it in a db. I have created the ETL using IC AMQP plugin. But i couldn't find a way to run the job as stream. The…
Gowsikan
  • 5,571
  • 8
  • 33
  • 43
0
votes
1 answer

Process all messages before posting result

I am doing a kaafka assignment and the task is to generate random messages by players for given seasons/rounds/game. There is a consumer which listens to all these messages and will keep track of the scoring logic. Once the scoring is done, it will…
Em Ae
  • 8,167
  • 27
  • 95
  • 162
0
votes
0 answers

Yet another producer - consumer , C++

I have written the producer - consumer as follows: void producer(int n) { std::unique_lock lock(mtx); for (int i = 0; i < n; ++i) { cv.wait(lock, [] {return !notified; }); std::cout << "Producing.." << i <<…
Eduard Rostomyan
  • 7,050
  • 2
  • 37
  • 76
0
votes
1 answer

LMAX Disruptor Producer Incorrectly Wraparound + Overwrite Before Consumer Completes Read

I am recently introduced to the LMAX Disruptor and decided to give it a try. Thanks to the developers, the setup was quick and hassle free. But I think I am running into an issue if someone can help me with it. The issue: I was told that when the…
0
votes
1 answer

kafka consumer can't get previous unconsumed event

Step 1: create Topic with only one partition: bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test Step 2: Produce some message to topic test. Step 3: Start a consume on topic test. It can get…
IRON
  • 43
  • 1
  • 6
0
votes
1 answer

Producer-Consumer : Parallel Programming

my question is really simple : is this program valid as a simulation of the producer-consumer problem ? public class ProducerConsumer { public static void main(String[] args) { Consumers c = new Consumers(false, null); …
0
votes
0 answers

Producer and Consumer in java with consuming timeout

I want to write a CommunicationSupervisor for connection to a device with these features. Outgoing messages should be queued in a data structure. There is just one Consumer. Consumer should check if there is an element in queue. Consumer should…
Govan
  • 2,079
  • 4
  • 31
  • 53
0
votes
1 answer

Reading from file in producer/consumer model

I'm trying to read a string from a file, do an HTTP request with that string, and if the request returns a 200 then do another HTTP request with it. I thought a good model for this would be the producer consumer model, but for some reason I'm…
LivingRobot
  • 883
  • 2
  • 18
  • 34
0
votes
1 answer

Faster Producer in Producer/Consumer Queue Pattern

We have a MySQL table which gets about 5 million records written to it daily. Each record requires a bit of time to process some meta data associated with it. So we have a SINGLE "producer" process which send each record ID to a message queue…
phirschybar
  • 8,357
  • 12
  • 50
  • 66
0
votes
0 answers

Download many images asynchronously with SemaphoreSlim

I have a large number of images I'm try to download (and then resize and save). I'm trying to do this in the most efficient way possible. I have opted for to use a BlockingCollection and with a producer which downloads the images and a consumer that…
Andy Furniss
  • 3,814
  • 6
  • 31
  • 56
0
votes
0 answers

Halt producers until a consumer is free

Requirement: In a poller / worker scenario, pollers should stop polling remote service until a worker is available to take the task. Background: Due to throttling constraints on the number of requests to the remote service, I am trying to segregate…
0
votes
1 answer

I want to replicate race condition into deadlock using the Producer-Consumer model in Java

On wiki I found the following pseudo-code for a producer-consumer program which has a race condition that can lead into a deadlock: int itemCount = 0; procedure producer() { while (true) { item = produceItem(); if…
0
votes
1 answer

synchronous issue of multiple consumers and multiple producers using pthread conditional variables?

Three producers and each has a queue. Two consumers consume msg both from a specific queue and a common queue. How to synchronize them with pthread_cond?
wangsquirrel
  • 135
  • 2
  • 12
0
votes
1 answer

Shutting down producers/consumers in spring-integration with poison pill

How would one utilize poison pill to stop message handlers (and message suppliers) with spring-integration? I have a setup of N producers (subclassed Supplier) and M consumers (subclasses GenericHandler). They are connected via unbounded queue. …
ioreskovic
  • 5,531
  • 5
  • 39
  • 70
0
votes
1 answer

Multiple Producers and Multiple Consumers Java Semaphore Blocking Queue

We are trying to do multiple producers and multiple consumers around a semaphore lock queue. We are running into the issue of having more items produced than I told it to produce. Can someone please help me figure out where we are going wrong?…
Anker
  • 3
  • 3