Questions tagged [producer-consumer]

The Producer-Consumer Problem (also known as the bounded-buffer problem) is a classical example of a multi-process synchronization problem. The problem describes two processes, the producer and the consumer, who share a common, fixed-size buffer used as a queue.

The Producer-Consumer Problem (also known as the bounded-buffer problem) is a classical example of a multi-process synchronization problem. The problem describes two processes, the producer and the consumer, who share a common, fixed-size buffer used as a queue. The producer's job is to generate a piece of data, put it into the buffer and start again. At the same time the consumer is consuming the data (i.e., removing it from the buffer) one piece at a time. The problem is to make sure that the producer won't try to add data into the buffer if it's full and that the consumer won't try to remove data from an empty buffer.

The solution for the producer is to either go to sleep or discard data if the buffer is full. The next time the consumer removes an item from the buffer, it notifies the producer who starts to fill the buffer again. In the same way, the consumer can go to sleep if it finds the buffer to be empty. The next time the producer puts data into the buffer, it wakes up the sleeping consumer. The solution can be reached by means of inter-process communication, typically using semaphores. An inadequate solution could result in a deadlock where both processes are waiting to be awakened. The problem can also be generalized to have multiple producers and consumers.

References

1564 questions
7
votes
5 answers

Muti-threaded access to the same text file

I have a huge line-separated text file and I want to make some calculations on each line. I need to make a multithreaded program to process it because it is the processing of each line that takes the most time to complete rather than reading each…
Alexandros
  • 4,425
  • 4
  • 23
  • 21
7
votes
3 answers

Producer Consumer - Using Executors.newFixedThreadPool

My understanding of a Producer-Consumer pattern is that it could be implemented using a queue shared between the producer and the consumer. Producer submits work to a shared queue, consumer retrieves it and processes it. It could also be implemented…
Oxford
  • 139
  • 1
  • 2
  • 5
7
votes
2 answers

HornetQ messages still remaining in queue after consuming using core api

I am new to HornetQ so please bear with me. Let me first tell you my requirements : I need a messages queuing middleware which can pass messages , of about 1k in size, between different process with low latency and persistence(i.e. it should…
7
votes
3 answers

Why is asyncio queue await get() blocking?

Why is await queue.get() blocking? import asyncio async def producer(queue, item): await queue.put(item) async def consumer(queue): val = await queue.get() print("val = %d" % val) async def main(): queue = asyncio.Queue() …
7
votes
1 answer

How to take an item from any two BlockingCollections with priority to the first collection?

I have two BlockingCollection objects, collection1 and collection2. I want to consume items from these collections giving priority to items in collection1. That is, if both collections have items, I want to take items from collection1 first. If…
Yacoub Massad
  • 27,509
  • 2
  • 36
  • 62
7
votes
1 answer

Kafka consumer: fetching topic metadata for topics from broker [ArrayBuffer(id:0,host:user-Desktop,port:9092)] failed

Tried to run Kafka producer in one machine and consumer in another machine. Set the following properties: advertised.host.name advertised.port But getting the following error on console consumer: bin/kafka-console-consumer.sh --zookeeper ip:2181…
GKVM
  • 335
  • 1
  • 5
  • 17
7
votes
5 answers

Java: High-performance message-passing (single-producer/single-consumer)

I initially asked this question here, but I've realized that my question is not about a while-true loop. What I want to know is, what's the proper way to do high-performance asynchronous message-passing in Java? What I'm trying to do... I have…
Mr. Burgundy
  • 351
  • 1
  • 4
  • 5
7
votes
3 answers

How to aggregate the data from an async producer and write it to a file?

I'm learning about async/await patterns in C#. Currently I'm trying to solve a problem like this: There is a producer (a hardware device) that generates 1000 packets per second. I need to log this data to a file. The device only has a ReadAsync()…
AlefSin
  • 1,086
  • 9
  • 20
7
votes
1 answer

C Confused on how to initialize and implement a pthread mutex and condition variable

I'm a little bit confused on how to initialize and implement a pthread mutex and condition variable. The goal of the program is to have producers place a set number of ints in a queue and consumers take the ints out of the queue. I must also be able…
7
votes
1 answer

Akka (1-node prod/cons): BalancingDispatcher being deprecated soon. What takes it's place?

According to "Effective Akka" balancing dispatcher is being deprecated soon. I'm gonna start working on some (single machine) producer/consumer code that deals with processing workload of drastically different shapes. What should I use ? I'd like…
Hassan Syed
  • 20,075
  • 11
  • 87
  • 171
7
votes
2 answers

Prioritized queues in Task Parallel Library

Is there any prior work of adding tasks to the TPL runtime with a varying priority? If not, generally speaking, how would I implement this? Ideally I plan on using the producer-consumer pattern to add "todo" work to the TPL. There may be times…
7
votes
6 answers

C# Producer/Consumer pattern

I have simple one-producer/two-consumers code as follows but the output shows that only C2 is consuming. Are there any bugs in my code? class Program { static void Main(string[] args) { Object lockObj = new object(); …
Southsouth
  • 2,659
  • 6
  • 32
  • 39
7
votes
1 answer

Spring Integration - Concurrent Service Activators

I have a queue channel, and a service activator with a poller which reads from that queue. I'd like to have configuration to say "I want 50 threads to poll that queue, and each time you poll and get a message back, on this thread, invoke the service…
DeejUK
  • 12,891
  • 19
  • 89
  • 169
7
votes
5 answers

producer - consume; how does the consumer stop?

So I have simulated my producer consumer problem and I have the code below. My question is this: how does the consumer stops if he's in constant while(true). In the code below, I've added if (queue.peek()==null) …
adhg
  • 10,437
  • 12
  • 58
  • 94
7
votes
2 answers

is it good to use BlockingCollection as single-producer, single-consumer FIFO query?

I need single-producer, single-consumer FIFO query because I need to process messages in the order they received. I need to do this asynchronous because caller should not wait while I'm processing message. Next message processing should be started…
Oleg Vazhnev
  • 23,239
  • 54
  • 171
  • 305