Questions tagged [producer-consumer]

The Producer-Consumer Problem (also known as the bounded-buffer problem) is a classical example of a multi-process synchronization problem. The problem describes two processes, the producer and the consumer, who share a common, fixed-size buffer used as a queue.

The Producer-Consumer Problem (also known as the bounded-buffer problem) is a classical example of a multi-process synchronization problem. The problem describes two processes, the producer and the consumer, who share a common, fixed-size buffer used as a queue. The producer's job is to generate a piece of data, put it into the buffer and start again. At the same time the consumer is consuming the data (i.e., removing it from the buffer) one piece at a time. The problem is to make sure that the producer won't try to add data into the buffer if it's full and that the consumer won't try to remove data from an empty buffer.

The solution for the producer is to either go to sleep or discard data if the buffer is full. The next time the consumer removes an item from the buffer, it notifies the producer who starts to fill the buffer again. In the same way, the consumer can go to sleep if it finds the buffer to be empty. The next time the producer puts data into the buffer, it wakes up the sleeping consumer. The solution can be reached by means of inter-process communication, typically using semaphores. An inadequate solution could result in a deadlock where both processes are waiting to be awakened. The problem can also be generalized to have multiple producers and consumers.

References

1564 questions
0
votes
1 answer

Whats wrong with the code? Why are threads not interleaving?

class Producer implements Runnable{ //producer job here Vector sQueue; public Producer(Vector queue) { this.sQueue=queue; } public void run(){ int i=0; try{ …
0
votes
2 answers

Synchronizing producer, consumer and a producer queue

I have a producer and a consumer. Producer fills its internal queue with objects, consumer takes these objects one by one. I want to synchronize the cosumer with the producer, so that the consumer blocks when there are no objects ready, and I want…
zoul
  • 102,279
  • 44
  • 260
  • 354
0
votes
1 answer

How to get Message no of Messages per topic From Kafka using JAVA API

I would like to know how to get the number of messages per topic in kafka through java api, i don't know want to use the command line tool which is mentioned in the following post. Any idea how to do this? PS: i dont want to loop through the KAFKA…
amateur
  • 941
  • 4
  • 22
  • 33
0
votes
2 answers

Some confusion about blockingqueue and multithreaded consumer

I'm working on some multithreaded code like this (a lot of details omitted because of relevance): public class Producer { private static BlockingQueue transactionsQueue = new LinkedBlockingQueue(); private static…
Liumx31
  • 1,190
  • 1
  • 16
  • 33
0
votes
1 answer

Setting up multiple partitions in Apache Kafka

Im trying to set the no of partitions to 2 from the code,and i have single node setup, (1 zookeeper, 1kafka). when i consume the message i see that kafka is using only one partition to store the data, Do i need to make any modifications to the setup…
amateur
  • 941
  • 4
  • 22
  • 33
0
votes
1 answer

Best way to manage large "work queues" / "input queues"?

Our system has jobs that are consuming input queues containing ids of items that need to be taken as input for the jobs. There are a few thousand of those input queues and each queue contains from a few ten-thousands up to a few million ids. A job…
Peter Rietzler
  • 471
  • 5
  • 11
0
votes
1 answer

How can I have a producer and consumer communicate both ways?

I would like to have a worker thread wait for a command, do it, and then send the result back to the caller. This differs from the regular producer/consumer problem because of the response back. This is what I'm thinking: main { construct…
Josh
  • 313
  • 3
  • 9
0
votes
3 answers

Memory barrier in the implementation of single producer single consumer

The following implementation from Wikipedia: volatile unsigned int produceCount = 0, consumeCount = 0; TokenType buffer[BUFFER_SIZE]; void producer(void) { while (1) { while (produceCount - consumeCount == BUFFER_SIZE) …
0
votes
1 answer

Why doesn't Pascal FC want to run this code with monitors?

I'm programming concurrency in Pascal-FC using Eclipse Gavab 2.0. I haven't had any problems so far using it, as it always inform me of which the errors are when it's unable to execute a program. I did the producer-consumer problem using semaphores…
0
votes
2 answers

java producer consumer in batches

I am facing a problem in producer - consumer. My requirement is: Producer produces 100 objects together and wait for consumer to consume. Then Consumer consumes those 100 objects and wait for producer to produce. And this process repeats. The…
Dominic Philip
  • 108
  • 2
  • 10
0
votes
1 answer

Distribute the load of data on two kafka consumer using kafka.net client

How I can divide the load of data on my two kafka consumer using kafka .net client.
Nagu
  • 1
  • 1
0
votes
1 answer

How to send messages to same topic using kafka producer ?

I have setup kafka client that produce and consume messages, it is working as expected when we send payloads from producer to topic, So i have problem with producer now first message i was able to send it to topic and i was also able to consume it…
hussain
  • 6,587
  • 18
  • 79
  • 152
0
votes
1 answer

consumer.poll not fetching records from Kafka Topic having 3 partitions

Note: Kafka version used: kafka_2.11-0.9.0.1 I have 1 topic named test-kafka which as 3 partitions and 1 replication factor and this topic has some string data i.e. key and value pair in each partition. when I am trying to fetch records via consumer…
0
votes
2 answers

Consumer/Producer AWS SQS akka scala with synchrone consumer

My application have a producer and a consumer. My producer produces messages irregularly. Sometime my queue will be empty, sometime I will have a few messages. I would like to have my consumer listen to the queue and when a message is in it, take…
GermainGum
  • 1,349
  • 3
  • 15
  • 40
0
votes
2 answers

Simple producer-consumer example C# Dictionary

I make first steps in parallel programming. I wrote a simplest code, but it result confused. This code takes top 10 recent items with producer-consumer pattern. Only 1 consumer and 1 producer. With two consumers in works better, but incorrect too. …
Surgerer
  • 157
  • 9