0

I have a problem with producers and consumers.

I have one producer that produces data very fast and it takes data with a specific period from an external server. I have three consumers which need that data. The consumers do some processing on this data, so consumers are slower than the producer.

I use a queue to pass data and after a specific size I am blocking the producer and consumers with mutexes and conditional variables. But when the conditional_variable waits, the data that came from the server will be updated and when I wait I miss some data. How can this be avoided?

Martin Atkins
  • 62,420
  • 8
  • 120
  • 138
barp
  • 6,489
  • 9
  • 30
  • 37
  • 1
    Make your consumers faster! Profile & optimise. Or buy a faster computer. – Nigel Harper Feb 02 '14 at 16:59
  • 1
    There's a simple logical error in your reasoning. Once you obtain data, it will inevitably become stale. You didn't quantify the severeness of the staleness. Is it bad that it is out of data by a microsecond? A second? How can you guarantee that you have an upper limit at all? Perhaps the consumer needs to query it instead of the producer to set a reasonable upper bound at all. – Hans Passant Feb 02 '14 at 17:00
  • From this description you've got inbound data and no place to put it. The only answer is to find or make room somewhere. Either compress the queued data to give yourself a larger effective buffer or spill it to disk, either way this is going to get very sophisticated, very fast, and the step from there to something like ZeroMQ starts to look like a good one to take. – jthill Feb 02 '14 at 17:14

1 Answers1

0

You can't block the producer's capture of data without losing information, so all you can handshake on is the communications. Either the producer or the consumer or both need buffering so arriving data can be captured at full speed and then delivered to the consumer's data processing at a rate it can keep up with.

Depending on what you're doing, it may actually be fine to lose some intermediate state when later events make it obsolete.

keshlam
  • 7,931
  • 2
  • 19
  • 33