0

I have a single producer multiple consumer program with threads for each role. I am thinking of implementing a circular buffer for tcp on each of the consumers and allow the producer to keep pointers to the circular buffers' memory then handing out pointer space to the tcp to offload data into.

My problem, how to have consumer threads know when data is in?

I am thinking of busy wait checking the pointer location for something other than a 0; I don't mind being a cpu hog.

I should mention each thread is cpuset and soft RT by SCHED_FIFO, and of course c implemented.

BAR
  • 15,909
  • 27
  • 97
  • 185
  • are you doing this on linux? does it have to be portable? – mensi Jun 11 '12 at 18:53
  • Fedora/Rhel and no non portable, do you imply interrupts? – BAR Jun 11 '12 at 18:54
  • Can you elaborate a bit more on your setup? Is it data coming in on the producer that you want to evenly distribute among workers? Is the data in packets/work units? – mensi Jun 11 '12 at 18:59
  • Is the time until a consumer is ready to consume the next unit constant? (-> will round robin distribution of the units be fair enough?) – mensi Jun 11 '12 at 19:00
  • Io data will come off the net and producer should hand out to free consumers. Threads are bound to core no need to use linux scheduler – BAR Jun 11 '12 at 19:19
  • still, you're wasting power and producing heat if you are spinning for no reason. This might make a big difference in costs for power and cooling. – mensi Jun 11 '12 at 19:43
  • is there a reason you can't use pthread condition variables? – evil otto Jun 11 '12 at 20:26
  • Would it not be better to loop checking for data (non zero) in a specified mem location? – BAR Jun 11 '12 at 23:28

1 Answers1

1

In my experience, the problem with multiple consumer datastructures is to properly handle concurrency while avoiding issues with false sharing or excessivly wasting CPU cycles.

So if your problems allow it, I would use pipe to create a pipe to each consumer and putting items into these pipes in a round robin fashion. The consumers can then use epoll to watch the file handles. This avoids having to implement and optimize a concurrent datastructure and you won't burn CPU cycles needlessly. The cost is that you have to go through syscalls.

If you want to do everything yourself with polling to avoid syscalls, you can build a circular buffer but you have to make sure that only one process reads an item at the same time and only after the item has been written. Usually this is done with 4 pointers and proper mutexes.

This article about Xen's I/O ringbuffers might be of interest.

mensi
  • 9,580
  • 2
  • 34
  • 43
  • I want to avoid any and all syscalls possible. Mutex involves a syscall or two no? What would be wrong with having consumers constantly read a variable to check for 1 then process whatever data is in circular buffer, thereby negating the need for a mutex? – BAR Jun 11 '12 at 22:20
  • 1
    That will most likely result in all consumers taking the same piece of data, then work on it, repeat. Effectivly voiding the advantage of multiple threads since all do the same thing redundantly. Keep in mind you cannot read a pointer and write to it atomically without a lock. You could try to build something with the compare-and-swap CPU instruction but this is also not as simple as it sounds. Perhaps elaborate a bit more about what you are trying to achieve in your question, since the approach relies heavily on your requirements. – mensi Jun 11 '12 at 23:29
  • Low latency dedicated cores for my threads non portable linux. What more? – BAR Jun 12 '12 at 00:24