How to buffer and write to disk a low latency input with C

Question

I need to write C code (not C# or C++) to get data from hardware analyzer (through 100Mb TCP/IP network) and write it to disk.

Why I say "Low latency", well, Hardware analyser have a 9KB internal buffer, this is 2ms of data storing, after that it runs into buffer overflow and I lose information packets.

My application is able to get this buffer without losing any packet, but I noticed I wasn't able to write data to the disk at this rate.

My code looks like this:

int main (int argc, char * argv [] )
{
  pthread_t th_rx; /* Thread for receiving packets */

  outputfile = fopen ("output.log", "wb");

  link_open(); // open device link

  pthread_create ( &th_rx, NULL, read_packets, 0 );

  // running loop
  fclose (outputfile);
  pthread_exit(NULL);
  link_close(); // close device link 
  return 0; 
}

static thread_result read_packets (void *arg)
{
  char  rxbuf[40960];

  while (receiving)
  {
    bytes_received = read_packet(); //read packet from device buffer
    rxbuf = extract_data(bytes_received);
    fwrite (rxbuf, 1, bytes_received, outputfile); 
  }
  return thread_return;
}

What I need here are ideas to how to do that.

1: don't write packet upon received, create a circular? buffer
2: Creating 2 thread circular buffer?

Any ideas of how to improve my code and what I can do to get stable writes?

Code examples will be very appreciated because I'm feeling lost :(

Thanks

Two threads, as suggested by @nos, yes. Also, use a producer-consumer queue of buffer object references/pointers to avoid the continual, and avoidable, bulk-copying. — Martin James, Nov 29 '13 at 12:06
This stackoverflow question can be relevant - http://stackoverflow.com/questions/20186859/c-low-latency-threaded-asynchronous-buffered-stream-intended-for-logging-b — Evgeny Lazin, Nov 29 '13 at 12:07

score 2 · Answer 1 · answered Nov 29 '13 at 11:37

2

As you guessed correctly, the firs thing you should do is, do not write to disk while you are reading from device.

create circular/ ring buffer (or just list of buffers) to store data read from device. This operation will be fast enough, so that device memory does not fill up.
- However, if device read is non-blocking you may want o decide to sleep so that your loop do not hog the cpu.
One thread reads data from device and puts in buffer.
In another thread, keep writing the buffer to disk.

answered Nov 29 '13 at 11:37

Rohan

52,392
12
90
87

Thanks Rohan, please can you enlighten me with some code examples? – hiroru Nov 29 '13 at 11:55
1

On quick google I got couple of useful links, you need to improve them for multiple threads http://en.wikipedia.org/wiki/Circular_buffer http://c.learncodethehardway.org/book/ex44.html – Rohan Nov 29 '13 at 12:01

Michael · Accepted Answer · 2013-11-29T12:55:02.827

1

As you said hw analyzer has 9 kb memory for only 2 ms. Then speed required for writing to HDD is 9*1024*500 = 4608000 b/s or 4.6 Mb/s. I believe that test computer has speed of writing at HDD at least 10 Mb/s. So the only one problem here is to write to HDD with such a big portions of data which can give enough time to do writing. For instance, buffer can be 1 sec or 4.6 Mb. Storing of such portion of data will be faster then collecting of 1 s of data from hw device and "writing thread" takes no more than 500 ms to do its job. In real program this time can differ but in any way it must be enough for single-core systems as well.
Therefore you may use two ping-pong (one is filled with data from hw, one is stored to HDD) buffers with two threads and without circular buffer. Thus you can use producer/consumer model which is well described and has a lot of examples.

edited Nov 29 '13 at 12:55

answered Nov 29 '13 at 12:24

Michael

1,505
14
26

1

Michael, you have a error in your calcs, because we recive 9KB not 9Kb, this is 9*8*1024*500 = 36Mb/s, that's all, by the way, I will need a better throughput from HDD. – hiroru Dec 04 '13 at 14:16
@hiroru Yep, sorry, I've missed it for some reason, 8x difference... But anyway, 40-50 Mb/s must not be a problem for modern HDD/SSD. Thus the same idea should work only with a greater size of ram. – Michael Dec 04 '13 at 17:18
I am coding a 1 producer - 1 consumer model with N queues as you described. – hiroru Dec 04 '13 at 19:01

score 0 · Answer 3 · answered Nov 29 '13 at 11:35

0

You should look into asynchronous I/O. However Async-I/O APIs are operating system dependent and you didn't specify which OS you're using. Also your use of threads makes no sense. If you want to use threads (instead of async I/O) then you'd have one thread doing the packet reading and the other thread should to the writing. Putting it both into a single thread gives you no benefit.

answered Nov 29 '13 at 11:35

datenwolf

159,371
13
185
298

This must be cross-platform (Win/Linux) – hiroru Nov 29 '13 at 11:55

How to buffer and write to disk a low latency input with C

3 Answers3