5

I have a problem with readahead performance of a large binary file.

My strategy is

  • Read a fixed size chunk from the large binary file with fread(). (e.g. 10MB)

  • Sleep for n milliseconds with usleep() just for readahead performance measuring.

  • Repeat it until the entire file is read.

Then, I did an experiment of the above strategy for measuring readahead performance. The environment is

  • CentOS 7
  • 4GB binary file
  • Chunk sizes are 2 ~ 300MB per 2MB
  • Sleeping times I tested are 5, 10, 50, 100, 300 ms and no sleep

The results: enter image description here

As you can see in the result, when the chunk size is small and sleeping time is long enough for readahead, the reading time is short. The bigger chunk size is or the shorter sleeping time is , the closer to about 20 seconds reading time gets.

Then, I want to do regression analysis to this graph. What function should I use? What is the complexity of my strategy?

Jean-François Fabre
  • 137,073
  • 23
  • 153
  • 219
Yuf
  • 59
  • 1
  • can you include the image in the question? some of us can't visit those links. – Jean-François Fabre Dec 06 '18 at 08:38
  • Welcome to Stack Overflow. Please read [the help pages](http://stackoverflow.com/help), take [the SO tour](http://stackoverflow.com/tour), read about [how to ask good questions](http://stackoverflow.com/help/how-to-ask), as well as [this question checklist](https://codeblog.jonskeet.uk/2012/11/24/stack-overflow-question-checklist/). Lastly learn how to create a [mcve]. – Some programmer dude Dec 06 '18 at 08:39
  • Have you tried the same experiment with unbuffered IO or `read`? – paddy Dec 06 '18 at 08:43
  • @Jean-FrançoisFabre Maybe the image is already embedded. Anyway, sorry for inconvenience. – Yuf Dec 06 '18 at 08:58
  • @Someprogrammerdude I appreciate your advice. I'll check them. – Yuf Dec 06 '18 at 09:00
  • @paddy I tested setvbuf() for unbuffered I/O but not in this experiment. The result was same. – Yuf Dec 06 '18 at 09:01
  • now your question looks good. that doesn't mean I'm able to answer :) – Jean-François Fabre Dec 06 '18 at 09:09
  • Testing with `posix_fadvise()` might be interesting too. – Shawn Dec 06 '18 at 09:33
  • @Shawn Thank you for your proposition. I tested posix_fadvise() with POSIX_FADV_SEQUENTIAL flag but the result made no change. – Yuf Dec 06 '18 at 09:55

0 Answers0