0

My test will randomly read 4K pages (random 4k page at time in a tight loop) in both 4GB and 32MB files using direct IO (O_DIRECT) and pread on NVME disk. The latency on 4GB file is about 41 microsecond per page and the latency is 79 microsecond for small 32MB file. Is there a rational explanation for such difference?

  b_fd = open("large.txt", O_RDONLY | O_DIRECT);
  s_fd = open("small.txt", O_RDONLY | O_DIRECT);

  std::srand(std::time(nullptr));
  
  void *buf;
  int ps=getpagesize();
  posix_memalign(&buf, ps, page_size);

  long long nano_seconds = 0;

  // number of random pages to read
  int iter = 256 * 100;

  for (int i = 0; i < iter; i++) {
    int page_index = std::rand() % big_file_pages;

    auto start = std::chrono::steady_clock::now();
    pread (b_fd, buf, page_size, page_index * page_size);
    auto end = std::chrono::steady_clock::now();
    nano_seconds += std::chrono::duration_cast<std::chrono::nanoseconds>(end - start).count();
  }

  std::cout<<"large file average random 1 page direct IO read in nanoseconds is:"<<nano_seconds/iter<<"\n";```
Kenneth
  • 561
  • 1
  • 5
  • 13
  • How reliable are results? How do you measure the latency? (Please provide reproducible example). What filesystem is used? Have you tried to create multiple 4Gb and 32Mb files in different order and see correlation between file size and latency? – dimich Nov 08 '22 at 22:23
  • 1
    The hypothesis may be that beginning of your 32Mb file is not aligned to internal page so reading 4kb requires two read cycles from different pages. – dimich Nov 08 '22 at 22:30
  • Added the code snippet above and entire code is in https://github.com/kennthhz/testingground/blob/master/src/test_direct_io.cc – Kenneth Nov 09 '22 at 00:00
  • File system is ext4 – Kenneth Nov 09 '22 at 00:08
  • 1
    My results are different for different sets of files. For one set read of large file is faster, for another one read of small file is faster. Probably it depends on internal layout of particular file in filesystem. – dimich Nov 09 '22 at 02:52

0 Answers0