0

I run stat64() for 1000 files in a folder and took less than 1s but when there were 5000 in the same directory, the time increased to 15s.

Why stat64 () is getting slow non -linearly ? I was expecting the time to be 5s

EDIT I am reading data from a USB with FAT file system.

Werner Henze
  • 16,404
  • 12
  • 44
  • 69
Naveen
  • 7,944
  • 12
  • 78
  • 165
  • 3
    Because you are using a toy file system which is scanning the list of entries for every name lookup. Try XFS (best) or ext4 or enabling "dir_index" on ext3. – Nemo Nov 14 '14 at 05:48
  • what's your file system type? – tristan Nov 14 '14 at 05:58
  • @Insane Coder: Are you stat64-ing 1000 files in both cases or are you stat64-ing every file that is present? – Werner Henze Nov 14 '14 at 08:17
  • @WernerHenze : Every file that is present. In first case for 1000 files and in second case for 5000 files. – Naveen Nov 14 '14 at 08:35

1 Answers1

3

When you are calling stat64 for a file in a directory which contains N entries, then the complexity is O(N) because with FAT file system the system has to walk through all directory entries and compare each to the one you are looking for.

When you are calling stat64 M times in a folder which contains N entries, then the complexity is O(M*N) and in the case there M=N you end up with O(N*N).

Looking at your example: when you have and stat64 factor 5 files, then you need factor 25 time. If your 1s time in fact was 0.6 seconds, then the results are as expected.

Werner Henze
  • 16,404
  • 12
  • 44
  • 69