loading XML files by path

Question

I have 814,652 XML files on a single folder, the names of the files are integers from 1 to 814,652.

In my application I load some xml file to a datatable but the loading is not instant, some time a delay up to 3 seconds.

Is it because the folder contains too many files?

If i could divide the files into multiple folders, will that speed things up?

simply i read a single time at a time based on the file name that i get from a selected row in a datagrid

note that i don't need to loop through the file list, all i need is to load a specified file name.

 int filename = ;//get the value from the focused row in the datagrid
 mydatatable.ReadXml(path + @"\xmlfiles\" + filename + ".xml");

update : dividing the files to sub folders and minimizing the number of files per folder made it even slower.

Are you reading the files by specific names or reading all files in directory and then reading each one ? — Shyju, Apr 11 '13 at 14:24
Depending on the size of the document, 3 seconds sounds reasonable to me. How big are these files? — DGibbs, Apr 11 '13 at 14:24
How are you loading the files, sequentially or in parallel? And how are you listing the files (if at all)? — Lloyd, Apr 11 '13 at 14:25
@Lloyd i don't list the files, i just load a single file by its directory dynamically , just added a sample — user1590636, Apr 11 '13 at 14:33
Instead of guessing, I suggest that you [profile this issue](http://stackoverflow.com/q/308816/87698). This should tell you exactly *where* the time is lost. — Heinzi, Apr 11 '13 at 14:34
@Shyju i am reading the required files by name, as in the sample — user1590636, Apr 11 '13 at 14:36

score 1 · Accepted Answer · edited May 23 '17 at 11:43

1

Assuming that you have NTFS file system (FAT32 would not hold 800k files in one dir), this post provides answer to your question, along with some metrics and suggestions. No, separating into several directories would only hurt performance.

In my opinion, it would be better to either combine all files into one, and map it into memory, or use more convenient storage, specifically designed for quick indexing and access, like database.

edited May 23 '17 at 11:43

Community

1
1

answered Apr 11 '13 at 19:01

Alexander

4,153
1
24
37

yes sure its NTFS, i thought that dividing files into sub Directories would speed things up but it got worse, i wonder why – user1590636 Apr 11 '13 at 19:57
1

Probably because NTFS loads single directory listing into cache, and then have to reload cache each time directory is changing. – Alexander Apr 11 '13 at 20:09

score 0 · Answer 2 · answered Apr 12 '13 at 14:18

0

i ended up using sql server dividing the files into tables and using EXI to compress XML.

answered Apr 12 '13 at 14:18

user1590636

1,174
6
26
56

loading XML files by path

2 Answers2