I cannot list large directory

Question

I've a large directory (very large), in where ls is not working, I've also tried iterate it with for, and NO luck.

This directory is in a gluster volume... Any idea?

More information:

Gluster vol is mounted in fstab with simple options.
I've tried for f in * ; do something with $f ; done
Now I'm trying rsync -a --delete empty-dir/ my-dir/

regards

yes sr.! this was the first I've tried... Now I'm running a for and it's taking 12 hs and running :( — Federico Aguirre, Dec 16 '18 at 20:23
How *exactly* have you run `ls`? Are you sure it is not some `alias` ? Have you tried *exactly* `/bin/ls > /tmp/ls.out` ? Or `/bin/ls -f > /tmp/unsorted-ls.out` ? — Basile Starynkevitch, Dec 16 '18 at 20:35
I guess that your `for` loop (which is not entirely mentioned in your question) won't answer. You want `/bin/ls -f` which is probably the fastest. — Basile Starynkevitch, Dec 16 '18 at 20:42
I'm mounting my fs simply in the fstab defaults,_netdev,backupvolfile-server=my2nd. I'm now trying /bin/ls -f as you suggested... — Federico Aguirre, Dec 16 '18 at 20:45
Please [edit](https://stackoverflow.com/posts/53806058/edit) your question to improve it. Don't give these important details only in comments. In general, avoid commenting your own question! — Basile Starynkevitch, Dec 16 '18 at 20:48
`for f in *` is bad for big list of arguments. You can try `find . -type f -print0 | while IFS= read -d '' -r f` — KamilCuk, Dec 16 '18 at 20:57

Basile Starynkevitch · Answer 1 · 2018-12-16T21:01:53.567

You almost certainly could use system calls (listed in syscalls(2)) and related functions, e.g. opendir(3), readdir(3) (they internally use getdents(2) that you don't want to use directly), closedir, stat(2), or nftw(3) on that; of course you'll need to write a specific program (perhaps in C, or at least in Python).

Next time, even for huge (petabyte sized) file systems, take care to avoid more than a few thousand entries per directory (so organize your files like 00/00/0001.jpeg .... 01/23/4567.jpeg and so on, not 00000001.jpeg ... 01234567.jpeg and so on in a single directory). If you keep small sized directories (even in petabyte sized filesystems) you'll make your shell happier and more usable (and perhaps even your kernel).

However, recent file systems like EXT4 or BTRFS are capable to have many millions of entries in a directory. But shells (and globbing, see glob(7)...) are unhappy with that. My guess would be that /bin/ls without arguments (or at least /bin/ls -f to get an unsorted listing) should work (see ls(1)).

Some file systems, even when capable of many millions of entries per directory, may need a linear time to access them (see path_resolution(7), ext4(5) etc...), or else you might need to supply specific arguments to mount(8) (e.g. dir_index for ext4).

Remember that globbing is done by your shell before calling execve(2), which does have limitations (generally, a few hundred thousands bytes at most for program arguments, so no more than a few thousand of expanded arguments to a program, otherwise execve fails with E2BIG).

`getdents` man page has a nice sample program that I used once in a similar situation. — James Brown, Dec 16 '18 at 20:27

I cannot list large directory

1 Answers1