1

On extfs, if there are only file-creations and no -deletions in a directory, I expect that find . -type f would list the files either in their chronological order of creation (or mtime), or if not, at least in their reverse chronological order... depending on how a directory's contents are traversed.

But that isn't the behavior I'm seeing.

The following code, eg, creates a fresh set of directories and files:

#!/bin/bash -u

for i in a/ a/{1,2,3,4,5} b/ b/{1,2,3,4,5}; do
    if echo "$i" | egrep -q "/$"; then
        echo "Creating dir $i"
        mkdir -p "$i"
    else
        echo "Creating file $i"
        touch "$i"
    fi
    sleep 0.500
done

Output of the above snippet:

Creating dir a/
Creating file a/1
Creating file a/2
Creating file a/3
Creating file a/4
Creating file a/5
Creating dir b/
Creating file b/1
Creating file b/2
Creating file b/3
Creating file b/4
Creating file b/5

However, find lists files in somewhat random order. For example, a/2 doesn't follows a/1, and b/2 doesn't follow b/1:

$ find . -type f
./a/1
./a/3
./a/4
./a/2   <----
./a/5
./b/1
./b/3
./b/4
./b/2   <----
./b/5

Any idea why this should happen?

My main problem is: I have a very large volume storing 100s of 1000s of files. I need to traverse these files and directories in the order of their creation/modification (mtime) and pipe each file to another process for further processing. But I don't necessarily want to first create a temporary list of this large set of files and then sort it based on mtime before piping it to my process.

Harry
  • 3,684
  • 6
  • 39
  • 48

1 Answers1

2

find lists objects in the order that they are reported by the underlying filesystem implementation. You can tell ls to show you this "raw" order by passing it the -f option.

The order could be anything at all -- alphabetical, by mtime, by atime, by length of name, by permissions, or something completely different. The ordering can even vary from one listing to the next.

It's common for filesystems to report in an order that reflects the filesystem's strategy for allocating directory slots to files. If this is some sort of hash-based strategy based on filename then the order can appear nonsensical. This is what happens with widely-used Linux and BSD filesystem implementations. Since you mention extfs this is probably what causes the ordering you're seeing.

So, if you need the output from find to be ordered in a particular way then you'll have to create that order yourself. Maybe based on something like:

  find . -type f -exec ls -ltr --time-style=+%s {} \; | sort -n -k6
ottomeister
  • 5,415
  • 2
  • 23
  • 27
  • I was trying to avoid `sort` which really is a 'sponge filter' instead of being a 'filter filter' - but anyways. Btw, `xargs` may be better than a per-item `-exec`. – Harry Jun 03 '18 at 02:06
  • 1
    If you need a global ordering (`mtime` across all files) then the "sponge" behaviour (in `sort` or otherwise) is unavoidable -- the last file you examine might be the oldest, therefore the first to be emitted. Re `xargs`, sure, provided that you take steps to deal with whitespace and other metacharacters in pathnames, such as `find ... -print0 | xargs -0 ...`. Also, if you really are on Linux then `stat -c "%Y %n"` has less overhead than `ls -l`. – ottomeister Jun 03 '18 at 02:25