Performant check whether a directory contains at least n files

Asked Oct 30 '14 at 14:58

Active Oct 30 '14 at 14:58

Viewed 17 times

I a script that processes files from a folder calles source. As I process these files in batches I need to verify that at least n files are in source. The problem is that source possibly contains a few hundred thousand or millions of files, thus len(os.listdir(src)) is too slow for my needs.

I've considered having a function that os.walks in source until the number if minimum required files reached, then returns True, else False. I just tested walk really quick

for rwalk,dwalk,fwalk in os.walk(C:\\tmp\\folder\\):
    for f in fwalk: 
        print(f)

which blurts out in tons of information even if C:\\tmp\\folder\\ only contains a handful of files.

How can I have a function that checks if at least n files are in source without listing every contained file first?

asked Oct 30 '14 at 14:58

Daedalus Mythos

1

Python does not (yet) have a streaming version of `os.listdir()`. That method returns a list of all entries, always. – Martijn Pieters Oct 30 '14 at 14:59
That is what I understood from reading the `os.listdir()` documentation, but does `os.walk()` do the same? Is there no other achieve a one-after-the-other file counter? – Daedalus Mythos Oct 30 '14 at 15:01
1

All other directory listing / walking code uses `os.listdir()` under the hood. – Martijn Pieters Oct 30 '14 at 15:02

Performant check whether a directory contains at least n files

0 Answers0