-4

I am dealing with a rather large amount of files in a given directory. I need to get a list of all files but I find for loops to be too slow.

Is there any way I can list files in a directory without using a for loop?

peterh
  • 11,875
  • 18
  • 85
  • 108
Barry
  • 311
  • 3
  • 13
  • You have to loop, and `for` is how you do it in Go. It is unlikely to be specifically `for` and instead what you're using to get list the files and what you're doing inside the for loop. Could you show your slow code? – Schwern Jan 28 '21 at 23:41
  • What I'm wondering if there is a way to list all files in a given directory without looping at all, any other way? If we are speaking about hundreds of thousands to millions of files is there a faster way than looping? – Barry Jan 28 '21 at 23:44
  • 3
    It's not "looping" that is slow, your benchmark is wrong. – zerkms Jan 28 '21 at 23:45
  • 2
    @Barry [There are three main ways to get the list of files in a directory](https://stackoverflow.com/questions/14668850/list-directory-in-go) and none of them require loops. And millions of files in a single directory becomes a filesystem problem, not a Go problem. Please show us your code and more detail about your situation, otherwise we can't help. – Schwern Jan 28 '21 at 23:46
  • @Schwern, sadly not in liberty to share. But thank you for the pointer. I will check out the link and study ReadDir. I hope a function of a package like ReadDir won't be slower than a loop. Im looking specifically for the fastest method. I guess I will have to try out and benchmark. Thank you! – Barry Jan 28 '21 at 23:50
  • 2
    @Barry You could sanitize the code to show just reading files in a directory; I cannot fathom how you've been doing it without using standard libraries. Again, it's not Go's loop that's slow, it's how you're using it. Go's standard libraries will loop. You can look at the underlying code yourself: [readdirnames](https://golang.org/src/os/dir_unix.go?h=readdirnames#L29) and [readdir](https://golang.org/src/os/file_unix.go?h=readdir#L336). And, again, millions of files in a single directory will strain most filesystems. Consider a different approach. – Schwern Jan 29 '21 at 00:43
  • @Barry For example, show us how you'd print the names of all the files in a directory using your current approach. That would help with understanding. – Schwern Jan 29 '21 at 00:57
  • Here some fast code : `fmt.Println("foo.txt bar.go"). It is not 100% correct, but _very_ fast, even on slow disks and large directories. Please make sure you understand the problem space well enough until trying to find a solution. – Volker Jan 29 '21 at 06:21

1 Answers1

0
files, err := filepath.Glob(path + "*.xyz")

works nicely for me.

Barry
  • 311
  • 3
  • 13