1

I'm looping big ~10k files using go-exiftool. I'm using one instance of the go-exiftool to get info for all required files. This code is called 10k times in the loop, where the file is always different.

fileInfos := et.ExtractMetadata(file)

After the ~7k loops the program hangs. I debugged go-exiftool and found that it hangs in https://github.com/barasher/go-exiftool/blob/master/exiftool.go#L121 on the line:

fmt.Fprintln(io.WriteCloser, "-execute")

if i understood correct io.WriteCloser has instance of exec.Command(binary, initArgs...).StdinPipe()

so, the questions are:

  1. Does exec.Command has a limit of execution?
  2. If 1) - not, what can be the reason else?
  3. Does it depends on the file sizes? I tried another folder and it worked with 35k files and then hanged. How to check that?

UPDATE: i tried to run the same file in 10k loops. Works fine. It looks like it runs out of memory, can it be? I see no problem in the system memory graph. Or stdin is overflowed. Have no idea how to check that.

Anthony
  • 3,218
  • 3
  • 43
  • 73
  • Have you tried to inspect the underlying exiftool process when this happens (e.g., with strace)? I suspect this is not a problem with Go or go-exiftool, but with the underlying process stopping reading commands from stdin. – chash Jul 08 '20 at 17:57
  • @hmm, thank you for the response. I would assume that it's stopping reading, but it has debug output right after ```fmt.Fprintln(io.WriteCloser, "-execute")``` and this shows that it's stopping on this line. I tried to run with dtruss (for macos). I hope it's a correct tool. I've got pretty big output and in a couple of seconds. Should it work like that? it looks too fast to be true. And what i have to look for there? – Anthony Jul 08 '20 at 18:38

0 Answers0