2

I hate to be the person to post another FileNotFoundError question, but most of them that I see are about not giving the full path to the file, that is not my problem here.

I have a number of log files in folders in ../../Data/. I create a glob of those files using

DataFiles = glob('../../Data/2021*/*.log')

I want to open each of the files in that glob, so I use

for i, file in enumerate(DataFiles):
    with open(file, "r") as f:
...

etc. 99% of these open correctly and the rest of the code runs. For some reason, a few will not. I get an error like

FileNotFoundError: [Errno 2] No such file or directory: '../../Data\\20210629_081706\\20210629_081706_data.log'

The file definitely exists, that's why it was found by glob. The full path is used. And,

from pathlib import Path
Path('../../Data\\20210629_081706\\20210629_081706_data.log')

returns

WindowsPath('../../Data/20210629_081706/20210629_081706_data.log')

So does anyone know what might be happening here?

Jerup
  • 21
  • 1
  • Try to hard code the path to the problematic file and run your program on it only, then see if the problem persists. – Ilian Zapryanov Oct 28 '21 at 20:38
  • Yes it persists. Running ```file = ...``` and ```open(file, "r")``` gives the same error. ```open``` is having a problem with opening that path. – Jerup Oct 28 '21 at 20:40
  • You should work with absolute paths where possible. At least before try to open the file, the path should be absolute to ensure that the right file is meant. – Michael Butscher Oct 28 '21 at 20:42
  • Try to hard code the path but in `raw` string ex.: `s = r"../../Data\\20210629_081706\\20210629_081706_data.log"` and try again. – Ilian Zapryanov Oct 28 '21 at 20:42
  • By the way: You can create a "Path" (or subclasses of it) object although the referenced element doesn't exist in the file system. – Michael Butscher Oct 28 '21 at 20:45
  • @MichaelButscher, sure this is a relative path and I would prefer to continue using that since this code is shared with others. Why would this matter when it works for the other 1000+ log files? The file does exist in the file system, I can touch it using pathlib's ```Path``` or in the command prompt. – Jerup Oct 28 '21 at 20:48
  • @IlianZapryanov Done, same error. – Jerup Oct 28 '21 at 20:48
  • Ok, then time to use a debugger, if you have `vscode` try to put a break point on the `glob` line then step with F11 inside the glob and check what are the parameters in the side window. – Ilian Zapryanov Oct 28 '21 at 20:51
  • @IlianZapryanov before doing so, I don't think it is a ```glob``` issue since hardcoding the path gives the same error... right? – Jerup Oct 28 '21 at 20:55
  • @Jerup can you post how you hard coded the path just to be on the same track here. – Ilian Zapryanov Oct 28 '21 at 20:57
  • @Jerup It can help to rule out some kinds of errors (e. g. a hidden change of current working directory). – Michael Butscher Oct 28 '21 at 20:59
  • @IlianZapryanov ```file = r"../../Data\\20210629_081706\\20210629_081706_data.log" open(file, "r") ``` I just noticed that if I use the absolute path it actually works (```open``` opens the file), so it probably is a ```glob``` issue. – Jerup Oct 28 '21 at 21:02
  • @MichaelButscher good point and I think you are right. What's weird is if I just remove this log file everything continues right on without a problem, so it's clearly this path that is the issue. – Jerup Oct 28 '21 at 21:03
  • @Jerup then let's go back to debugging it and try to investigate what is going on. Might some missformated variable later on. Step in `glob` and see. – Ilian Zapryanov Oct 28 '21 at 21:04
  • So for instance if I run ```file = r"../../Data\\20210629_091706\\20210629_101706_data.log" open(file, "r")``` (Note that this is a different file), that opens fine. Then if I do ```file = r"../../Data\\20210629_081706\\20210629_081706_data.log" open(file, "r") ``` I get the error. This makes me think that I have not changed directories, since those two actually are side by side. – Jerup Oct 28 '21 at 21:06
  • What happens if you convert all of the files returned from glob to a path before opening them? Example: ```for i, file in enumerate(DataFiles): file_path = Path(file) if file_path.exists(): with open(file_path, "r") as f:``` – Jeff Gruenbaum Oct 28 '21 at 21:23
  • @IlianZapryanov thank you for the help but maybe this will never be solved. I'm now down the rabbit hole of trying to setup vscode to run on my Jupyter Notebook and I don't think this has an end in sight. I'm going to do something hacky like Jeff suggests. – Jerup Oct 28 '21 at 21:38

1 Answers1

0

A bit late, but I had the same error when using glob in a network folder with way to many levels.

There was a particular folder where some of the files caused that error, and those files couldn't even be opened by the explorer itself:

enter image description here

In my case this was caused by the path being over 260 characters in length.

You can try something like suggested here to allow handling files with larger paths, or just make sure the path is short enough for the explorer to handle it.

berna1111
  • 1,811
  • 1
  • 18
  • 23