TypeError: 'NoneType' object is not iterable - but still seems to iterate through the list?

Question

I'm pretty new to coding so thanks for any help, and sorry if this is obvious.

full_image_list = glob.glob(WORKING_DIR+'/data/JPEGImages/*.jpg')
fnames, classes = [], []

for jpg_fname in full_image_list:
    fs, data = return_yolo_data(jpg_fname, ann_dir=WORKING_DIR+'/data/Annotations/')
    for f, d in zip(fs, data):
        fnames.append(f)

The code above gives the following error:

> ---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/.virtualenvs/dl4cv/lib/python3.8/site-packages/numpy/lib/npyio.py in genfromtxt(fname, dtype, comments, delimiter, skip_header, skip_footer, converters, missing_values, filling_values, usecols, names, excludelist, deletechars, replace_space, autostrip, case_sensitive, defaultfmt, unpack, usemask, loose, invalid_raise, max_rows, encoding)
   1753             fid_ctx = contextlib_nullcontext(fid)
-> 1754         fhd = iter(fid)
   1755     except TypeError:

TypeError: 'NoneType' object is not iterable

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
<ipython-input-187-80766db0462f> in <module>
      2 
      3 for jpg_fname in full_image_list:
----> 4     fs, data = return_yolo_data(jpg_fname, ann_dir=WORKING_DIR+'/data/Annotations/')
      5     for f, d in zip(fs, data):
      6         fnames.append(f)

<ipython-input-159-ecbad441c355> in return_yolo_data(jpg_fname, ann_dir)
     19     base_fname = jpg_fname.split('/')[-1] # no path
     20     txt_fname = get_yolo_file(base_fname, dir_=ann_dir)
---> 21     data = np.genfromtxt(txt_fname, names='class,x,y,w,h')
     22     # don't fail if there is only one labeled object in an image
     23     data = np.atleast_1d(data)

~/.virtualenvs/dl4cv/lib/python3.8/site-packages/numpy/lib/npyio.py in genfromtxt(fname, dtype, comments, delimiter, skip_header, skip_footer, converters, missing_values, filling_values, usecols, names, excludelist, deletechars, replace_space, autostrip, case_sensitive, defaultfmt, unpack, usemask, loose, invalid_raise, max_rows, encoding)
   1754         fhd = iter(fid)
   1755     except TypeError:
-> 1756         raise TypeError(
   1757             "fname must be a string, filehandle, list of strings, "
   1758             "or generator. Got %s instead." % type(fname))

TypeError: fname must be a string, filehandle, list of strings, or generator. Got <class 'NoneType'> instead.

If I use this code:

full_image_list = glob.glob(WORKING_DIR+'/data/JPEGImages/*.jpg')
fnames, classes = [], []

for jpg_fname in full_image_list:
    fs, data = return_yolo_data(jpg_fname, ann_dir=WORKING_DIR+'/data/Annotations/')
    for f, d in zip(fs, data):
        print(f)

Then it iterates through to print out the name of each file, each on a separate line with no commas/syntax, before showing the same error.

Here is the code for the get_yolo_file function and others as requested.

def get_yolo_file(fname, dir_=WORKING_DIR+'/data/Annotations/'):
    '''
    takes image filename and returns annotation filename
    e.g. data/JPEGImages/IMG_XXX.jpg
         data/Annotations/IMG_XXX.txt
    '''
    txt_file = fname.replace('.jpg', '.txt')
    full_path = '/'.join([dir_, txt_file])
    if os.path.isfile(full_path):
        return full_path
    else:
        print('ERROR')
        return None

def return_yolo_data(jpg_fname, ann_dir=WORKING_DIR+'/data/Annotations/'):
    '''
    Reads annotation file, returns name and data
    '''
    base_fname = jpg_fname.split('/')[-1] # no path
    txt_fname = get_yolo_file(base_fname, dir_=ann_dir)
    data = np.genfromtxt(txt_fname, names='class,x,y,w,h')
    # don't fail if there is only one labeled object in an image
    data = np.atleast_1d(data)
    # keep file name associated with each label
    fname_arr = [base_fname]*data.shape[0]
    return fname_arr, data

def str_class(val):
    '''
    Return string label for index. 
    YOLO is zero indexed.
    '''
    if val == 0:
        return 'one'
    elif val == 1:
        return 'two'
    elif val == 2:
        return 'three'
    elif val == 3:
        return 'four'
    elif val == 4:
        return 'five'
    elif val == 5:
        return 'six'

def create_image_dict(image_list, ann_dir=WORKING_DIR+'/data/Annotations/'):
    '''
    For input list of images, return a dictionary with label information.
    dictionary keys: image file, class (string), x, y, h, w 
    '''
    # empty lists for dictionary
    fnames, classes = [], []
    x,y,h,w = [],[],[],[]
    for jpg_fname in image_list:
        fs, data = return_yolo_data(jpg_fname, ann_dir=ann_dir)
        for f, d in zip(fs, data):
            fnames.append(f)
            classes.append(str_class(d['class']))
            x.append(d['x'])
            y.append(d['y'])
            h.append(d['h'])
            w.append(d['w'])
    out_dict = {'filename':fnames, 'class':classes, 'x':x, 'y':y, 'h':h, 'w':w}
    return out_dict

Answer 1

Got <class 'NoneType'> instead.

This error message clearly states the problem... os.path.isfile(full_path) returns False so get_yolo_file returns None.

So txt_fname == None. And as the error says, the file path for np.genfromtxt can't be None.

So instead of 'printing' an error (print('ERROR')) you should do something like this:

class MyException(Exception):
    pass

def get_yolo_file(fname, dir_=WORKING_DIR+'/data/Annotations/'):
    '''
    takes image filename and returns annotation filename
    e.g. data/JPEGImages/IMG_XXX.jpg
         data/Annotations/IMG_XXX.txt
    '''
    txt_file = fname.replace('.jpg', '.txt')
    full_path = '/'.join([dir_, txt_file])
    if os.path.isfile(full_path):
        return full_path
    else:
        raise MyException("File path is wrong!")

This will raise an error saying File path is wrong! and will terminate the execution of the program.

---

As to why os.path.isfile(full_path) returns False, you will have to check if such a file actually exists at that location and if the path that you are creating is the actual path to the file.

---

For more info on custom exceptions: See custom exceptions

Answer 2

def get_yolo_file(fname, dir_=WORKING_DIR+'/data/Annotations/'):
'''
takes image filename and returns annotation filename
e.g. data/JPEGImages/IMG_XXX.jpg
     data/Annotations/IMG_XXX.txt
'''
txt_file = fname.replace('.jpg', '.txt')
full_path = '/'.join([dir_, txt_file])
if os.path.isfile(full_path):
    return full_path
else:
    print('ERROR')
    return None

I replaced the '/' from the full_path with ''. As the error was being thrown everytime even on files I knew all existed. This got rid of that problem.

Then noticed that I was missing a .txt file (thank you to @Hampus Larrson for putting me on the right track to solve this part)