Memory leak using flow_from_directory

Question

I'm trying to implement the technique described on the documentation page

https://keras.io/preprocessing/image/

under the heading "Example of transforming images and masks together".

After the following,

image_generator = image_datagen.flow_from_directory(
    'data/images',
    class_mode=None,
    seed=seed)

mask_generator = mask_datagen.flow_from_directory(
        'data/masks',
        class_mode=None,
        seed=seed)

the problem arises with the command:

# combine generators into one which yields image and masks
train_generator = zip(image_generator, mask_generator)

This results in memory usage rising to the maximum possible, and then swapping also rises to the max, at which point my system freezes and requires rebooting.

Does anyone have a clue as to what's going on here?

score 0 · Accepted Answer · answered Aug 07 '17 at 14:56

SOLUTION: The problem is that I was using Python 2, and in Python 2 the zip command on these iterators will keep iterating forever. Any amount of memory would soon be exhausted. With Python 3 this wouldn't be a problem.

The solution, if you're using Python 2, is to use itertools.izip instead of zip.

Memory leak using flow_from_directory

1 Answers1