Using Keras ImageDataGenerator in a regression model

Question

I want to use the flow_from_directory method of the ImageDataGenerator to generate training data for a regression model, where the target value can be any float value between 1 and -1. flow_from_directory has a "class_mode" parameter with the description

class_mode: one of "categorical", "binary", "sparse" or None. Default: "categorical". Determines the type of label arrays that are returned: "categorical" will be 2D one-hot encoded labels, "binary" will be 1D binary labels, "sparse" will be 1D integer labels.

Which of these values should I take? None of them seems to really fit...

Initially, there is a folder with pics and a list mapping the pics to their target values. I am open for suggestions :-) Unfortunately, the keras documentation doesn't expand much on regression models. — Oblomov, Jan 19 '17 at 19:07
Actually - you need a really neat in order to get the behaviour you want. — Marcin Możejko, Jan 19 '17 at 19:32
Sorry, I didn't completely get your answer - what do I need? — Oblomov, Jan 19 '17 at 19:45

score 15 · Answer 1 · edited Sep 06 '22 at 09:19

With Keras 2.2.4 you can use flow_from_dataframe which solves what you want to do, allowing you to flow images from a directory for regression problems. You should store all your images in a folder and load a dataframe containing in one column the image IDs and in the other column the regression score (labels) and set class_mode='other' in flow_from_dataframe.

Here you can find an example where the images are in image_dir, the dataframe with the image IDs and the regression scores is loaded with pandas from the "train file"

train_label_df = pd.read_csv(train_file, delimiter=' ', header=None, names=['id', 'score'])

train_datagen = ImageDataGenerator(rescale = 1./255, horizontal_flip = True,
                                   fill_mode = "nearest", zoom_range = 0.2,
                                   width_shift_range = 0.2, height_shift_range=0.2,
                                   rotation_range=30) 

train_generator = train_datagen.flow_from_dataframe(dataframe=train_label_df, directory=image_dir, 
                                              x_col="id", y_col="score", has_ext=True, 
                                              class_mode="other", target_size=(img_width, img_height), 
                                              batch_size=bs)

score 7 · Answer 2 · answered Aug 31 '17 at 19:45

I think that organizing your data differently, using a DataFrame (without necessarily moving your images to new locations) will allow you to run a regression model. In short, create columns in your DataFrame containing the file path of each image and the target value. This allows your generator to keep regression values and images properly synced even when you shuffle your data at each epoch.

Here is an example showing how to link images with binomial targets, multinomial targets and regression targets just to show that "a target is a target is a target" and only the model might change:

df['path'] = df.object_id.apply(file_path_from_db_id)
df

       object_id   bi  multi                                    path     target
index                                                               
0         461756  dog  white    /path/to/imgs/756/61/blah_461756.png   0.166831
1        1161756  cat  black   /path/to/imgs/756/61/blah_1161756.png   0.058793
2        3303651  dog  white   /path/to/imgs/651/03/blah_3303651.png   0.582970
3        3367756  dog   grey   /path/to/imgs/756/67/blah_3367756.png  -0.421429
4        3767756  dog   grey   /path/to/imgs/756/67/blah_3767756.png  -0.706608
5        5467756  cat  black   /path/to/imgs/756/67/blah_5467756.png  -0.415115
6        5561756  dog  white   /path/to/imgs/756/61/blah_5561756.png  -0.631041
7       31255756  cat   grey  /path/to/imgs/756/55/blah_31255756.png  -0.148226
8       35903651  cat  black  /path/to/imgs/651/03/blah_35903651.png  -0.785671
9       44603651  dog  black  /path/to/imgs/651/03/blah_44603651.png  -0.538359
10      49557622  cat  black  /path/to/imgs/622/57/blah_49557622.png  -0.295279
11      58164756  dog   grey  /path/to/imgs/756/64/blah_58164756.png   0.407096
12      95403651  cat  white  /path/to/imgs/651/03/blah_95403651.png   0.790274
13      95555756  dog   grey  /path/to/imgs/756/55/blah_95555756.png   0.060669

I describe how to do this in great detail with examples here:

https://techblog.appnexus.com/a-keras-multithreaded-dataframe-generator-for-millions-of-image-files-84d3027f6f43

Marcin Możejko · Accepted Answer · 2017-01-21T02:07:32.753

At this moment (newest version of Keras from January 21st 2017) the flow_from_directory could only work in a following manner:

You need to have a directories structured in a following manner:

directory with images\
    1st label\
        1st picture from 1st label
        2nd picture from 1st label
        3rd picture from 1st label
        ...
    2nd label\
        1st picture from 2nd label
        2nd picture from 2nd label
        3rd picture from 2nd label
        ...
    ...

flow_from_directory returns batches of a fixed size in a format of (picture, label).

So as you can see it could only be used for a classification case and all options provided in a documentation specify only a way in which the class is provided to your classifier. But, there is a neat hack which could make a flow_from_directory useful for a regression task:

You need to structure your directory in a following manner:

directory with images\
    1st value (e.g. -0.95423)\
        1st picture from 1st value
        2nd picture from 1st value
        3rd picture from 1st value
        ...
    2nd value (e.g. - 0.9143242)\
        1st picture from 2nd value
        2nd picture from 2nd value
        3rd picture from 2nd value
        ...
   ...

You also need to have a list list_of_values = [1st value, 2nd value, ...]. Then your generator is defined in a following manner:

def regression_flow_from_directory(flow_from_directory_gen, list_of_values):
    for x, y in flow_from_directory_gen:
        yield x, list_of_values[y]

And it's crucial for a flow_from_directory_gen to have a class_mode='sparse' to make this work. Of course this is a little bit cumbersome but it works (I used this solution :) )

Thanks. Obviously, flow_from_directory is not designed to be applied to regression problems, since the target values are inferred from the folder structure. I will think about using your hack - the folder structure looks complicated at first, but it can easily be created automatically. — Oblomov, Jan 21 '17 at 04:31

score 2 · Answer 4 · answered Oct 30 '18 at 04:38

There's just one glitch in the accepted answer that I would like to point out. The above code fails with an error message like:

TypeError: only integer scalar arrays can be converted to a scalar index

This is because y is an array. The fix is simple:

def regression_flow_from_directory(flow_from_directory_gen,
            list_of_values):
    for x, y in flow_from_directory_gen:
        values = [list_of_values[y[i]] for i in range(len(y))]
        yield x, values

The method to generate the list_of_values can be found in https://stackoverflow.com/a/47944082/4082092

Using Keras ImageDataGenerator in a regression model

4 Answers4

Linked