preprocessing images generated using keras function ImageDataGenerator() to train resnet50 model

Question

I am trying to train resnet50 model for image classification problem.I have loaded the 'imagenet' pretrained weights before training the model on the image dataset I have. I am using keras function flow_from_directory() to load images from directory.

train_datagen = ImageDataGenerator()
train_generator = train_datagen.flow_from_directory(
        './train_qcut_2_classes',
        batch_size=batch_size,
        shuffle=True,
        target_size=input_size[1:],
        class_mode='categorical')  
test_datagen = ImageDataGenerator()
validation_generator = test_datagen.flow_from_directory(
        './validate_qcut_2_classes',
        batch_size=batch_size,
        target_size=input_size[1:],
        shuffle=True,
        class_mode='categorical')

And I pass the generators as parameters in the fit_generator function.

hist2=model.fit_generator(train_generator,
                        samples_per_epoch=102204,
                        validation_data=validation_generator,
                        nb_val_samples=25547,
                        nb_epoch=80, callbacks=callbacks,
                        verbose=1)

Question:

With this setup how do I use preprocess_input() function to preprocess the input images before passing them to the model?

from keras.applications.resnet50 import preprocess_input

I tried using preprocessing_function parameter as below

train_datagen=ImageDataGenerator(preprocessing_function=preprocess_input)
train_generator = train_datagen.flow_from_directory(
        './train_qcut_2_classes',
        batch_size=batch_size,
        shuffle=True,
        target_size=input_size[1:],
        class_mode='categorical')  
test_datagen = ImageDataGenerator(preprocessing_function=preprocess_input)
validation_generator = test_datagen.flow_from_directory(
        './validate_qcut_2_classes',
        batch_size=batch_size,
        target_size=input_size[1:],
        shuffle=True,
        class_mode='categorical')

When i tried to extract the preprocessed output, I got this below result.

train_generator.next()[0][0]

array([[[  91.06099701,   80.06099701,   96.06099701, ...,   86.06099701,
       52.06099701,   12.06099701],
    [ 101.06099701,  104.06099701,  118.06099701, ...,  101.06099701,
       63.06099701,   19.06099701],
    [ 117.06099701,  103.06099701,   88.06099701, ...,   88.06099701,
       74.06099701,   18.06099701],
    ..., 
    [-103.93900299, -103.93900299, -103.93900299, ...,  -24.93900299,
      -38.93900299,  -24.93900299],
    [-103.93900299, -103.93900299, -103.93900299, ...,  -52.93900299,
      -27.93900299,  -39.93900299],
    [-103.93900299, -103.93900299, -103.93900299, ...,  -45.93900299,
      -29.93900299,  -28.93900299]],

   [[  81.22100067,   70.22100067,   86.22100067, ...,   69.22100067,
       37.22100067,   -0.77899933],
    [  91.22100067,   94.22100067,  108.22100067, ...,   86.22100067,
       50.22100067,    6.22100067],
    [ 107.22100067,   93.22100067,   78.22100067, ...,   73.22100067,
       62.22100067,    6.22100067],
    ..., 
    [-116.77899933, -116.77899933, -116.77899933, ...,  -36.77899933,
      -50.77899933,  -36.77899933],
    [-116.77899933, -116.77899933, -116.77899933, ...,  -64.77899933,
      -39.77899933,  -51.77899933],
    [-116.77899933, -116.77899933, -116.77899933, ...,  -57.77899933,
      -41.77899933,  -40.77899933]],

   [[  78.31999969,   67.31999969,   83.31999969, ...,   61.31999969,
       29.31999969,   -7.68000031],
    [  88.31999969,   91.31999969,  105.31999969, ...,   79.31999969,
       43.31999969,   -0.68000031],
    [ 104.31999969,   90.31999969,   75.31999969, ...,   66.31999969,
       53.31999969,   -2.68000031],
    ..., 
    [-123.68000031, -123.68000031, -123.68000031, ...,  -39.68000031,
      -53.68000031,  -39.68000031],
    [-123.68000031, -123.68000031, -123.68000031, ...,  -67.68000031,
      -42.68000031,  -54.68000031],
    [-123.68000031, -123.68000031, -123.68000031, ...,  -60.68000031,
      -44.68000031,  -43.68000031]]], dtype=float32)

To confirm this, I directly used the preprocessing function on a particular image,

import cv2
img = cv2.imread('./images.jpg')
img = img_to_array(img)
x = np.expand_dims(img, axis=0)
x = x.astype(np.float64)
x = preprocess_input(x)

which gives the below output,

array([[[[ 118.061,  125.061,  134.061, ...,   97.061,   99.061,  102.061],
     [ 118.061,  125.061,  133.061, ...,   98.061,  100.061,  102.061],
     [ 113.061,  119.061,  126.061, ...,  100.061,  101.061,  102.061],
     ..., 
     [  65.061,   64.061,   64.061, ...,   60.061,   61.061,   57.061],
     [  64.061,   64.061,   63.061, ...,   66.061,   67.061,   59.061],
     [  56.061,   59.061,   62.061, ...,   61.061,   60.061,   59.061]],

    [[ 113.221,  120.221,  129.221, ...,  112.221,  114.221,  113.221],
     [ 116.221,  123.221,  131.221, ...,  113.221,  115.221,  113.221],
     [ 118.221,  124.221,  131.221, ...,  115.221,  116.221,  113.221],
     ..., 
     [  56.221,   55.221,   55.221, ...,   51.221,   52.221,   51.221],
     [  55.221,   55.221,   54.221, ...,   57.221,   58.221,   53.221],
     [  47.221,   50.221,   53.221, ...,   52.221,   51.221,   50.221]],

    [[ 109.32 ,  116.32 ,  125.32 , ...,  106.32 ,  108.32 ,  108.32 ],
     [ 111.32 ,  118.32 ,  126.32 , ...,  107.32 ,  109.32 ,  108.32 ],
     [ 111.32 ,  117.32 ,  124.32 , ...,  109.32 ,  110.32 ,  108.32 ],
     ..., 
     [  34.32 ,   33.32 ,   33.32 , ...,   30.32 ,   31.32 ,   26.32 ],
     [  33.32 ,   33.32 ,   32.32 , ...,   36.32 ,   37.32 ,   28.32 ],
     [  25.32 ,   28.32 ,   31.32 , ...,   30.32 ,   29.32 ,   28.32 ]]]])

Any ideas on why this happens?

The output is consistent with the preprocessing function. If you **didn't have** a preprocessing, then your values would be between 0 and 255. — Daniel Möller, May 02 '18 at 14:29
I think you got "unlucky" with the image you chose. I don't see anything greater than 135 either :) — Daniel Möller, May 02 '18 at 14:33
You may print `x.max()` and `x.min()` to see the results better. The negative values may be hidden in the `...`, and the only thing that will surely indicate that no preprocessing was done is the presence of values greater than 152. — Daniel Möller, May 02 '18 at 15:01
So i am getting the maximum value as 151.061 and minimum value as -123.68 — AKSHAYAA VAIDYANATHAN, May 02 '18 at 15:05
Then your preprocessing is working totally ok. The function is the same as the one you used [in your other question](https://stackoverflow.com/questions/50136032/modify-layers-in-resnet-model/50136494#50136494). `255-103 = 152` (Or, exactly: `255 - 103.939 = 151.061)` — Daniel Möller, May 02 '18 at 16:02

score 7 · Accepted Answer · answered May 02 '18 at 13:55

7

As an argument when creating ImageDataGenerator:

train_datagen = ImageDataGenerator(preprocessing_function=preprocess_input)

answered May 02 '18 at 13:55

Daniel Möller

84,878
18
192
214

1

Have you created the `flow_from_directory` generator again from the new `ImageDataGenerator`? Can you share how you confirmed that this does not produce preprocessed output? – Daniel Möller May 02 '18 at 14:19
I have edited the question which shows you the results i got – AKSHAYAA VAIDYANATHAN May 02 '18 at 14:26
Then, Why do I not get negative values in the same image when I use the preprocessing_function directly on the image? – AKSHAYAA VAIDYANATHAN May 02 '18 at 14:33
1

Try more images, I think you got unlucky. Notice that `shuffle=True`. – Daniel Möller May 02 '18 at 14:34
1

And also 'loss' continues to be 'nan' during training when i use the preprocessing_function parameter. Without preprocessing, the loss decreases significantly. – AKSHAYAA VAIDYANATHAN May 02 '18 at 14:53
That's something for another question.... as your edit shows, your data is being properly preprocessed. – Daniel Möller May 02 '18 at 14:54
Things that may be causing that are the `relu` activations (Resnet uses that a lot). Maybe you need lower learning rates. – Daniel Möller May 02 '18 at 15:03
But I also doubt if my input images after preprocessing got 'nan' values. (an entirely black image for instance) and that has introduced 'nan' output. Do you think that would be the case? – AKSHAYAA VAIDYANATHAN May 02 '18 at 15:15
2

An entirely black image would enter as [-103.939, -116.779, -123.68] with this preprocessing. So, if all your convolutional weights are positive, it might get a zero from relu (thus no gradient, and probably nan). But usually weights are well distributed (but a high learning rate may push everything to zero/no gradient quite quickly) – Daniel Möller May 02 '18 at 16:00

preprocessing images generated using keras function ImageDataGenerator() to train resnet50 model

1 Answers1

Linked