How to fix the image preprocessing difference between tensorflow and android studio?

Question

I'm trying to build a classification model with keras and deploy the model to my Android phone. I use the code from this website to deploy my own converted model, which is a .pb file, to my Android phone. I load a image from my phone and everything worked fine, but the prediction result is totally different from the result I got from my PC.

The procedure of testing on my PC are:

1. load the image with cv2, and convert to np.float32

2. use the keras resnet50 'preprocess_input' python function to preprocess the image

3. expand the image dimension for batching (batch size is 1)

4. forward the image to model and get the result

Relevant code:

img = cv2.imread('./my_test_image.jpg')
x = preprocess_input(img.astype(np.float32))
x = np.expand_dims(x, axis=0)
net = load_model('./my_model.h5')
prediction_result = net.predict(x)

And I noticed that the image preprocessing part of Android is different from the method I used in keras, which mode is caffe(convert the images from RGB to BGR, then zero-center each color channel with respect to the ImageNet dataset). It seems that the original code is for mode tf(will scale pixels between -1 to 1).

So I modified the following code of 'preprocessBitmap' to what I think it should be, and use a 3 channel RGB image with pixel value [127,127,127] to test it. The code predicted the same result as .h5 model did. But when I load a image to classify, the prediction result is different from .h5 model.

Does anyone has any idea? Thank you very much.

I have tried the following:

1. Load a 3 channel RGB image in my Phone with pixel value [127,127,127], and use the modified code below, and it will give me a prediction result that is same as prediction result using .h5 model on PC.

2. Test the converted .pb model on PC using tensorflow gfile module with a image, and it give me a correct prediction result (compare to .h5 model). So I think the converted .pb file does not have any problem.

Entire section of preprocessBitmap

// code of 'preprocessBitmap' section in TensorflowImageClassifier.java
TraceCompat.beginSection("preprocessBitmap");
        // Preprocess the image data from 0-255 int to normalized float based
        // on the provided parameters.
        bitmap.getPixels(intValues, 0, bitmap.getWidth(), 0, 0, bitmap.getWidth(), bitmap.getHeight());

        for (int i = 0; i < intValues.length; ++i) {
            // this is a ARGB format, so we need to mask the least significant 8 bits to get blue, and next 8 bits to get green and next 8 bits to get red. Since we have an opaque image, alpha can be ignored.
            final int val = intValues[i];

            // original
            /*
            floatValues[i * 3 + 0] = (((val >> 16) & 0xFF) - imageMean) / imageStd;
            floatValues[i * 3 + 1] = (((val >> 8) & 0xFF) - imageMean) / imageStd;
            floatValues[i * 3 + 2] = ((val & 0xFF) - imageMean) / imageStd;
            */

            // what I think it should be to do the same thing in mode caffe when using keras
            floatValues[i * 3 + 0] = (((val >> 16) & 0xFF) - (float)123.68);
            floatValues[i * 3 + 1] = (((val >> 8) & 0xFF) - (float)116.779);
            floatValues[i * 3 + 2] = (((val & 0xFF)) - (float)103.939);
        }
TraceCompat.endSection();

Answer 1

This question is old, but remains the top Google result for preprocess_input for ResNet50 on Android. I could not find an answer for implementing preprocess_input for Java/Android, so I came up with the following based on the original python/keras code:

/*
    Preprocesses RGB bitmap IAW keras/imagenet

    Port of https://github.com/tensorflow/tensorflow/blob/v2.3.1/tensorflow/python/keras/applications/imagenet_utils.py#L169
    with data_format='channels_last', mode='caffe'

     Convert the images from RGB to BGR, then will zero-center each color channel with respect to the ImageNet dataset, without scaling. 
     Returns 3D float array
*/
static float[][][] imagenet_preprocess_input_caffe( Bitmap bitmap ) {
    // https://github.com/tensorflow/tensorflow/blob/v2.3.1/tensorflow/python/keras/applications/imagenet_utils.py#L210
    final float[] imagenet_means_caffe = new float[]{103.939f, 116.779f, 123.68f};

    float[][][] result = new float[bitmap.getHeight()][bitmap.getWidth()][3];   // assuming rgb
    for (int y = 0; y < bitmap.getHeight(); y++) {
        for (int x = 0; x < bitmap.getWidth(); x++) {

            final int px = bitmap.getPixel(x, y);

            // rgb-->bgr, then subtract means.  no scaling
            result[y][x][0] = (Color.blue(px) - imagenet_means_caffe[0] );
            result[y][x][1] = (Color.green(px) - imagenet_means_caffe[1] );
            result[y][x][2] = (Color.red(px) - imagenet_means_caffe[2] );
        }
    }

    return result;
}

Usage with a 3D tensorflow-lite input with shape (1,224,224,3):

Bitmap bitmap = <your bitmap of size 224x224x3>;
float[][][][] imgValues = new float[1][bitmap.getHeight()][bitmap.getWidth()][3];
imgValues[0]=imagenet_preprocess_input_caffe(bitmap);

... <prep tfInput, tfOutput> ...

tfLite.run(tfInput, tfOutput);