0

I have following code in place which pass camera taken image as Bitmap to be feeded in to Machine learning model.

CameraFragment.kt

private lateinit var photo: Bitmap

private fun takePhoto() {
    val imageCapture = imageCapture ?: return
    imageCapture.takePicture(cameraExecutor, object :
        ImageCapture.OnImageCapturedCallback() {
        override fun onCaptureSuccess(image: ImageProxy) {
            super.onCaptureSuccess(image)
            photo = imageProxyToBitmap(image)
        }
    })
}


override fun onCreateView(
    inflater: LayoutInflater, container: ViewGroup?,
    savedInstanceState: Bundle?
): View {
    ...
    val ml = MachineLearning()
    ml.downloadModel()
    val prediction = ml.predict(photo)
    ...
}

MachineLearning.kt

fun predict(photo: Bitmap): Boolean {

    // image pre-processing

    ...

}


private fun downloadeModel() {
    val conditions = CustomModelDownloadConditions.Builder()
        .requireWifi()
        .build()
    FirebaseModelDownloader.getInstance()
        .getModel("model-name", DownloadType.LOCAL_MODEL_UPDATE_IN_BACKGROUND, conditions)
        .addOnCompleteListener { customModel ->

            customModel.addOnSuccessListener { model: CustomModel? ->
                modelFile = model?.file!!
                interpreter = Interpreter(modelFile)
            }
        }
        .addOnFailureListener {
            Log.d(TAG, "Internet is necessary to download the model!")
        }
}

But I am not sure how to do the image pre-processing. According to this codelab ImageProcessor class should be used for this. But I am not sure how to do this?

Given below is analogous Python code to test the model using single image

def testing_tflite_model(sample_type, quantized):
    converted_model = "models/converted/model.tflite"
    if quantized:
        converted_model = "models/converted/model-quantized.tflite"

    bad_image_path = "..."
    good_image_path = "..."
    img = io.imread(bad)

    if sample_type == "good":
        img = io.imread(good_image_path)

    resized = resize(img, (106, 106)).astype('float32')
    test_image = np.expand_dims(resized, axis=0)
    normalized_image = test_image - 0.5

    prediction = run_tflite_model(converted_model, normalized_image)
    if prediction == 1:
        print("Bad")
    else:
        print("Good")

My try so far:

val imageProcessor = ImageProcessor.Builder()
        .add(ResizeOp(106, 106, ResizeOp.ResizeMethod.NEAREST_NEIGHBOR))
        .build()

But how to normalize the image as in python code?

user158
  • 12,852
  • 7
  • 62
  • 94

1 Answers1

1

For the image normalization in the TFLite support library, you can add the following normalization op into the image processor. For example,

ImageProcessor imageProcessor =
    new ImageProcessor.Builder()
        .add(ResizeOp(106, 106, ResizeOp.ResizeMethod.NEAREST_NEIGHBOR))
        .add(new NormalizeOp(127.5, 127.5))
        .build();

For the details, please check out from this link.

Jae sung Chung
  • 835
  • 1
  • 6
  • 7
  • How did you decide on the value `127.5`? – user158 Apr 16 '21 at 11:25
  • For the parameter values, please check out this [link](https://www.tensorflow.org/lite/convert/metadata#normalization_and_quantization_parameters). – Jae sung Chung Apr 16 '21 at 21:41
  • In order to accept the answer please add the reasoning behind the `127.5`, the link you mentioned in the comment have `127.5` for MobileNet how that is the correct value for my case? – user158 Apr 17 '21 at 06:10
  • The mean and std values should be determined by the data that will be encountered with the model to reflect the image data set. – Jae sung Chung Apr 17 '21 at 08:22
  • Model trained / test dataset mean = .5 and std dev = 1 in my case but plugging them to `NormalizeOp` give me incorrect prediction in compared to 127.5, 127.5. Why is that? – user158 Apr 17 '21 at 08:54
  • `0.5` in `normalized_image = test_image - 0.5` is the mean – user158 Apr 17 '21 at 08:56
  • Normalization option can be applicable when there is a normalization process in the training. I am not sure that the model is trained with the normalized inputs. – Jae sung Chung Apr 17 '21 at 09:27
  • that's why I said model trained / test dataset mean = .5 and std dev = 1 but plugging them to `NormalizeOp` give me in correct prediction but 127.5, 127.5 works fine. which is strange. I am using slightly modified version of [this script](https://github.com/jasonchang0/SoBr/blob/master/bin/convNetKerasLarge.py) have not change the the normalization specified in https://github.com/jasonchang0/SoBr/blob/master/bin/convNetKerasLarge.py#L56 – user158 Apr 17 '21 at 09:36
  • Better to ask it in the another post. Maybe another model experts can help your case further. – Jae sung Chung Apr 17 '21 at 09:53