How to pass image to tflite model in android

Question

I have converted a Yolo model to .tflite for use in android. This is how it was used in python -

net = cv2.dnn.readNet("yolov2.weights", "yolov2.cfg")
classes = []
with open("yolov3.txt", "r") as f:
    classes = [line.strip() for line in f.readlines()]
layer_names = net.getLayerNames()
output_layers = [layer_names[i[0] - 1] for i in net.getUnconnectedOutLayers()]
colors = np.random.uniform(0, 255, size=(len(classes), 3))

cap= cv2.VideoCapture(0)


while True:
    _,frame= cap.read()
    height,width,channel= frame.shape
    blob = cv2.dnn.blobFromImage(frame, 0.00392, (320, 320), (0, 0, 0), True, crop=False)
    net.setInput(blob)
    outs = net.forward(output_layers)
    for out in outs:
        for detection in out:
            scores = detection[5:]
            class_id = np.argmax(scores)
            confidence = scores[class_id]
            if confidence > 0.2:
            # Object detected
                center_x = int(detection[0] * width)
                center_y = int(detection[1] * height)
                w = int(detection[2] * width)
                h = int(detection[3] * height)
                # Rectangle coordinates
                x = int(center_x - w / 2)
                y = int(center_y - h / 2)

I used netron https://github.com/lutzroeder/netron to visualize the model. The input is described as name: inputs, type: float32[1,416,416,3], quantization: 0 ≤ q ≤ 255, location: 399 and the output as name: output_boxes, type: float32[1,10647,8], location: 400.

My problem is regarding using this model in android. I have loaded the model in "Interpreter tflite", I am getting the input frames from the camera in byte[] format. How can I convert it into the required input for tflite.run(input, output)?

Answer 1

You need to resize the input image to match with the input size of TensorFlow-Lite model, and then convert it to RGB format to feed to the model.

By using the ImageProcessor from TensorFlow-Lite Support Library, you can easily do image resizing and conversion.

ImageProcessor imageProcessor =
        new ImageProcessor.Builder()
            .add(new ResizeWithCropOrPadOp(cropSize, cropSize))
            .add(new ResizeOp(imageSizeX, imageSizeY, ResizeMethod.NEAREST_NEIGHBOR))
            .add(new Rot90Op(numRoration))
            .add(getPreprocessNormalizeOp())
            .build();
return imageProcessor.process(inputImageBuffer);

Next to run inference with the interpreter, you feed the preprocessed image to the TensorFlow-Lite interpreter:

tflite.run(inputImageBuffer.getBuffer(), outputProbabilityBuffer.getBuffer().rewind());

Refer this official example for more details, additionally you can refer this example as well.