0

I have a tflite model trained on BGR data. How can I make it work properly with RGB images?

UPDATE

I want to use it with the material-showcase app: https://github.com/googlesamples/mlkit/tree/master/android/material-showcase

@Farmaker @JaredJunyoungLim . Thank you very much for your answers. I've updated the question. At first I was thinking about converting the model itself, so it wouldn't require any changes in the code. For example, the converter to the OpenVINO format has an option to reverse input channels. I have also tried to set the BGR ColorSpace in the metadata, but have found out that it's most probably not possible.

I guess I'll go with your suggestion then. In the linked code, there is indeed the ByteBuffer (FrameProcessorBase.kt). I guess this is the place to change the order of the channels (after the line 70):

val frame = processingFrame ?: return

However, how can I change the order of channels, if this is just a ByteBuffer? Do I need to figure out the way data is stored in it? For example there is R,G,B,R,G,B,R,G,B,... etc. for every pixel? Or maybe there is some more elegant way to that?

I can see that the format is set to IMAGE_FORMAT_NV21, which is YCrCb

UPDATE 2

For what I've tested (

Log.d("ByteBuffer", frame.toString())

), it seems that the ByteBuffer takes 1.5 bytes per pixel:

java.nio.HeapByteBuffer[pos=0 lim=3110401 cap=3110401]

(Resolution: 1920x1080; 3110400/1920/1080=1.5)

So it uses 12 bits per pixel, which means 4 bits per channel per pixel. That's a bit strange, because I would suspect at least 8 bits per channel per pixel (0-255).

So I guess that maybe it's compressed.

Hououin Kyouma
  • 11
  • 4
  • 1
    Without looking at the code, it's hard to provide a detailed answer, but if the model itself is trained with BGR data, then the general suggestion is to preprocess RGB data to convert them to BGR data (e.g. change the column order), and then use that data for further training or inference. – Jared Junyoung Lim Sep 20 '22 at 01:21
  • You have to create a proper ByteBuffer that will have the order BGR from an image that is RGB by changing the order of the channels. Then you feed your Interpreter. – Farmaker Sep 20 '22 at 03:27
  • Take a look at this project https://github.com/farmaker47/photos_with_depth it contains some useful functions inside the ImageUtils class. You will get some hints how to change the order of the channels. – Farmaker Sep 22 '22 at 04:56

0 Answers0