0

I'm trying to convert a tensorflow model to CoreML model.

Here is what I do

import tensorflow as tf import coremltools as ct

# copy paste the labels class_labels = [ 'label 1', ['label 2'] .. ['label n']]

model = tf.keras.models.load_model('my_tf_model') 
config = model.get_config() # check out the model config   
print(config)
#convert to CoreMl model  mlmodel = ct.convert(model
                     , convert_to="mlprogram", 
                     inputs=[ct.ImageType(
                         name="sequential_1_input",
                            shape=(1, 256, 256, 3),
                            # scale=1.0 / 255.0,
                            bias=[-10,-10,-10],
                            color_layout="RGB"
                     )],
                     # outputs=[ct.ImageType()],
                     source="tensorflow",
                     classifier_config=ct.ClassifierConfig(class_labels)#set the labels
                     ) shape=(1, 256, 256, 3) -- 1(I don't know what it is)

256,256(model width and height), 3 again - I don't know what it is

bias=[-100,-100,-100] - I'm using this to normalize the probability, else I get -0.2, -2.0. With this I'm actually getting something like 20%, 50%

Than I save the model

mlmodel.save("MyModel.mlpackage")

The model is saved with the name MyModel.mlpackage as expected

Then in xcode I'm using the model

let config  = MLModelConfiguration()
let imageClassifier = try MyModel(modelconfig: config)
   
let imageClassifierModel = imageClassifier.model
let result = try imageClassifier.prediction(sequential_1_input: buffer!)
          
// get the label and probability
let className: String = result.classLabel
let confidence: Double = result.classLabel_probs[result.classLabel] ?? 0.0

Again, all good.

The problem is that the prediction is wrong, each time. If I make the same prediction with the same image in Python works with high probability (100% most of the time)

So Why? Is coreML that bad? Am I doing something wrong?

Should I switch to tensorlite? Is there any other option for ios?

As far as I understood the CoreML it supposed to normalize the image pixel, and it looks like it doesn't, This is the only thing that I can think of, but I have no idea how to do this in CoreML.

update, I managed to figure out a problem. The autotuneAUTOTUNE =

tf.data.AUTOTUNE

train_ds = train_ds.cache().shuffle(1000).prefetch(buffer_size=AUTOTUNE)
val_ds = val_ds.cache().prefetch(buffer_size=AUTOTUNE)
normalization_layer = layers.Rescaling(1./255)

normalized_ds = train_ds.map(lambda x, y: (normalization_layer(x), y))
image_batch, labels_batch = next(iter(normalized_ds))
first_image = image_batch[0]
# Notice the pixel values are now in `[0,1]`.
print(np.min(first_image), np.max(first_image))

So I'm guessing is the normalization. I tried to use the scaling to reproduce this in coremltools but it does not work.

Once I commented out this code and went to xcode and drag and drop images in the preview, it works as expected, but not in code.

So I'm guessing the issue is within the buffer, where I'm converting the image to what coremltools need

func buffer(with size:CGSize) -> CVPixelBuffer? {
       if let image = self.cgImage {
           let frameSize = size
           var pixelBuffer:CVPixelBuffer? = nil
           let status = CVPixelBufferCreate(kCFAllocatorDefault, Int(frameSize.width), Int(frameSize.height), kCVPixelFormatType_24ARGB , nil, &pixelBuffer)
           if status != kCVReturnSuccess {
               return nil
           }
           CVPixelBufferLockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags.init(rawValue: 0))
           let data = CVPixelBufferGetBaseAddress(pixelBuffer!)
           let rgbColorSpace = CGColorSpaceCreateDeviceRGB()
           let bitmapInfo = CGBitmapInfo(rawValue: CGBitmapInfo.byteOrder32Little.rawValue | CGImageAlphaInfo.premultipliedFirst.rawValue)
           let context = CGContext(data: data, width: Int(frameSize.width), height: Int(frameSize.height), bitsPerComponent: 8, bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer!), space: rgbColorSpace, bitmapInfo: bitmapInfo.rawValue)
           context?.draw(image, in: CGRect(x: 0, y: 0, width: image.width, height: image.height))
           CVPixelBufferUnlockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
           
           return pixelBuffer
       }else{
           return nil
       }
   }

And I think is because of the way the buffer is created

kCVPixelFormatType_24RGBA

If I print the mlmodel I get

input {
  name: "sequential_1_input"
  type {
    imageType {
      width: 64
      height: 64
      colorSpace: RGB
    }
  }
}
output {....

So this means the input must be rgb and not rgba or argb or anything But it doesn't say if is 24 or 32

If I drag and drop images in the model previwer works perfect with the code commented(see up)

And I might be missing something else, I just couldn't figure out what.

iulian.flester
  • 25
  • 5
  • I'd double-check to make sure you are doing the same preprocessing/postprocessing as when the inference works. – Jeshua Lacock Sep 11 '22 at 19:32
  • As for how to normalize a image, that is a completely different question which you can find answers. – Jeshua Lacock Sep 11 '22 at 19:32
  • @JeshuaLacock I added more details, So first is the autotune that makes the conversion wrong in the first place (adding same 1./255 scaling has almost no effect) and then is the buffer that supply the wrong data to the model, but how to fix this, I have no idea for now – iulian.flester Sep 13 '22 at 06:57

1 Answers1

0

I finally make it work

So the conversion was inaccurate because of a library I was using, it was a warning asking me to downgrade, and initially, I thought that it works, why should I use an older version? But then I said, let me give to caesar what belongs to caesar. So now it actually works!!!

Now for CoreML:

  • I used the preview from CoreMl model and I saw that it was working
  • fixed the buffer function and all is working now, the idea is garbage in garbage out.

Thank you Jeshua Lacock for your comment

iulian.flester
  • 25
  • 5