0

I have a Image Classification problem (black and white stick figures) The issue is that two classes never get inferred.

The training dataset has 500 classes with 100 or more (299x299,1) samples per class. The classes that fail to be recognized have been augmented from 100 samples to 140+ samples. Most recently I have trained CreateML with autosplit, but have used a separately generated Validation dataset. No difference in results. The internal "FeaturePrint" model in CoreML just seems to refuse to recognize these two glyphs.

Of note, a simple TensorFlow/Keras model (8 layer CNN/ANN) works just fine for all classes. However, the tf model has poorer performance on untrained, 'wild' input images, and the tf model when converted to ML format for deployment is 160 MB, whereas the CreateML model is 8 MB.

Any suggestions on wrangling CreateML?

BlueskyMed
  • 765
  • 7
  • 24
  • 1
    Yes, don't use Create ML if it doesn't give good results for you. The image classifier is basically SqueezeNet that's pretrained on ImageNet. The only thing that gets trained is a single layer on top of that. You can probably design a better architecture for this, especially if you already know TF/Keras. – Matthijs Hollemans Mar 06 '21 at 19:32
  • Hi Matthijs, I have appreciated reading your articles and answers over the past few years. They have been helpful and inciteful. In this case, it is interesting that the simple CNN/ANN is much less accurate (45% vs 85% on purely novel test samples) than the FeaturePrint. If you look at top5 results, the Apple model is better than 95%. It just fails to ever recognize two fairly common classes. I mean they always have 0.0 probability. All the other 498 classes get inferred with good results. Just weird! The 20:1 size differential is motivation to keep poking at CreateML. – BlueskyMed Mar 06 '21 at 21:22
  • In case that leads nowhere, I am glad I got PlaidML running for Keras! Now, only Intel would make Plaid play nice with TensorBoard! Thanks! – BlueskyMed Mar 06 '21 at 21:23
  • Ah yes if you're using the FeaturePrint model then that should give pretty good results, although it's weird that it doesn't see those two classes. Note that the 20:1 size difference is only because FeaturePrint is built into the OS already. ;-) If you're adventurous, you can also train your own model on top of FeaturePrint. – Matthijs Hollemans Mar 07 '21 at 10:57
  • Yes, considering how different the dataset contents are, the CreateML Classifier does pretty well with black and white stick figures. I think the size difference between the models may also be due to the nature of squeezenet which prides itself on economy of weights. I think I will try to roll my own version of squeeze to train with my data and see what happens. Have you got a link or pointer to where I can explore layering a custom model over FeaturePrint? – BlueskyMed Mar 07 '21 at 14:23
  • I wrote a bit about it in my Core ML book, but basically you write a Swift program that uses FeaturePrint to extract the features for all your images, then train your own model on top of these features. – Matthijs Hollemans Mar 07 '21 at 19:25

0 Answers0