Using GluonCV models in TensorFlow (Lite)

Question

I'm working on the deployment of models on embedded devices, making performance comparisons and the like. This is an internship, so I'm really constrained on time and can't go about re-implementing / re-trained models but I have to use what is available (I actually asked this explicitly to my supervisor). Since TorchScript is not really as mature as TF Lite, at least from what I've gathered, I'm going with the latter. It's going well with pre-2018 models, but many SotA models like ResNeSt only have code in PyTorch. GluonCV, however, seems to provide a nice selection of models in their zoo and is based on TensorFlow, so I thought there'd be a way of exporting those to a SavedModel, a Keras .h5 or whatever, but I found none after a lot of searching. I found MMdnn but trying it on JSON exported models fails during conversion to IR (I'm attaching the output at the bottom, it seems that MXNet JSON and Gluon JSON are not the same format).

Has anybody else worked with exporting Gluon models to the wild? How did it go?

Thank you!

Output of mmtoir -f mxnet -n resnest200-symbol.json -d resnest200 --inputShape 3,257,257:

/home/kmfrick/Gluon_Tinkering/venv/lib/python3.8/site-packages/mxnet/module/base_module.py:55: UserWarning: You created Module with Module(..., label_names=['softmax_label']) but input with name 'softmax_label' is not found in symbol.list_arguments(). Did you mean one of:
    data
    _defaultpreprocess1_init_mean
    _defaultpreprocess1_init_scale
  warnings.warn(msg)
Warning: MXNet Parser has not supported operator null with name data.
Warning: convert the null operator with name [data] into input layer.
Warning: MXNet Parser has not supported operator null with name _defaultpreprocess1_init_scale.
Warning: convert the null operator with name [_defaultpreprocess1_init_scale] into input layer.
terminate called after throwing an instance of 'dmlc::Error'
  what():  [09:24:49] src/c_api/c_api_symbolic.cc:540: InferShapeKeyword argument name data not found.
Candidate arguments:
    [0]_defaultpreprocess1_init_scale

Stack trace:
  [bt] (0) /home/kmfrick/Gluon_Tinkering/venv/lib/python3.8/site-packages/mxnet/libmxnet.so(+0x307d3b) [0x7f0127eb9d3b]
  [bt] (1) /home/kmfrick/Gluon_Tinkering/venv/lib/python3.8/site-packages/mxnet/libmxnet.so(+0x33a3755) [0x7f012af55755]

gluoncv is based on MXNet, not on TensorFlow – Olivier Cruchant May 16 '20 at 18:47 — Olivier Cruchant, May 16 '20 at 18:47

score 1 · Accepted Answer · answered May 16 '20 at 18:56

1

Gluoncv is an excellent MXNet-based toolkit for computer vision! Several options to deploy gluoncv models to embedded runtimes:

You can use ONNX to convert models to other runtimes, for example CoreML for iOS or NNAPI for Android
You can use TVM
You can use SageMaker Neo + DLR runtime, probably the easiest solution. The git includes examples for Android.

Keep in mind that compilation and portability from a framework to another depends on operators coverage, it may not work for exotic or very recent models

answered May 16 '20 at 18:56

Olivier Cruchant

3,747
15
18

I tried following your suggestions, but I'm having issues. I converted my model to ONNX, but in order to use it with NNAPI it is necessary to convert it to daq. The script provided in the repo returns an error, and in the issues section they report that "It actually fails for all onnx object detection models". Then, I tried using DLR. As I understand, I need to compile my custom model with TVM before...I tried the conversion script from the Docker container, but it says that 'ssd_512_mobilenet1.0_custom is not supported', and lists a bunch of standard models that are available. – Carlo Mar 02 '21 at 11:45
I asked a similar question about this topic (https://stackoverflow.com/questions/66439247/running-gluoncv-object-detection-model-on-android), it would be awesome if you could take a look at it – Carlo Mar 02 '21 at 12:21

Using GluonCV models in TensorFlow (Lite)

1 Answers1

Linked