0

At the End is a benefit analysis with Hi Guys,

at the moment I'm stuck with the conversion of an .pb Modell to a fully quantized integer TFLite model in TF2. I used a pre-trained model (SSD MobileNet v2 320x320) from the TensorFlow 2 Detection Model Zoo and pre-trained it on a small own dataset just to test the Workflow (I'm new to Tensorflow and machine learning in general). The .pb Modell is working on my Computer.

Now I want to use the model on a Raspberry Pi with a Coral USB Edge TPU to perform Object Detection. Therefore the model has to be fully quantized to Integer Values. I tried to use the code from the official TensorflowLite documentary, but had a few Errors with the tf2.3.1 version. Therefore I upgraded to tf-nightly=dev2.4.0-dev20200920 which solved some Errors.

I'm able to create an optimized float16 and Dynamic range quantified TFLite model. Or at least I get no errors and a TFLite file with a reduced size. But when I'm trying to convert the model to an int8 model with a representative dataset and I get a Runtime Error. Right now I'm not sure if my code to create a representative_dataset is wrong or the Problem is inside the TFLite scripts in TF2.

Help is very much appreciated because I'm stuck on this problem for days now and can't find any solution.

RuntimeError: Quantization not yet supported for op: 'CUSTOM'.

You can find the Python Script, 2 images and the original saved_model.pb file in the following Google Drive Folder here.

You have to adjust the path saved_model_dir and images_path in the script tflite_full_int.py.

Here is my Code:

import tensorflow as tf
import os
import numpy as np
import cv2


saved_model_dir = '**YOURPATH**/modelandimage/my_model/saved_model'
images_path = '**YOURPATH**/modelandimage/new'


def rep_data():
  for f_name in os.listdir(images_path):
                 file_path = os.path.normpath(os.path.join(images_path, f_name))
                 img = cv2.imread(file_path)
                 img = cv2.resize(img, (320, 320))
                 img = img / 255.0
                 img = np.reshape(img, (1, 320, 320, 3))
                 img = img.astype(np.float32)
                 test = tf.dtypes.as_dtype(img)
                 yield[img]


converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = rep_data
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8  
converter.inference_output_type = tf.uint8 
tflite_quant_model_full_int = converter.convert()

with open('model_quant_int.tflite', 'wb') as f:
  f.write(tflite_quant_model_full_int)

System:

Linux: Ubuntu 18.04.5

tf-nightly=dev2.4.0-dev20200920

Cuda=11.0, Cudnn=8.0.2

Python=3.7.9

Cheers Guys

Ole Pannier
  • 3,208
  • 9
  • 22
  • 33
DemolationPeter
  • 1
  • Looks like an open issue : https://github.com/tensorflow/tensorflow/issues/33925 – Manoj Nov 02 '20 at 05:05
  • May you please share FP or partially quantized model? Tried to convert yours, got only inputs and outputs. – Alex K. Nov 14 '20 at 16:19
  • Sorry for the inactivity. I moved from TF2 back to TF1.15. With TF1.15 I can use QAT and finally convert a graph to nearly full UINT8. I create the quantized model with the command-line. I think the problem is/was the conversion of the Custom Op included from the Object Detection API. The Object Detection API creats at the end of the Graph a custom Operation called PostProcess (with 4 outputs PostProcess:0 .. to Post Process:3). With TF1.15 I can convert the model but at the end the output (PostProcess) is still in float32 but no error is raised and I can convert it. – DemolationPeter Nov 21 '20 at 11:10
  • @Ales K. I created a git repository with my used files. https://github.com/DemolationPeter/TF2_Project.git – DemolationPeter Nov 21 '20 at 11:14

0 Answers0