2

In the previous TensorFlow version I used (1.9) I was able to quantize my network with UINT8 weights stored in the Conv2D operation inputs. Now with TensorFlow 2.0 using a Keras model, post-training quantization gives me INT8 weights with seemingly no option for unsigned weights. Is it not possible to control the sign on the weights in Conv layers with TF 2.0?

import tensorflow as tf
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
def representative_dataset_gen():
  for _ in range(num_calibration_steps):
    # Get sample input data as a numpy array in a method of your choosing.
    yield [input]
converter.representative_dataset = representative_dataset_gen
# there is no such tf.lite.OpsSet.TFLITE_BUILTINS_UINT8
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8  # or tf.int8 ( note this has zero effect on the tensors produced for Conv2D operations -- all of which include signed int8 unless you were to output the model as float16/32 )
converter.inference_output_type = tf.uint8  # or tf.int8
tflite_quant_model = converter.convert()
Kalen White
  • 77
  • 8

1 Answers1

1

The most recent version of TensorFlow (2.5) utilizes a more robust quantization scheme on the Convolutional networks where each filter depth maps to different quantization levels. As of current there is not a way to utilize the prior method.

Kalen White
  • 77
  • 8
  • Hi, I have a similar need to quantize my model to UINT8 , can you share how to do this in tensorflow 1.9 ? Thanks a lot! – cheng yang Aug 06 '21 at 04:43
  • Does this mean that UINT8 quantization is not supported by TF 2.x? – ai2ys Dec 01 '21 at 21:05
  • @ai2ys From what I can tell, yes. The signed int8 methods are the way forward and I can report it does definitely improve performance in its current implementation, if that helps. – Kalen Dec 02 '21 at 23:08
  • @Kalen I am asking specifically for the uint8 support, because of a hardware only supporting uint8 operations. I was hoping for using TF for the quantization task instead of the vendor tool. – ai2ys Dec 03 '21 at 05:41
  • @ai2ys As far as I can work out, the tf2x scripts involved in the export had zero functionality when giving it uint8 as params. I tried every permutation available at the time to no avail. I would recommend using the earlier version. – Kalen Dec 03 '21 at 21:13