5

I have a custom neural network written in Tensorflow.Keras and apply the hard-swish function as activation (as used in the MobileNetV3 paper):

$$h-swish = x \cdot \frac{ReLU6(x+3)}{6}$$).

Implementation:

def swish(x):
    return x * tf.nn.relu6(x+3) / 6

I am running quantization aware training and write a protobuf file at the end. Then, I am using this code to convert to tflite (and deploy it finally on the EdgeTPU):

tflite_convert --output_file test.tflite --graph_def_file=test.pb --inference_type=QUANTIZED_UINT8 --input_arrays=input_1 --output_arrays=conv2d_3/Sigmoid --mean_values=0 --std_dev_values=255 --default_ranges_min=0 --default_ranges_max=6

And this works perfectly, when I am not dividing by 6, however, when dividing by 6 I am getting this error:

Unimplemented: this graph contains an operator of type Div for which the quantized form is not yet implemented.

I am using TF 1.14 to train and TF 1.15 last nightly build to convert to TFLITE; I am struggling to get TF 2.x to work for some strange HDF5 incompatibilities, but it would be great if someone knows how to circumvent this issue... Thanks!

anki
  • 765
  • 5
  • 14

1 Answers1

11

Since it is a constant division, you could just multiply by (a close approximation of) the inverse:

def swish(x):
    return x * tf.nn.relu6(x+3) * 0.16666667
jdehesa
  • 58,456
  • 7
  • 77
  • 121
  • ok, I feel sooo stupid now - it works like a charm. Actually, do you know by any chance to convert this to a piece-wise function as suggested by the MobileNetV3 paper? I have actually no clue here. – anki Feb 21 '20 at 10:36
  • @anki Not stupid at all, I'm sure many other people have been / will be stumped by the same error. I'm not sure what they mean by the "piece-wise function" in the paper, I mean the piece-wise part here is the ReLU6 which TF implements... You could replace with something like `tf.where(x <= -3, 0., tf.where(x >= 3, x, x * (x + 3) * 0.16666667))`, but I don't think that should be any faster (doesn't seem like it is in a quick benchmark). – jdehesa Feb 21 '20 at 11:01
  • It is already a piecewise function due to relu6 being piecewise. – Radek Jul 14 '23 at 06:12