0

I trained and quantized a Tensorflow model on a Ubuntu 18.04 machine and I converted it to tflite format. Then I deployed it on a Linux Yocto board equipped with a NPU accelerator, tflite_runtime and NNAPI. I noticed that the same tflite model outputs different predictions when using the CPU on my PC and the NPU+NNAPI on the board for inference. The predictions are often similar, but in some cases they are completely different. I tried to disable NNAPI on the board and to make inference using the CPU and the results were the same as on the PC CPU. So I think that the problem is the NNAPI. However, I don't know why this happens. Is there a way to prevent it or to make the network more robust during training?

firion
  • 296
  • 3
  • 12
  • I have not heard that before. The tflite team will be very interested. I think you should open an issue at the github page with enough information. – Farmaker May 28 '21 at 12:55

1 Answers1

0

firion,

The NNAPI team is also interested in learning more.

Small variations are to be expected but completely different results should not be.

Have you tried on different devices? Do you see the same variations?

You mentioned a Linux Yocto build. Are you running your test using Android on Yocto, or using a Linux build of NNAPI?

Jean-Luc Brouillet
  • 41
  • 1
  • I also tried with pyarmnn instead of NNAPI delegate and the problem persisted. So the problem was below the NNAPI layer. However, after some trials, I managed to significantly reduce this behaviour. The original version of my network did not have the last Dense+Sigmoid layer quantized because I could not manage to do it. I changed the Sigmoid to Softmax and tried to quantize the whole network. Now the predictions are almost the same. I am using a Linux build. – firion Jun 04 '21 at 06:37
  • Thanks for the details. It seems like you've worked around the error. It would be good to figure out what the original error was however. Do you think you can produce a minimal test case? – Jean-Luc Brouillet Jun 11 '21 at 19:15
  • sorry for replying to an old answer. I encountered another problem that seems related to NNAPI. I opened an issue on github. Could you take a look please? https://github.com/tensorflow/tensorflow/issues/60563 – firion May 12 '23 at 12:42