Stack Overflow
Stack Overflow

Questions tagged [quantization]

Use this tag for questions related to quantization of any kind, such as vector quantization.

Quantization, in mathematics and digital signal processing, is the process of mapping a large set of input values to a (countable) smaller set.

For more, please read the Wikipedia article for more.

444 questions
2
votes
0 answers

Bitwise shift operator in Pytorch and ONNX export

I have a quantized network from a framework, that I want to export as an ONNX file. The quantization requires adding intermediary layers performing bitwise right shift to avoid overflow. I have to insert these layers in between some other layers of…
Fabrice Auzanneau
  • 95
  • 9
2
votes
1 answer

TensorFlow 2.0 TFLite Quantized export with UINT8 weights

In the previous TensorFlow version I used (1.9) I was able to quantize my network with UINT8 weights stored in the Conv2D operation inputs. Now with TensorFlow 2.0 using a Keras model, post-training quantization gives me INT8 weights with seemingly…
Kalen White
  • 77
  • 8
2
votes
1 answer

Palette quantization for pixel mixing

For graphic modes using a hardware palette, an illusion of more colors can be achieved by quickly switching between two images (I don't know about any generic name for this). What is some algorithm to calculate an optimal (or semi-optimal) palette…
Fabel
  • 1,711
  • 14
  • 36
2
votes
1 answer

Question about inconsistency between tensorflow lite quantization code, paper and documentation

In this paper (Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference) published by google, quantization scheme is described as follows: Where, M = S1 * S2 / S3 S1, S2 and S3 are scales of inputs and output…
chihyang
  • 187
  • 6
2
votes
2 answers

How to convert TFLite model to quantized TFLite model?

I have a tflite file and I want to quantize it. How to convert TFLite model to quantized TFLite model?
Uijeong Jeong
  • 21
  • 4
2
votes
1 answer

Quantization not yet supported for op: 'DEQUANTIZE' for tensorflow 2.x

I am conducting QAT by keras on a resnet model and I got this problem while converting to tflite full integer model. I have tried the newest version tf-nightly, but it does not solve the problem. I use quantization annotated model for Batch…
dtlam26
  • 1,410
  • 11
  • 19
2
votes
1 answer

Is it possible to configure TFLite to return a model with bias quantized to int8?

I'm working with Keras/Tensorflow to develop an ANN that will be deployed to a low-end MCU. For this purpose, I have quantized the original ANN using the post-training quantization mechanism offered by Tensorflow Lite. If the weights are indeed…
Miguel Costa
  • 33
  • 2
2
votes
2 answers

TensorFlow Quantization Crashing During Conversion

I'm trying to quantize a model with TensorFlow 2.3.0. I'm having some trouble with saving the final result, and it's not clear to me what the exact issue is. Here's my code import os os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2" import tensorflow as tf…
Alex R.
  • 1,397
  • 3
  • 18
  • 33
2
votes
2 answers

TFlite interpreter raises a RuntimeError when allocating tensors for a quantized model. Failed assertion involving scale_diff and output_scale

Dear developers and NN enthusiasts, I have quantized a model (8-bit post-training quantization) and I'm trying to do inference with the resulting model using tflite interprter. In some cases the interpreter runs properly, and I can do inference on…
Alberto Escalante
  • 31
  • 5
2
votes
0 answers

Understanding the quantization value

I have a model post-training-quantized in different ways and inspecting it via Netron. Model 1 is quantized with a Tensorflow version 1.3; Model 2 is quantized with a Tensorflow version 1.15.3 using also in and output quantization. model 2 uses…
Florida Man
  • 2,021
  • 3
  • 25
  • 43
2
votes
0 answers

TensorFlow Lite Full-Integer Quantization fails in TF 2

I've trained resnet50v2 and densenet 169 models. TensorFlow nightly 2.3.0-dev20200608. The model works fine and I tried some optimization such as "simple" tf lite, tf lite dynamic range, tf lite 16float, and they all work fine (the accuracy is…
Stefano555
  • 31
  • 3
2
votes
1 answer

Tensorflow - Converting a pruned model to a lower quantization with tflite

I'm trying to quantize a pruned tensorflow model using TFLite. Both the pruning and quantization work independently, but when I try to prune and then quant I get this error: ValueError: Input 0 of node…
Sig
  • 91
  • 2
  • 12
2
votes
1 answer

Tensorflow model quantization best strategy

I'm perplexed by the Tensorflow post-training quantization process. The official site refers to Tensorflow Lite Quantization. Unfortunately, this doesn't work in my case, that is, TFLiteConverter returns errors for my Mask RCNN model: Some of the…
Mariano Italiano
  • 301
  • 2
  • 8
2
votes
1 answer

Converting Keras model with TFLiteConverter to quantized tflite version results in NOTYPE error

When converting and doing 8-Bit quantization of a keras model I ran into a strange error that did not happen for image data sets. import tensorflow.python.keras.backend as K import tensorflow as tf from tensorflow import keras from…
Alexander
  • 1,422
  • 11
  • 15
2
votes
1 answer

post training quantization for mobilenet V1 not working

I am trying to convert mobilenet V1 .pb file to quantized tflite file. I used the below command to do the quantization: tflite_convert \ --output_file=/home/wc/users/Mostafiz/TPU/models/mobilnet/test2_4thSep/mobilenetv1_test5.tflite \ …
MMH
  • 1,676
  • 5
  • 26
  • 43
1 2 3
…
29 30