Stack Overflow
Stack Overflow

Questions tagged [quantization]

Use this tag for questions related to quantization of any kind, such as vector quantization.

Quantization, in mathematics and digital signal processing, is the process of mapping a large set of input values to a (countable) smaller set.

For more, please read the Wikipedia article for more.

444 questions
5
votes
1 answer

The Inverted Multi-Index

I am trying to understand The Inverted Multi-Index, from this paper, which has also a smaller version here. For that purpose, I constructed a toy example and would like someone to verify or/and share with me his/her opinion(s). The example: Assume…
gsamaras
  • 71,951
  • 46
  • 188
  • 305
5
votes
4 answers

How to use python to convert a float number to fixed point with predefined number of bits

I have float 32 numbers (let's say positive numbers) in numpy format. I want to convert them to fixed point numbers with predefined number of bits to reduce precision. For example, number 3.1415926 becomes 3.25 in matlab by using function…
tuming1990
  • 89
  • 3
  • 8
5
votes
1 answer

Matlab Color Quantization

I'm currently working on project which is related with color quantization.The algorithm implies as initial step k-means algorithm . My code untill now : N = 10; K=7; I=imread('baboon.bmp'); Idouble =…
user4272649
5
votes
1 answer

Ffmpeg, avconv and sameq

Earlier I wrote so: ffmpeg -i input.mp4 -sameq output.mp3 ...and thus receive audio from video file. Ffmpeg just taken out or converted audio to mp3 with an appropriate quality. All thanks to key: -sameq [use same quantizer as source] Now in…
xiaose
  • 616
  • 1
  • 7
  • 19
5
votes
1 answer

Custom quantization tables for JPEG compression in Java

As the title says, I'm trying to use custom quantization tables to compress an image in JPEG format. My problem is the resulting file can't be opened and the error is: Quantization table 0x00 was not defined This is how my code looks like: …
rhobincu
  • 906
  • 1
  • 7
  • 22
5
votes
2 answers

What is the fastest available color quantization algorithm for Android?

I'm using the NeuQuant quantization algorithm (https://code.google.com/p/android-gif-project/source/browse/trunk/GIFproject1/src/com/ui/NeuQuant.java?r=5) to reduce a jpeg to a 256-color image but it's very slow (~1 second for a 320x240 image, ~3…
Cat
  • 7,042
  • 8
  • 34
  • 36
5
votes
1 answer

Vector Quantization in Speech Processing Explanation

I'm having trouble determining from this research paper exactly how I can reproduce the Standard Vector Quantization algorithm to determine the language of an unidentified speech input, based on a training set of data. Here's some basic…
atp
  • 30,132
  • 47
  • 125
  • 187
5
votes
3 answers

Extracting Dominant / Most Used Colors from an Image

I would like to extract the most used colors inside an image, or at least the primary tones Could you recommend me how can I start with this task? or point me to a similar code? I have being looking for it but no success.
biquillo
  • 6,469
  • 7
  • 37
  • 40
5
votes
5 answers

Fastest dithering / halftoning library in C

I'm developing a custom thin-client server that serves rendered webpages to its clients. Server is running on multicore Linux box, with Webkit providing the html rendering engine. The only problem is the fact that clients display is limited with a…
Luka Birsa
5
votes
1 answer

create GIF image with fixed colormap using Image Magick

I want to save .gif conversions of other images with a specified colormap. The "-remap cmap.gif" option in Image Magick example: convert -remap cmap.gif input.png output.gif DOES process input.png using the specified colors from cmap.gif, but it…
holistone
  • 63
  • 3
4
votes
0 answers

TensorRT Post Training Quantization (INT8) generates incorrect results

This should really be a question for the NVIDIA team, but they are notoriously bad in providing support, so I am hoping instead that someone intimately familiar with the TensorRT C++ API can help me out. I am unable to put a full minimum…
cyrusbehr
  • 1,100
  • 1
  • 12
  • 32
4
votes
1 answer

"Didn't find engine for operation quantized" error while using dynamic quantization with Huggingface transformer

I am trying to do dynamic quantization(quantizes the weights and the activations) on a pytorch pre-trained model from huggingface library. I have referred this link and found dynamic quantization the most suitable. I will be using the quantized…
joel
  • 1,156
  • 3
  • 15
  • 42
4
votes
1 answer

Tensorflow Lite inference - how do I scale down the convolution layer outputs?

I built a simple CNN model with one convolutional layer and converted it with Tensorflow Lite. (for MNIST!!) So now my model gets 8-bit integer inputs and weights are 8-bit integers too. I wanted to test the parameters I got from TFLite, so I wrote…
C. Kim
  • 41
  • 2
4
votes
1 answer

Why the accuracy of TF-lite is not correct after quantization

I am trying TF-lite converter with TF1.12. And found that the accuracy of TF-lite is not correct after quantization. Take MNIST for example, if convert to f32 with the following command, it still can tell the correct when run…
S.H
  • 69
  • 9
4
votes
1 answer

How to make sure that TFLite Interpreter is only using int8 operations?

I've been studying quantization using Tensorflow's TFLite. As far as I understand it is possible to quantize my model weights (so that they will be stored using 4x less memory) but it doesn't necessary implies that the model won't convert it back to…
Gustavo Sutter
  • 94
  • 1
  • 7
1 2
3
…
29 30