Stack Overflow
Stack Overflow

Questions tagged [quantization]

Use this tag for questions related to quantization of any kind, such as vector quantization.

Quantization, in mathematics and digital signal processing, is the process of mapping a large set of input values to a (countable) smaller set.

For more, please read the Wikipedia article for more.

444 questions
3
votes
0 answers

TensorFlow Lite quantization fails to improve inference latency

TensorFlow website claims that Quantization provides up to 3x lower latency on mobile devices: https://www.tensorflow.org/lite/performance/post_training_quantization I tried to verify this claim, and found that Quantized models are 45%-75% SLOWER…
Dennis Kashkin
  • 449
  • 5
  • 15
3
votes
1 answer

ValueError: No 'serving_default' in the SavedModel's SignatureDefs. Possible values are 'name_of_my_model'

I am trying to quantize a Tensorflow SavedModel using the following command line: tflite_convert \ --output_file=/tmp/foo.tflite \ --saved_model_dir=/tmp/saved_model But I get the following Error: ValueError: No 'serving_default' in the…
Benjamin Breton
  • 1,388
  • 1
  • 13
  • 42
3
votes
1 answer

Run quantized tensorflow model on FPGA / pure python

I have a model trained in keras which is a simple model trained on MNIST dataset. What I try to do is to rewrite this model and run on FPGA device. In order to do this I want to fully understand how quantized model works. First I converted this…
Damian
  • 31
  • 4
3
votes
1 answer

Finding number of quantizing layers in MATLAB

I'm working on image processing, and I have an image that has been DCT'd and quantized for 8 x 8 blocks of the 512 x 512 matrix, now I have to find how many quantizing levels that the image has. Do I need to take the top left pixel and place it in…
Stina
  • 509
  • 1
  • 7
  • 19
3
votes
0 answers

Any python library for quantization where integer and decimal bits can be fixed

Generally, python libraries provide flexibility to play with the number of bits for decimals. I want something similar to quantizer in MATLAB where I can fix total number of bits and number of bits for decimals. Example: Let's have 8 bit as total…
user3572719
  • 31
  • 2
3
votes
1 answer

TensorFlow: Quantize model using python before save

There are tutorials online showing how to quantize a .pb TensorFlow model, see: https://petewarden.com/2016/05/03/how-to-quantize-neural-networks-with-tensorflow/ What I am wondering is if there is a way to quantize the graph using python before…
abagshaw
  • 6,162
  • 4
  • 38
  • 76
3
votes
1 answer

Matlab Image Quantization not working

I loaded a gray scale image to Matlab to perform the quantization. Unfortunately the output is nothing. I = imread('asa.jpg'); imshow(I) axis off title('Grayscale Image') thresh = multithresh(I,7); valuesMax = [thresh max(I(:))] [quant8_I_max,…
AsankaD7
  • 179
  • 15
3
votes
1 answer

D3 quantize color scale not showing the right color distribution

I'm trying to figure out how to get the right color palette given a dataset with the possibility of a wide range of values. (think github's contribution heat map) Note that the values are dynamic so I'm not sure I can use d3.scale.threshold.…
Sijia Din
  • 1,343
  • 9
  • 13
3
votes
1 answer

Quantize Integers into discrete buckets

I have a list of ~7500 items which all have a similar signature: { revenue: integer, title: string, sector: string } The revenue will range from 0 to ~1 Billion. I'd like to construct a scale such that, given a particular company's…
Nader Hendawi
  • 375
  • 1
  • 7
  • 17
3
votes
2 answers

Algorithm design: Image quantization for most prominent colors

So I'm working on a way to extract dominant colors as perceived by humans from an image. As an example, here's a photo: https://500px.com/photo/63897015/looking-out-for-her-kittens-by-daniel-paulsson Most humans would think the 'dominant' color is…
Joshua Comeau
  • 2,594
  • 24
  • 25
3
votes
1 answer

how to do histogram for dct coeffcient to show Double Quantiztion effect?

what i want to do is a histogram for quantized DCT coefficient for an image, to detect Double Quantization effect. when i use hist(x) it will categorize it in to 10s and if i changed it to hist(x,20) or 30 it does not really show the DQ effect. so…
nadeem
  • 215
  • 2
  • 10
3
votes
1 answer

What's the difference between quantize() and str.format()?

I don't mean what's the technical difference, but rather, what's the faster/more logical or Pythonic, etc. way to do this: def __quantized_price(self): TWOPLACES = Decimal(10) ** -2 return self.price.quantize(TWOPLACES) or …
orokusaki
  • 55,146
  • 59
  • 179
  • 257
3
votes
2 answers

Dithering gouraud-shaded vertex-colored polygons to remove banding

I'm using OpenGL ES on a low-resolution, embedded device. I've applied a vertical color gradient, using vertex coloring, to a large polygon serving as a backdrop to my rendered scene. I can see clear visible banding artifacts in the color…
Hans Sjunnesson
  • 21,745
  • 17
  • 54
  • 63
3
votes
3 answers

Does C have a Quantization function?

I have a buffer with many positive 16bit values (which are stored as doubles) that I would like to quantize to 8bit (0-255 values). According to Wikipedia the process would be: Normalize 16 bit values. I.e. find the largest and divide with…
Louise
  • 6,193
  • 13
  • 36
  • 36
3
votes
2 answers

NeuQuant.js (JavaScript color quantization) hidden bug in JS conversion

NeuQuant.js works well when the image width and height are a multiple of 100: 300x300 Otherwise, there is obviously a bug: 299x300 (These were made with this web app.) I'm 90% sure that the bug is in NeuQuant.js. I have made tests using it with…
forresto
  • 12,078
  • 7
  • 45
  • 64
1 2 3
…
29 30