7

I really don't think this is a precision problem, the answer should be about 0.226. Here's the exact code:

val = I(i,j)
bucketSize    
pos = val / bucketSize

I is just a matrix I'm taking values from. Here is the output from MATLAB:

val =

   29

bucketSize =

   128

pos =

   0

What am I missing?

gnovice
  • 125,304
  • 15
  • 256
  • 359
jakev
  • 2,815
  • 3
  • 26
  • 39

4 Answers4

17

My guess would be that your matrix I is pixel data loaded from an image file, which will have values that are typically unsigned 8-bit integers. As already mentioned, converting both integer values to a double precision value will ensure that MATLAB performs floating point division instead of integer division (which will round off the result).

Converting one value to double precision is insufficient:

For all binary operations in which one operand is an array of integer data type (except 64-bit integers) and the other is a scalar double, MATLAB computes the operation using elementwise double-precision arithmetic, and then converts the result back to the original integer data type.

If you'd like to find out more about the different numeric data types in MATLAB, you can check out this documentation.

gnovice
  • 125,304
  • 15
  • 256
  • 359
12

try:

double(val)/double(bucketSize)
AmirW
  • 816
  • 9
  • 20
1

I got it, the problem is that my matrix for some reason contained uint8's, not doubles. Just changed val=I(i,j) to val=double( I(i,j) ) and all is well. Thanks.

jakev
  • 2,815
  • 3
  • 26
  • 39
0

These variables are probably ints rather than doubles or longs. Does 1/2 return .5? Do other operations work?

gary
  • 4,227
  • 3
  • 31
  • 58
  • Yes. For instance, 29/128 results in the correct answer. I was under the impression Matlab was typeless? How do I set the variables to be doubles? – jakev Sep 11 '10 at 02:58