as3 Number precision

Question

I am trying to round some numbers in two decimal point and I run into a bizare behavior.

please try the following code:

var num:Number  = 30.25
for (var i = 0 ; i < 100 ; i++){
    var a:Number  = (Math.round(num * 100) / 100)
    var b:Number  =  (Math.round(num * 100) * 0.01 )
    trace (num.toString() + " -- " + a.toString() + " -- " + b.toString())
    num += 0.999;
}

x = y /100 and x = y * 0.01 should be equal.

(And x = y * 0.01 should be faster).

But if I run the above code the result is not always equal.

I get for example

46.23400000000003 -- 46.23 -- 46.230000000000004 47.23300000000003 -- 47.23 -- 47.230000000000004 48.232000000000035 -- 48.23 -- 48.230000000000004 49.23100000000004 -- 49.23 -- 49.230000000000004

while x=y/100 is always correct x=y*0.01 sometimes adds a small value like 0.000000000000004 at the end.

Am I doing something wrong? Has anyone else observed this behavior?

thats just how it is... http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems — arahaya, Mar 27 '12 at 10:36
http://stackoverflow.com/questions/632802/how-to-deal-with-number-precision-in-actionscript — Kodiak, Mar 27 '12 at 12:54

score 2 · Answer 1 · answered Mar 27 '12 at 15:01

In general, in floating point computations you should try to avoid numbers of really different magnitude in the same calculation. That's precisely the issue with these types: the point "floats", so you want to keep the point of one number of the computation close to the point of the other number.

Your question is simply put as

Why is 4623/100 == 46.23 but 4623*0.01 == 46.230000000000004?

For the specific reason, you can dig in the specific of floating point computation, for example here.

4623 is 4.623*10^3 while 0.01 is 1*10^{-3}, notice how the exponent is really different (6 orders of magnitude of difference). While 100 is just 1*10^{2}, much "closer" to 4.623*10^3.

as3 Number precision

1 Answers1