2

My problem is quite simple to describe, in flex:

0.8 - 0.2 = 0.6000000000000001

Anyone got this before, I'm sure the first two members are 0.8 and 0.2 and are Number class, why would this happen??

And another thing, I cast the value from * type like this:

var value:*=0.8;
var castValue:Number = Number(value);

But nothing more, if I trace value I get 0.8 not 0.800000000000001 or something.

Rad'Val
  • 8,895
  • 9
  • 62
  • 92

2 Answers2

4

This is a fairly common problem in all languages because of Floating Point Number Precision. There's not much you can do about as it is an error value that's possible on all machine.

What you can do is set a precision that you want from the number:

trace(castValue.toFixed(5)); // Gives five decimals after the point

Often times, Flash rounds off these number for you in the conversion of Number into a String (for visual displaying) which removes most of these errors from being viewable to the user.

J_A_X
  • 12,857
  • 1
  • 25
  • 31
  • +1 great answer. The AS3 docs actually explain to expect slightly different values for ops involving floating point numbers on different machines due to varied CPU architecture, which, as noted, is to be expect with any language. –  May 27 '11 at 20:04
1

Essentially, there are some numbers that just can't be represented exactly in binary.

Although this page about floating point numbers is about Java and not Actionscript, the difficulty exists in a lot of languages. There's a section in that article (see "Rounding errors") that mentions the problem with numbers like .6

It's a long read, but I found it quite helpful (and not as long or math-heavy as this page about Floating-Point Arithmetic).

dustmachine
  • 10,622
  • 5
  • 26
  • 28
  • Indeed, good point, but strange enough this works fine `var a:* = -8; var castA:Number = Number(a); trace(castA) var b:* = -2; var castB:Number = Number(b); trace(castB) trace(castA-castB);` – Rad'Val May 27 '11 at 16:05
  • And still the values are the same, so, probably is something wrong with my code... – Rad'Val May 27 '11 at 16:06