When using excel if I calculate 6/100 to get the percentage value it returns 0.060000 but when i do the same calc in objective-c I get 0.059999998658895493
Does anyone know why there is a difference?
Asked
Active
Viewed 117 times
1
-
4This is not specific to Objective-C. You may want to peruse this: < http://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html > Basically, 0.06 cannot be represented exactly using a `float`, so you're seeing the closest it can get. – John Calsbeek Feb 14 '12 at 11:01
-
3Also, it's likely that Excel is lying to you. Note that if you round off the number you get in C after six decimal places, you get the same result that Excel gave you. – John Calsbeek Feb 14 '12 at 11:02
-
Please reference this answer: http://stackoverflow.com/questions/8758156/rounding-double-values-in-c-like-ms-excel-does-it – BeRecursive Feb 14 '12 at 11:05
1 Answers
4
Floating point values cannot represent all fractional numbers exactly. This the same behavior we see with some numbers in decimal, 1/3 can not be represented exactly in decimal, it is only approximated as 0.33333333...
Computer fractional numbers are base 2 where our decimal numbers are base 10. Different fractional numbers can be represented with complete accuracy in different bases.
Because of this there is a number class in Cocoa, NSDecimalNumber that operates in base 10 and can represent 6/100 exactly as 0.060000
The difference you see in Excel could be either because it is using decimal math (probably not) or that it is rounding the number so that the display inaccuracy is not apparent.
