0

I am trying to perform subtraction of the numbers

in tcl or python: expr { 202.2246-0.42} = 201.80460000000002

ideally the answer should be 201.8046

One way to solve it would be to convert it to integer and do the subtraction and convert it back but the no of decimal digits are not fixed and I would not prefer to multiply it with 10^9 (int limit) - maybe as a last resort.

The precision is highly important. Why does the result varies from the exact result ? How can I control it in a generic way - not only for these operands?

kil47
  • 53
  • 1
  • 9
  • 2
    Most Python implementations use the IEEE-754 binary64 format for floating-point. That format represents numbers by scaling integers by powers of two. 202.2246 and .42 are not representable in that format. The nearest representable values are 202.22460000000000945874489843845367431640625 and 0.419999999999999984456877655247808434069156646728515625. So rounding errors occur even before the subtraction is performed; they occur when the numerals are converted to binary64. You cannot “control” this in the floating-point format; it is inherent to the format… – Eric Postpischil Feb 23 '23 at 11:54
  • 1
    … It is possible to determine from the floating-point result what the real-number arithmetic or decimal result would be given known limits on how many digits are involved (left and right of the decimal point, or total) and how many and what arithmetic operations are performed. For simple cases, that is done simply by rounding to the known number of digits. – Eric Postpischil Feb 23 '23 at 11:55
  • this could work.. round to the nearest mantissa- max no of digits after decimal - after the operation . I give it a shot.. thx for the hint – kil47 Feb 23 '23 at 12:04
  • thank you ! this indeed worked . I will close the topic. Can you please also tell me the library how can i visualize the full value of the floating numbers - 202.2246 = 02.22460000000000945874489843845367431640625 – kil47 Feb 27 '23 at 08:34

0 Answers0