Significant precision difference in floating point calculations in Java / C#

Question

I know that similar questions have been asked before, but none of the answers solves my problem.

I have 2 functions:

Java

public static void main(String[] args) {
    double h, z, lat0, n0, eSq;
    z    = 4488055.516;
    lat0 = 0.7853981634671384;
    n0   = 6388838.290122733;
    eSq  = 0.0066943799901975545;
    h    = z / Math.sin(lat0) - n0 * (1 - eSq);
    System.out.println(h);
}

C#

public static void Main (string[] args)
{
    double h, z, lat0, n0, eSq;
    z    = 4488055.516;
    lat0 = 0.7853981634671384;
    n0   = 6388838.290122733;
    eSq  = 0.0066943799901975545;
    h    = z / Math.Sin(lat0) - n0 * (1 - eSq);
    Console.WriteLine(h);
}

or

4488055,516/sin(0,7853981634671384)-6388838,290122733*(1-0,0066943799901975545)

for SpeedCrunch, Maxima and LibreOffice Calc.

Results are:

Java:             1000.0000555226579 (same with and without strictfp)
C#:               1000,00005552359
SpeedCrunch (15): 1000,000055524055155
SpeedCrunch (50): 1000,00005552405515548724762598846216107366705932894830
LibreOffice Calc: 1000,000055523590000
Maxima:           1000.000055523589
Maxima:           1.00000005552391142b3 (bfloat - fpprec:20)

As you can see, Java and C# are different at the 9th decimal place. Others are not so uniform either. This is tested on the same OS and same CPU. Tests are done also on 32-bit and 64-bit systems.

How to solve this kind of problem? I thought that precision should be equal to 15 decimal places.

"As you can see, Java and C# are different at 9th decimal place" - but that's the 13th *significant digit*. While that's still a little earlier than I'd have expected, it's important to understand that the precision of `double` is in terms of significant digits (well, binary digits really, but...), not "digits after the decimal point". — Jon Skeet, Jan 31 '15 at 11:33
For interest, the result according to GNU/Linux `bc` is `1000.00005552405516802582` using a default scale of 20. — Stephen C, Feb 01 '15 at 12:55

Patricia Shanahan · Accepted Answer · 2015-01-31T16:37:28.497

The cause of not getting 15 digits is the subtraction of two similar numbers. z / Math.sin(lat0) is about 6347068.978968251. n0 * (1 - eSq) is about 6346068.978912728. With 7 digits before the decimal point, a change in the 9th decimal place in the subtraction result corresponds to a change of less than one part in 10^15 in one of those inputs.

The simplest solution to this type of problem is usually to display only the digits that are supported by reliable digits in the inputs. Very few measurements can be done to one part in 10^12, so in this case it is almost certain that the digits that differ due to floating point rounding error would be dropped.

For example, it looks as though your data relates to location. One of the most carefully measured pieces of such data is the height of Mount Everest. The current best estimate, based on high precision GPS measurement, is "29,035 feet, with an error margin of plus or minus 6.5 feet" Encyclopedia Britannica. An error in the 13th significant digit corresponds to an error of less than one thousandth of an inch in measuring the circumference of the earth.

If the rounding error really is significant relative to the result requirements, and to what is realistically achievable given the accuracy of the inputs, then you might need to look at more clever ways of arranging the calculation or at higher precision arithmetic.

Thanks, I think I kinda get it now. So it turns out that C# handles double operations better than Java (if we compare results with Maxima and SpeedCrunch)? — zoran, Jan 31 '15 at 16:56
@zoran To reach any conclusion about that, you would have to look at millions of calculations, and consider carefully what properties you want. — Patricia Shanahan, Jan 31 '15 at 16:57

Stephen C · Answer 2 · 2015-02-01T00:54:30.200

The precision difference you are observing is not in floating point support per se. Java and C# will be using the same (IEE 768) floating point representations, and quite likely the same instructions.

What you are observing is probably differences in the algorithms that calculate transcendental functions; e.g. the sine function. The theoretical calculation involves summing an infinite series. To get the most accurate answer possible, you keep summing the series forever. To get the answer to a certain precision, you keep summing until the "deltas" are less than the required precision.

In practice, this approach is impractically slow. Practical algorithms are implemented using tables and interpolation, where possible. This is much faster, though you don't get maximal precision.

The precision of Java Math.sin is determined by the algorithms used to compute it. These algorithms are platform specific. Here is what the javadocs for Math say on the subject of precision.

Unlike some of the numeric methods of class StrictMath, all implementations of the equivalent functions of class Math are not defined to return the bit-for-bit same results. This relaxation permits better-performing implementations where strict reproducibility is not required.

And the precision guarantee for Math.sin is:

The computed result must be within 1 ulp of the exact result. Results must be semi-monotonic.

where "ulp" and "semi-monotonic" are specified in the javadoc.

By contrast the javadocs for StrictMath state that a specific version of a specific open-source library is used to do the calculations. The goal is reproducibility; i.e. the same answer on all Java platforms.

Q: So why didn't they make Math.sin more precise? It is possible to get to within 0.5 ulp ...

It is an engineering trade-off between speed and precision. Read this wikipedia article to get a feel for the problem.

My advice is that if you want maximal precision, look for a 3rd party open-source Java library that implements the transcendental functions. And be prepared for your code to be a lot slower.

I don't think that Math.sin is the problem. In this particular case, both Java and C# return the same value for Math.sin. I also used StrictMath and strictfp in Java and there was no difference. — zoran, Feb 01 '15 at 09:45

Significant precision difference in floating point calculations in Java / C#

2 Answers2