RoundingMode.DOWN for BigDecimal rounding up instead

Question

I am trying to write a program that calculates the square root of an integer (n) with a specified precision (k) provided by user using the newton's method. The program is working fine however I am noticing what I think is a rounding issue. Can someone help me solve this?

For example:

for input: n = 81 and k = 7 / for input: n = 87026 and k = 11
program is printing: 9.0000001 / program is printing: 295.00169491039
what I want to print: 9.0000000 / what I want to print: 295.00169491038

Here is my code:

Scanner userInput = new Scanner(System.in);
//print instructions and scan user input
System.out.println("~ This program computes the square root of an integer ~");
System.out.print("\n" + "Enter a non-negative integer [not greater than 1 billion (10^9)] n: ");
int n = userInput.nextInt();
System.out.print("\n" + "Enter a non-negative integer [not greater than 10 thousand (10^4)] k: ");
int k = userInput.nextInt();
userInput.close();
// declaring and converting variables
int p = (k + 1);
BigDecimal num = new BigDecimal(n);
BigDecimal guess = new BigDecimal(n);
BigDecimal newGuess;
BigDecimal sqrt;
// calculating error using int p
BigDecimal error = BigDecimal.ONE.movePointRight(-p);
// calculating guess using int p
BigDecimal diff = BigDecimal.ONE.scaleByPowerOfTen(p);
// newton's loop 
while (diff.compareTo(error) == 1) {
    newGuess = guess.subtract(
        ((guess.multiply(guess)).subtract(num))
        .divide(guess.add(guess), k, RoundingMode.DOWN));
    diff = newGuess.subtract(guess);
    if (diff.signum() == -1) {
        diff = diff.abs();
    }
    guess = newGuess;
}
// printing sqrt to screen    
sqrt = guess;
System.out.println("loop calculated: " + "\n" + sqrt);

Topaco · Accepted Answer · 2019-11-18T07:52:11.110

In the loop, a sum is calculated iteratively according to the Newton's method, whereby each summand is rounded and contributes to the error. To illustrate this, it is useful to additionally output the initial value for guess and all values for diff. For the example n = 81, k = 7 the output is:

guess: 81
diff: 40.0000000
diff: 19.5121951
diff: 8.8591124
diff: 3.1073634
diff: 0.5070567
diff: 0.0142611
diff: 0.0000112
diff: 0E-7
loop calculated: 
9.0000001

If all diff-values are subtracted from guess, the exact value is 9.0000001, i.e. the individual rounding errors have resulted in a deviation of 0.0000001. To change this behavior, the scale can be increased e.g. by one, k += 1, before the loop, and reset to the actual value with guess = guess.setScale(k - 1, RoundingMode.DOWN); after the loop. Then the output is:

guess: 81
diff: 40.00000000
diff: 19.51219512
diff: 8.85911242
diff: 3.10736339
diff: 0.50705669
diff: 0.01426108
diff: 0.00001129
diff: 0E-8
loop calculated: 
9.0000000

Now the result matches the expectation. The change can be verified using the second example n = 87026 , k = 11, for which the output is without the change:

guess: 87026
diff: 43512.50000000000
diff: 21755.75001149068
diff: 10876.87510915519
diff: 5436.43840462864
diff: 2714.22604199009
diff: 1349.16761702433
diff: 659.01405739182
diff: 300.74946658011
diff: 107.35187252479
diff: 18.35523727653
diff: 0.56993647588
diff: 0.00055055104
diff: 5.1E-10
diff: 0E-11
loop calculated: 
295.00169491039

with an analog error in the result. With the modification the output is:

guess: 87026
diff: 43512.500000000000
diff: 21755.750011490686
diff: 10876.875109155187
diff: 5436.438404628644
diff: 2714.226041990089
diff: 1349.167617024336
diff: 659.014057391816
diff: 300.749466580112
diff: 107.351872524786
diff: 18.355237276533
diff: 0.569936475871
diff: 0.000550551041
diff: 5.13E-10
diff: 0E-12
loop calculated: 
295.00169491038

in accordance with the expected value.

RoundingMode.DOWN for BigDecimal rounding up instead

1 Answers1