VB6 floating point computation issue

Question

This is floating point computation issue (i think).

This is 2 round function i found somewhere.

They look very similar in code but when i run it, i get 2 different result.

In this example when run this code with value is 87.13225 and precision 4 :

a = 87.1322

b = 87.1323

Anyone can explain what happen?

Private Sub Form_Load()
    Dim a#
    Dim b#
    a = Round1(87.13225, 4) '87.1322
    b = Round2(87.13225, 4) '87.1323
End Sub

Private Function Round1(ByVal value#, ByVal vPrecision%)
    Round1 = Fix((value + Sgn(value) / 10 ^ vPrecision / 2) * _
                10 ^ vPrecision) _
            / 10 ^ vPrecision
End Function

Private Function Round2(ByVal value#, ByVal vPrecision%)
    Dim a#
    a = (value + Sgn(value) / 10 ^ vPrecision / 2) * 10 ^ vPrecision
    Round2 = Fix(a) / 10 ^ vPrecision
End Function

this may help: [Intermediate Floating-Point Precision](https://randomascii.wordpress.com/2012/03/21/intermediate-floating-point-precision/) — deblocker, May 27 '17 at 08:27
Floating point operations are not perfectly precise. I could write a detailed answer explaining exactly why this is happening in your code, but it's very likely to be complicated and more than you really wanted to know. More importantly, it won't help you to solve your problem. Speaking of which, what *is* the problem that you're trying to solve? There is never a guarantee that you'll get the same result when you do a floating-point operation different ways. See also: https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html — Cody Gray - on strike, May 27 '17 at 11:59

score 0 · Answer 1 · answered May 30 '17 at 14:10

Part of your problem, I believe, involves type conversion. Since the literal parameters you are passing are not defined as double, I believe they are being passed as singles, which then get converted to doubles, and this likely affects your outcomes. Try passing the literal to each method as 87.12335#. Also, you're not defining either function's output type, so I believe it's returning variants. I believe both methods should end with " As Double"

Finally, why are you attempting to re-invent the wheel? VB6 is capable of two difference kinds of rounding. Banker's Rounding is the default rounding scheme for VB6. This surprised me, and I write accounting software. Banker's rounding always rounds to the nearest EVEN number, so both of your results should be 87.1234. In banker's rounding, 87.12345 would ALSO round to 87.1234. If you want the rounding where it always rounds away from zero, which is what I was more familiar with, you should use the various format methods, i.e. FormatNumber, Format$, etc. There is a link on the page referenced above that explains this further. When I discovered this discrepancy, I made my own rounding routine with uses either Banker's rounding or what I call Standard rounding, based upon what my customer prefers.

The speculation in the first paragraph is not correct. The `#` type suffix means Double, so the parameters *do* effectively end with `As Double`. And that means that the parameters are not being passed as Singles, but rather as Doubles, since that's their explicit type. Now, it is true that classic VB stores literals in the lowest-precision type in which they will fit, so these literals may actually be Singles, but type promotion happens when they get passed to a function that takes its parameters as Doubles, and there's no loss of precision there. (It's a widening conversion.) — Cody Gray - on strike, May 31 '17 at 07:08
Thank you for your answer. I'd to use banker's rounding too. But this is our customer code for 20 years so i want maintain its logic, not re-invent the wheel. And for my question, @deblocker's answer is correct. — Thinh Vu, Jun 01 '17 at 17:50

VB6 floating point computation issue

1 Answers1