VBScript division decimal by integer returns extra remainder

Question

I'm trying to validate my VBScript function that divides an input value by 1440. When I pass 43.56 as input parameter, it returns me the following result:

0.030250000000000002

while the correct answer is

0.03025

What should I do to make it work correctly both with evenly divided integers, like 1440 / 1440 = 1, and when one value is a decimal. Currently my function looks like so:

Function convertMinutesToDays(value)
    If IsNumeric(value) And TypeName(value) <> "Boolean" Then
        convertMinutesToDays = value / 1440
    Else
        convertMinutesToDays = 0
    End If
End Function

Actually, if you simply put Response.Write convertMinutesToDays(43.56), it will show 0.03025, but we are using it within an assert javascript method, like so:

Call AssertAreEqual(convertMinutesToDays(43.56), 0.03025, "convertMinutesToDays: pass 43.56 should return 0.03025")

The javascript code:

<script language="JavaScript" runat="server">
    function AssertAreEqual(val1, val2, message)
    {
        var retVal = false;
        var divAttributes = "";
        var equality = "";
        if (val1 === val2)
        {
            divAttributes = "class=\"unittest assertareequal pass\"";
            equality = "=";
            retVal = true;
        }
        else
        {
            divAttributes = "class=\"unittest assertareequal fail\"";
            equality = "!=";
            retVal = false;
        }
        Response.Write("<div " + divAttributes + ">Actual:" + val1 + " " + equality  + " " + val2 + ":Expected | " + message + "</div>");
        return retVal;
    }
</script>

The output:

Actual:0.030250000000000002 != 0.03025:Expected | convertMinutesToDays: pass 43.56 should return 0.03025

Any ideas?

What code do you use to display the result? I have tried `MsgBox convertMinutesToDays(43.56)` and this returns `0.03025`. — FrankM, Jul 03 '18 at 13:47

score 0 · Answer 1 · answered Jul 03 '18 at 14:18

0

This is due to the way floating point math works in VBScript(well most any language really). For your code, the Round function should suffice because you are always dividing by a single number so rounding errors won't add up on you the way they can in other types of functions. I chose 5 decimal places because it fixes your example. If you find others that still are off then you may need to up that but for representing partial days 5 is probably enough.

Function convertMinutesToDays(value)
    If IsNumeric(value) And TypeName(value) <> "Boolean" Then
        convertMinutesToDays = Round(value / 1440, 5)
    Else
        convertMinutesToDays = 0
    End If
End Function

answered Jul 03 '18 at 14:18

Thomas

64
6

Actually, that might fix my issue, because Round() function does not add extra zeroes after the decimal if they are less than specified number. I didn't know that. Thank you. – BohdanZPM Jul 03 '18 at 14:27
As long as you are always comparing to something with a fixed number of decimals it should work. – Thomas Jul 03 '18 at 14:36
My last comment should have said "with the same fixed number of decimals" – Thomas Jul 03 '18 at 14:43

VBScript division decimal by integer returns extra remainder

1 Answers1