2

I have written a method to take an integer and convert it to either decimal, hex, or binary depending on what you choose using commenting. I will build a choice in later. The problem I am having is that the wrong values are being chosen and I think it has to do with the modulus operator returning the wrong value. It may be a rounding issue because I always seem to be off by 1 digit, but not for every char in the string. Any help would be appreciated. For example, right now it is set up give the decimal value. So if you run this and enter 1705, it should return 1705, but instead it returns 2705. Hex and Binary are not much better.

Do While val >= 1    'finds the remainder and assigns it array

    r = val Mod base 
    buf(i) = arr(r)

    'Possible rounding issue:
    val = val / base     'need whole number division to find val. 

    i = i - 1 'decrement the buf array    

Loop
Smandoli
  • 6,919
  • 3
  • 49
  • 83
BDMurphy
  • 23
  • 5
  • I have an opinion that you should work on this some more. You seem to pretty much know where the problem is, so post a smaller block of code that lets people work with the problem with less eye strain. And, welcome to Stack Overflow. – Smandoli Apr 28 '14 at 01:46
  • 1
    Thank you. I will definitely do that in the future. – BDMurphy Apr 28 '14 at 03:02

1 Answers1

1

Mod() operates on integer values. If the value you're passing isn't an integer value, VBScript will convert it to one by rounding it to the nearest integer. That's what's happening with your value of 1705. After three iterations, you're left with 1.705, which Mod() rounds to 2.

So now the question becomes, why are you passing floating-point values to Mod() in the first place? The reason is because VBScript is doing floating point division in the following statement:

val = val / base

Since you don't care about the remainder (you've already captured it), this statement needs to be performing integer division, not floating-point division. Fortunately, VBScript makes integer division easy. Just use a backslash instead of a forward slash:

val = val \ base
Bond
  • 16,071
  • 6
  • 30
  • 53
  • 1
    Thank you! That worked perfectly. Also thank you for the thorough explanation. I thought that was what was happening, but when I tried to debug using VBSEdit, the value it was showing me was the rounded value, not the floating point value so I could not pinpoint it. I appreciate the help! – BDMurphy Apr 28 '14 at 03:01