2

I'm having a problem with password decryption that has worked correctly for some 10 years in a VB6 DLL but now on newer, faster servers, it's returning the last few encrypted characters of the stored password. For example, "1234" is stored as "Žl¹è=" but when that is decrypted it's coming back with "1234 ¹è=". This code was written well before I started working with the program and I know little about encryption/decryption.

What appears to be the relevant code is shown below:

'Prepare sCryptBuffer for CryptDecrypt
lCryptBufLen = Len(sInputBuffer)
sCryptBuffer = String(lCryptBufLen, vbNullChar)
LSet sCryptBuffer = sInputBuffer

'Decrypt data: lHkey=Encryption Key, 0 = No Hash, 1=True (for final?), 0=dwFlags?, sCryptBuffer=data to be decrypted, lCryptBufLen=length of the pbData buffer
If Not CBool(CryptDecrypt(lHkey, 0, 1, 0, sCryptBuffer, lCryptBufLen)) Then
    GoTo Finished
End If

'Setup output buffer with just decrypted data
strDecryptedText = Mid$(sCryptBuffer, 1, GetResultsBufferLength(sInputBuffer))
strDecryptedText = Replace(strDecryptedText, vbNullChar, "")

'If the last characters ASCII value is 8 or less, this number represents the amount
'of padding that is on the decrypted text.
If Asc(Right(strDecryptedText, 1)) <= 8 Then
   DecryptEx = Left(strDecryptedText, Len(strDecryptedText) - Asc(Right(strDecryptedText, 1)))
Else
    DecryptEx = strDecryptedText
End If

The GetResultsBufferLength function appears to use the following line of code for 3DES:

lngReturn = lngTempLength - ((lngTempLength + 8) Mod 8) + 8

Any ideas will be much appreciated.

John Hunt
  • 21
  • 1
  • 1
    Isn't the `pdwDataLen` parameter of `CryptDecrypt` an out parameter as well? I'm not familiar enough with the API, but would that have the length you're looking for? https://learn.microsoft.com/en-us/windows/win32/api/wincrypt/nf-wincrypt-cryptdecrypt – User51 Feb 11 '22 at 17:20
  • Has the upgrade changed from ASCII to UTF-8 as its default character encoding? ASCII is strictly one character = one byte. That is no longer true for UTF-8, so previously assumed lengths may be incorrect. – rossum Feb 12 '22 at 12:17
  • User51, I think you're right. Are you suggesting I just truncate the decrypted string after the length found in pdwDataLen? – John Hunt Feb 13 '22 at 02:47
  • rossum, The encrypted value is stored in a MSSQL database field. Could it be that a default has changed? – John Hunt Feb 13 '22 at 07:23
  • 1
    Most probably `GetResultsBufferLength` is buggy. For `strDecryptedText` it seems you just need `strDecryptedText = Left$(sCryptBuffer, lCryptBufLen)` with no helper function used. – wqw Feb 13 '22 at 12:02
  • wqw, That worked! User51, You seemed to have the right idea, as well. What's funny is that this had occurred to me but I dismissed it too quickly, asking myself, "If it was that simple, why wouldn't the original programmer have done that in the first place instead of using a mysterious, and buggy calculation?" I guess the answer to that question will remain a mystery. Many thanks to you both. – John Hunt Feb 14 '22 at 19:39

0 Answers0