Convert OpenSSL key to CryptoAPI

Question

I have spent quite a bit of time on Google searching for the answer to this. I have found several pieces of code describing solutions but most are in C, .NET, or Java. My case is VB6.

I have a pretty straightforward RSA1 application: sign some data (in "appA" with private key), then validate signature (in "appB" with public key, this is the VB6 app). Right now everything is running fine via the CryptoAPI library.

The "appA" signature portion needs to be moved to a unix server and carried out by OpenSSL (preferably). The problem is converting the key format from PEM to the PublicKeyBlob that CryptoAPI expects.

I have tried to port this C code to VB. CryptStringToBinary succeeds but CryptDecodeObjectEx just hangs then crashes VB.

I haven't been able to find any documentation showing this use in VB. I'm not sure that it's possible even. I'm hoping someone may be able to shed some light on this. I have also tried the CryptDecodeObject (sans "Ex") function hoping that the lack of all the structures needed would solve the issue... but same problem.

My test key was generated by OpenSSL using openssl_pkey_new

Answer 1

Thee only things I can think of is check to make sure your declarations are right and to debug/print out the paramaters you are passing and verify they are correct.

Declare Function CryptDecodeObject lib "crypt32" (ByVal dwCertEncodingType As Long, ByVal lpszStructType As String, ByVal pbEncoded As String, ByVal cbEncoded As Long, ByVal dwFlags As Long, pvStructInfo As Any, ByRef pcbStructInfo As Long) As Long`

Declare Function CryptDecodeObjectEx lib "crypt32" (ByVal dwCertEncodingType As Long, ByVal lpszStructType As String, ByVal pbEncoded As String, ByVal cbEncoded As Long, ByVal dwFlags As Long, ByRef pDecodePara As PCRYPT_DECODE_PARA, pvStructInfo As Any, ByRef pcbStructInfo As Long) As Long

There is always a way or a workaround, vb6 is still coughing blood after verification, just write a c++ stub dll that does work and call it from vb6.

Answer 2

Well, I found a problem with one of my structures (didn't declare a byte array member as an array) and I'm no longer having crash issues. I'm still not having any success with CryptDecodeObject however. The code below is what I'm working with. GetLastErr just returns 0 (not much help). If anyone has a thought on where I may be going wrong, please let me know!

Dim iFile As Integer
Dim sPEM As String, sDER As String
Dim lenPEM As Long, lenDER As Long
Dim publicKeyInfo As CERT_PUBLIC_KEY_INFO
Dim publicKeyInfoLen As Long


iFile = FreeFile
Open app.Path & "\publickey.txt" For Binary As iFile
sPEM = Space(LOF(iFile))
Get #iFile, , sPEM
Close iFile

lenPEM = Len(sPEM)

' Determine buffer length required for the DER string
CryptStringToBinary sPEM, lenPEM, CRYPT_STRING_BASE64HEADER, 0, lenDER, 0, 0
sDER = Space(lenDER)

' Do conversion to binary
If Not CryptStringToBinary(sPEM, lenPEM, CRYPT_STRING_BASE64HEADER, sDER, lenDER, 0, 0) Then
    Debug.Print sDER
Else
    MsgBox "CryptStringToBinary Error " & GetLastError
    Exit Sub
End If

' Do conversion to blob
If Not CryptDecodeObject(X509_ASN_ENCODING, X509_PUBLIC_KEY_INFO, sDER, lenDER, 0, publicKeyInfo, publicKeyInfoLen) Then
    MsgBox "CryptDecodeObject Error: " & GetLastError
    Exit Sub
End If

I can post all the function and type declarations if anyone thinks it will help, I believe they are correct.

Here is the public key as generated by OpenSSL:

-----BEGIN PUBLIC KEY----- MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBANWhFRxt/ZF56uGO7GsbvevmX42//thm JdseUwQNot/ihXCPRadf0SPYbtHS6/JA92pCX7NxfgYNoYlOFb0IYYcCAwEAAQ== -----END PUBLIC KEY-----