cryptoapi and openssl

Question

I'm trying to encrypt and sign a file with cryptoapi with some X.509 certificates. I want to verify and decrypt this file with openssl.

On windows I think I need to use the CryptSignAndEncryptMessage function to encrypt and sign data. I used this example from MSDN to create a signed and encrypted message.

How can I decrypt/verify this file using openssl? I removed the first 4 bytes from the message since it contained the length of the message (from the windows blob). When I call openssl -asn1parse I get some output that indicates it to be parsable by openssl.

When trying to verify the signature with openssl I recieve an error:

openssl rsautl -verify -inkey AlonsoCert.pem -keyform pem -certin -in sandvout-without-4byte.txt
RSA operation error
3073579208:error:0406706C:rsa routines:RSA_EAY_PUBLIC_DECRYPT:data greater than mod len:rsa_eay.c:680:

Why not encrypt and sign using OpenSSL. Then you avoid all the 2 tool set issues. — brian beuning, Dec 02 '12 at 14:51
@brianbeuning I want to use non exportable keys from the windows keystore for en/decryption. — David Feurle, Dec 03 '12 at 08:33

Duncan Jones · Answer 1 · 2012-12-03T16:15:25.413

1

Your ASN.1 dump information shows you've created a PKCS#7 CMS output from your CryptoAPI code. As a result you cannot use the basic OpenSSL decryption and verification methods.

Instead, use the cms mode:

openssl cms -decrypt -inform DER -in sandvout-without-4byte.txt 
    -out decrypted.bin -recip testkey.pfx

(Note: I've not used this mode before, so I think the syntax I've suggested is correct. Either way, this should hopefully be the step in the right direction that solves this.)

edited Dec 03 '12 at 16:15

answered Dec 03 '12 at 11:04

Duncan Jones

67,400
29
193
254

I tried: openssl rsautl -in sandvout.txt -inkey testkey.pfx -keyform pkcs12 -decrypt -out decrypted.bin which only gives me: 3073743048:error:0406506C:rsa routines:RSA_EAY_PRIVATE_DECRYPT:data greater than mod len:rsa_eay.c:523: – David Feurle Dec 03 '12 at 15:26
@DavidFeurle I just noticed the contents of your ASN.1 output and I've radically altered my answer. – Duncan Jones Dec 03 '12 at 16:16
I tried to decrypt the pkcs7 cms message as you suggested without success. Details: http://pastebin.com/x038d3Rx – David Feurle Dec 04 '12 at 10:45

score 1 · Accepted Answer · answered Dec 05 '12 at 19:16

1

CryptSignAndEncrypt message seems to use RC4 cipher with empty ASN.1 parameters field and, looking at OpenSSL sources, openssl chokes on try to generate IV (which is not needed for RC4).

Try to use other cipher (AES for example) in CryptAndSignMessage. Anyway, RC4 is very old, insecure, and obsolete.

answered Dec 05 '12 at 19:16

Nickolay Olshevsky

13,706
1
34
48

After changing the encryption algorithm from szOID_RSA_RC4 to szOID_NIST_AES256_CBC the decrypt commando works. The decrypted message already contains the plain text message. Now I try to verify the signature of the message. This does not yet work http://pastebin.com/V3YM5aFX. The two things still missing is verifiying the signature and getting the raw plain text message. – David Feurle Dec 05 '12 at 21:18
So now output is PKCS#7 signed message, you should pick up the correct openssl command line to verify this signature and output tha raw data. – Nickolay Olshevsky Dec 05 '12 at 21:29
I'm trying to find the correct way to verify the message. Do you have any clue on how to do it? I test with rsautil and with cms commandos. Cms says http://pastebin.com/r3WAQYcs – David Feurle Dec 05 '12 at 21:40
try something like: openssl cms -verify -in inputdata -inform DER – Nickolay Olshevsky Dec 05 '12 at 21:44
Sorry, didn't read previous comment before posting. Looks like openssl cannot find and verify certificate's issuer – Nickolay Olshevsky Dec 05 '12 at 21:47
after changing the hashing algorithm to md5 in windows the message can be decrypted and verified by openssl. The last thing I'm missing is: How do I extract the original plain from the pkcs7 message. Meanwhile thanks for your help. -> resolved – David Feurle Dec 06 '12 at 18:21

score 1 · Answer 3 · answered Dec 05 '12 at 19:31

1

Try using openssl smime to verify and/or decrypt. The syntax is fairly straight-forward but you can find the information here: http://www.openssl.org/docs/apps/smime.html

answered Dec 05 '12 at 19:31

Nik Bougalis

10,495
1
21
37

cryptoapi and openssl

3 Answers3