0

I have a two mails (SMIME encrypted) for a single recipient. One mail is encrypted using 3DES, the other one is encrypted using AES 256.

The mails where created using C# EnvelopedCms class.

I can successfully decrypt the 3DES message using

openssl smime -decrypt -in trippledes.eml -inkey keyfile.pem

However, if I try this with the AES encrypted file, OpenSSL outputs some gibberish and Fails with this comment:

Error decrypting PKCS#7 structure 4128:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:.\crypto\evp\evp_enc.c:539:

Thunderbird cannot open the mail either. But Outlook 2010 has no problem opening the message.

What is the best way to troubleshoot the issue? Is there any logging I can turn on?

I've already examined the ASN.1 structure of both mails using the ASN.1 Decoder on http://lapo.it/asn1js/. Both messages look OK to me, so I guess the culprit lies in the encrypted Content itself.

Henning Krause
  • 5,302
  • 3
  • 24
  • 37
  • Or the (format of) the symmetric key. Bad final blocks are mostly due to bad padding exceptions, and if you use the wrong key you get gibberish, which is unlikely to contain correct padding. – Maarten Bodewes Dec 17 '12 at 19:43
  • Hmm, but the content can be successfully decrypted with Outlook. So the symmetric key should be ok, don't you think? – Henning Krause Dec 20 '12 at 10:32

1 Answers1

2

I know this is years late, but it might be helpful to others...

I was using the EnvelopedCms very successfully and happily for a few years, exchanging messages with many other implementations. When someone this year decided to require the use of AES, I discovered that at least one Java-based system was failing to work with my messages. (Their error was "Unable to create PKCS #7 MIME content")

I used an ASN info utility to break down what I was sending, and discovered that EnvelopedCms was forcing the KeyEncryptionAlgorithm to RSA-OAEP when the content encryption was set to AES. (If the content was encrypted with anything else, the KeyEncryptionAlgorithm was just plain RSA.)

I could find no documentation or RFCs explaining this behavior, and there does not appear to be any way to change it.

My solution to the problem was to use the BouncyCastle CmsEnvelopedDataGenerator class. So far, it appears to work at least as well as EnvelopedCms, and avoids the RSA-OAEP key encryption issue. Almost a drop-in replacement (other than the certificate class used.)

While I could not find any documentation that specifically said that my recipient's Java libraries could not use the RSA-OAEP algorithm, once I eliminated it, their error was gone, and messages were successfully sent.

Example code using BouncyCastle:

private byte[] CmsEncrypt(byte[] message, string contentEncryptionOid, Org.BouncyCastle.X509.X509Certificate recipCertificate)
{
    var cmsGenerator = new CmsEnvelopedDataGenerator();
    var cmsData = new CmsProcessableByteArray(message);

    cmsGenerator.AddKeyTransRecipient(recipCertificate);

    var cmsEnvelope = cmsGenerator.Generate(cmsData, contentEncryptionOid);

    return cmsEnvelope.GetEncoded();
}
KentGeek
  • 21
  • 4
  • I had a similar experience when interoperating with some of the Android code. It is very unfortunate that you can't force that with EnvelopedCms class which otherwise works great. – zaitsman Dec 24 '15 at 12:45
  • @KentGeek do you have example code for decryption as well? I struggle with decrypting the message with the private key sitting on a smartcard. E.g. one can't access the private key (which various BC methods I found for decryption think would be perfectly normal). – omni Jan 07 '18 at 14:19