SMIME can't be validated by receiver when using newer version of Bouncy Castle

Question

I am using BC to encrypt and sign an SMIME message for use with AS2. The code we have works fine with an absolutely ancient version of bouncy castle, bcmail-1.4:125. Upgrading to anything newer causes the receiver of the message (not too ancient Cyclone server) to fail to verify the message. (e.g. the earliest v in maven causes this too. These are the versions without API changes (e.g. 1.38).

Since we use JDK 1.7 (and 1.8), I've been trying to update this to a newer version of BC, java-mail, etc. I've upgraded all of bouncy castle to bcmail-jdk15on:1.51 and bcprov-jdk15on:1.51, along with java mail, and followed the examples in the bcmail package. However, I am still getting an error from Cyclone saying integrity-check-failed.

I am fairly certain the error is with how I am doing signing. When I disable signing and only use encryption it processes correctly. Also, I can correctly receive a signed response from the remote server and verify the signature, which is how I get the error message out (from content-disposition on the MimeMultiPart).

• The certificates are created by openssl/self signed/etc, stored in pkcs12 file
• Unlimited strength policies are in place
• senderKey is a BCRSAPrivateCrtKey
• senderCert
  • org.bouncycastle.jcajce.provider.asymmetric.x509.X509CertificateObject

Failing: The current code is this, using bcmail-jdk15on:1.51 & etc

SMIMESignedGenerator gen = new SMIMESignedGenerator();
gen.addSignerInfoGenerator(new JcaSimpleSignerInfoGeneratorBuilder()
           .setProvider("BC")
           .build("SHA1withRSA", senderKey, senderCert));
// gen.addCertificates(new JcaCertStore(list(senderCert))); old v. doesn't add certs
MimeMultipart smime = gen.generate(part); // MimeBodyPart passed in to function
MimeBodyPart tmpBody = new MimeBodyPart();
tmpBody.setContent(signedData);
tmpBody.setHeader("Content-Type", signedData.getContentType()

Previously working code looks like this and uses bcmail-1.4:1.25. Upgrading to 1.3x also causes a failure on the other end when decrypting (irrespective of which jdk I run on, 1.6 - 1.8)

MimeBodyPart body = new MimeBodyPart();
body.setDataHandler(new DataHandler(new ByteArrayDataSource(bytes[], contentType, null);));
SMIMESignedGenerator sGen = new SMIMESignedGenerator();
// SHA1 resolves to "1.3.14.3.2.26", FWIW
sGen.addSigner(senderKey, senderCert, getBouncyCastleAlgorithmId("SHA1"));
MimeMultipart signedData = sGen.generate(part, "BC");
// this is then encrypted & streamed, no issues there

Common Setup Code

byte[] data = Files.readAllBytes(filePath);
MimeBodyPart part = new MimeBodyPart();
ByteArrayDataSource dataSource = new ByteArrayDataSource(data, "application/EDIFACT", null);
part.setDataHandler(new DataHandler(dataSource));
part.setHeader("Content-Transfer-Encoding", "8bit");
part.setHeader("Content-Type", "application/EDIFACT");

I have a feeling it has something to do with how I am adding (or manipulating) the senderCert, which is the local application's X509.

Update

I've made the new code more in-line with what the old produces by removing the certificate:

• It no longer includes the cert in the signed message. old version didn't
• The entire mime multi-part content is now exactly the same length (1095 bytes) as before
• The format (headers, etc) is now exactly the same
• The signed part is now nearly identical. There is a portion that seems to vary based on the time (???), though, and that changes each time. I can't get openssl to verify this message yet, no idea why.

Here is sample output, FWIW. The text in [] is the only part that changes.

------=_Part_1_1448572667.1409621469842
Content-Type: application/EDIFACT
Content-Transfer-Encoding: 8bit

this is a test

------=_Part_1_1448572667.1409621469842
Content-Type: application/pkcs7-signature; name=smime.p7s; smime-type=signed-data
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAMYIBpDCCAaAC
AQEwgZ4wgZAxCzAJBgNVBAYTAmNuMREwDwYDVQQIDAhzaGFuZ2hhaTESMBAGA1UEBwwJY2hhbmdu
aW5nMREwDwYDVQQKDAhwb3dlcmUyZTEOMAwGA1UECwwFaXRkZXYxEjAQBgNVBAMMCWFiLWNsaWVu
dDEjMCEGCSqGSIb3DQEJARYUYWItY2xpZW50QG15Q29ycC5jb20CCQClDAGwq37A/jAJBgUrDgMC
GgUAoF0wGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTQwOTAyMDEz
M[TA5]WjAjBgkqhkiG9w0BCQQxFgQUG6KkoqPBvE7Kd9dB0eop/aUTya0wDQYJKoZIhvcNAQEBBQAE
gYB[h9N4maow9aoTQ8QBGgXEYE+xgXSmRPy+ufIsMpuS0Yys/1t3AfXSSI7WKgLMRKYXve8gdb4Gn
dqecHzkBwBq4hebt9YK+E30E6DpZpCwErsgDVaU/ExBA5gauPWneysy+s2bE5Y6pNZ7Qf3kGU5kI
UjlOF/LUNkCsgT5z//]5N6QAAAAAAAA==
------=_Part_1_1448572667.1409621469842--

Answer 1

After a lot of debugging and dumping of files and so on, I proved that it had soemething to do with the digest calculation. In a specific location in the signed body part is the content MIC (base64 of the digest). Somehow this value did not match what the other side did...

Once I had that, and some more time with google, I finally turned up more information on sourceforge confirming this. Helpful since it mentions my specific version of BC. To quote:

The problem is that BC >= 1.27 will "canonicalize" all messages that are not sent with content-transfer-encoding binary.

What does this mean?

In the S/MIME rfc it says that all messages should be converted to a "canonical" form before computing the MIC. The "canonical" form for text messages is that EOL is indicated by CR, LF. The rfc's are silent on what the canonical form for other content types is. Many S/MIME implementations (e.g. openssl, Bouncy Castle after 1.27) incorrectly assume that the canonical form for all messages except those sent with content-transfer-encoding binary is that every LF should be preceeded by a CR.

So if a BC 1.25 used sends a message including bare LF characters then the MIC validation will fail if the message is received by an application using BC >= 1.27, or openssl smime, or many other S/MIME implementations.

OpenAS2 should be fixed to use content-transfer-encoding binary.

This only appears to work if you set the encoding on the MIME Body Part I had verified the same results (that failed on the server) by doing the JCA route manually, the lightweight route, and also the CMS route.

With that information I made a simple change to the sender....

MimeBodyPart part = //.. make mime body part from file
part.setHeader("Content-Transfer-Encoding", "binary");

The funny part about this is that changing anything to do with the SMIMESignedGenerator() seems to have no effect:

gen = SMIMESignedGenerator("binary");  // nothing, even though the docs say to set this

My final signing function looks like this, for anyone interested:

SMIMESignedGenerator gen = new SMIMESignedGenerator();
SignerInfoGenerator sigGen = new JcaSimpleSignerInfoGeneratorBuilder()
        .setProvider(BC)
        .build("SHA1withRSA", senderKey, senderCert);
gen.addSignerInfoGenerator(sigGen);
MimeMultipart smime = gen.generate(part);
MimeBodyPart tmpBody = new MimeBodyPart();
tmpBody.setContent(smime);
tmpBody.setHeader("Content-Type", smime.getContentType());
return tmpBody;

The original file is just one line:

this is a test

The input that gets signed is this:

Content-Type: application/EDIFACT
Content-Transfer-Encoding: binary

this is a test

Debugging info:

data bytes:
436F6E74656E742D547970653A206170706C69636174696F6E2F454449464143540D0
A436F6E74656E742D5472616E736665722D456E636F64696E673A2062696E6172790D
0A0D0A74686973206973206120746573740A

digest mic: {
   "algorithmId":   "1.3.14.3.2.26"
   "digest bytes":  "CEC2C6614A481DFDF45C801FD6F2A51BC53D3FDF"
   "digest base64": "zsLGYUpIHf30XIAf1vKlG8U9P98="
}

Not this does not attach the signature, or add any capabilities, and uses a v1 x509 cert. I may change that stuff now that this is all working again.

I really do wish all this was more transparent... BC internally is indirection upon indirection, though I understand why. It's still better internally than the old version. I can't say that I haven't found a lot of examples, but the BC test cases don't seem the best (e.g. I couldn't find one that verified against an expected digest value after SMIME'ing. maybe I missed it)