keytool versus FIPS when handling PKCS12 keystores

Question

Background: What Works

From time to time we have to use a piece of Java software that reads PKCS#12 keystores. For this particular project, we have to create public/private pairs on an as-needed basis, and we store the keys in PKCS12 files because it's stable and pretty much everything can read that format.

Because we do a lot of Java in-house, we have keytool sitting around, and so we figured hey, just use keytool to create the private key and certificate. A typical instance looks like this:

keytool -keystore MyLuggage.p12 -storepass 123456 -storetype pkcs12
   -alias "......"
   -genkeypair -keyalg RSA -keysize typically_2048_or_3072 -sigalg SHA256withRSA

   -ext "KeyUsage=dataEncipherment,digitalSignature,keyEncipherment"
   -startdate ....
   -dname "......."

The actual -keysize varies between 2048 and 8192 in practice; for the purposes of this question it hasn't seemed to make a difference what gets used, but obviously we use key lengths appropriate to the task if we get to choose them at all (usually dictated by the constraints of other software, or dictated by some regulation that's handed to us).

This has always worked, in that other software -- including the Java software mentioned at the start -- can read the keystore and make use of the private keys within. (And the public key can be exported and used, etc.)

Here's What Breaks

The software recently got upgraded to a version that uses the FIPS 140 certified Java libraries from RSA. ("BSAFE" or "JSAFE" depending on who you ask.) And now, trying to open previously-created PKCS#12 files fails with

java.lang.SecurityException: Algorithm not allowable in FIPS140 mode: PBE/PKCS12/SHA1/RC2/CBC/40
    at ......
    at java.security.KeyStore.load(Unknown Source)

The elided ...... frames are in the RSA source that we don't have and looks to use function names that have been obfuscated in any case. So looking at their source to try and figure out what exactly it's testing to cause this, isn't an option.

So, what is causing this? The only algorithms we've chosen are the "RSA" key generation and "SHA256withRSA" signature, both of which are permitted by FIPS 140-2. I've been looking again through keytool -genkeypair -help output and there don't seem to be any other algorithm or security options. (We've avoided using -keypass because PKCS#12 tools really hate it when the keystore password and the key password are different, and keytool -genkeypair defaults the key password to the keystore password.) The rest of the error message is confusing, as we don't specify the use of SHA-1 or RC2 (!) anywhere.

Googling around points to people having trouble with creating SSL certificates, which we're not doing here, and the solutions given seem to be specific to Tomcat.

Is this a problem with how we're creating the keystore, how we're creating the key pair in the keystore, or some "feature" of FIPS 140 that we haven't previously encountered?

Does creating PKCS12 files using some other application (maybe OpenSSL or the Windows Crypto libraries?) work with this software that you need to interoperate with? Does the software provider have a recommended or documented tool they use for creating the PKCS12 files that work with it? — , May 17 '18 at 17:35
If BSAFE-Java is in the form of a provider, which I expect, you can have `keytool` use it by specifying `-providername` or `-providerclass` to create the keystore, and it should do so using algorithms it will be able to read back. — dave_thompson_085, May 18 '18 at 17:51
@dave_thompson_085 The BSAFE provider classes aren't available on-disk, which is what the `-provider*` options require. We're going to have to end up writing the PKCS#12 file ourselves, and restrict the use of SunJSSE and BC to only handling the in-memory objects and not the disk storage. Unless the BC API allows control of the certbag algorithms... that'll be tomorrow's investigation! — Ti Strga, Jun 07 '18 at 21:47
Do you mean BSAFE (RSA/EMC/Dell) or BC (BouncyCastle)? Those are totally different. If you can use _BC_ provider it has keystore type `PKCS12-3DES-3DES` or `PKCS12-DEF-3DES-3DES` which should be okay for FIPS (though I can't test that). If you want to use _BC_'s own API aka LWAPI, see https://superuser.com/questions/1102971/how-to-set-attributes-to-private-key-on-pkcs12-key-usage for a starting point. — dave_thompson_085, Jun 08 '18 at 01:44
Right. The BSAFE provider isn't on disk, so we can't specify its use to `keytool`. I'm using parts of the BC provider instead, but as you pointed out below, it still defaults to the 40-bit-RC2 certbag. My plan tomorrow (well, today, after some coffee) is to investigate also using the BC provider to handle the PKCS#12 files, and specifying the less-dumb 3DES algorithms. (edit: yes, what your link says, that part!) — Ti Strga, Jun 08 '18 at 13:56

score 5 · Accepted Answer · edited Oct 07 '21 at 08:57

PKCS#12 stores the private key encrypted with a password derived key. It looks like keytool uses pbeWithSHAAnd128BitRC2-CBC (pkcs-12PbeIds 5), an PBES1 algorithm for doing so. Even the keytool.exe of Oracle Java 9 does use this algorithm as you can verify by uploading a .p12 file to the online ASN.1 decoder decoding a sample PKCS#12 file.

If I read the PKCS#12 standard correctly PBES1 was long ago superseeded by the "newer" version of the key derivation system named "PBES2" (mostly PBKDF2 based) with should be used instead. But keytool does not make use of it. This is my interpretation of the error message.

Therefore the certificate and the key may be acceptable, but the PKCS#12 container is not acceptable. You may try to extract key and certificate and save them in a new PKCS#12 file using a current software like OpenSSL (or you simply generate the whole PKCS#12 file directly using OpenSSL).

OpenSSL has the option to specify the PBE used for key and certificate encryption (parameters -keypbe and -certpbe in PKCS#12 mode). I have not checked it but and algorithm like AES-256-CBC should be FIPS140 compatible.

Actually, the Java implementation of PKCS12 in SunJSSE (and also in BC, and Windows and NSS, and OpenSSL by default), as your example shows, encrypts the keybag with 1.2.840.113549.1.12.1.3 pbeWithSHAAnd3-KeyTripleDES-CBC and the **certbag** with 1.2.840.113549.1.12.1.6 pbeWithSHAAnd40BitRC2-CBC; both defined in PKCS12 appendix C -- even though as you say PKCS12 has been updated to prefer PKCS5v2 PBES2. 3DES is FIPS-Approved (though AES is now _preferred_) but RC2 is not -- and the quality of the PKCS12 PBE doesn't matter because RC2-40 is deliberately weak. ... — dave_thompson_085, May 18 '18 at 17:45
... I've never understood why everyone encrypts the certbag very weakly when it doesn't need to be encrypted at all, but they do. OpenSSL can leave it _unencrypted_ with `-certpbe none` but no other software I've seen has this option. — dave_thompson_085, May 18 '18 at 17:54

keytool versus FIPS when handling PKCS12 keystores

1 Answers1