PKCS#11 Smartcard getting error: javax.net.ssl.SSLHandshakeException due to java.security.InvalidKeyException: The RSA asymmetric cipher

Question

Situation

I am working on a project where we must get Smartcard to work in Linux. The card is by manufacturer Izenpe. I get an javax.net.ssl.SSLHandshakeException every time when I am doing SSL handshake with the server.

I load the Izenpe driver with these code:

private final static String config_path3 = "name=IZENPE\nlibrary=/usr/lib/libbit4ipki.so";

...

String config = config_path3;
Provider provider = new SunPKCS11(new ByteArrayInputStream(config.getBytes()));
Security.removeProvider("IAIK");
Security.insertProviderAt(provider, 1);
try {
    keystore = KeyStore.getInstance("PKCS11", provider);
} catch (KeyStoreException e) {
    throw e;
}
KeyStore.Builder builder = KeyStore.Builder.newInstance("PKCS11", provider, 
        new KeyStore.CallbackHandlerProtection(new UtilTarjetas(). new callback()));
keystore = builder.getKeyStore();

It loads the certificates to show in a table, and I can proceed to the part to obtain the OutputStream from the response of server, where it fails when we do SSL handshake (signing the challenge). Full stacktraces below:

javax.net.ssl.SSLHandshakeException: Error signing certificate verify
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1904)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:279)
    at sun.security.ssl.ClientHandshaker.serverHelloDone(ClientHandshaker.java:1054)
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:341)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:901)
    at sun.security.ssl.Handshaker.process_record(Handshaker.java:837)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1023)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1332)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1359)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1343)
    at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:563)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1092)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
    at gestores.comunicacion.gisscide.HTTPS_TGSS.doHTTP_Get(HTTPS_TGSS.java:470)
    at gestores.comunicacion.Gestor_Comunicaciones$RecibeMensajesSSL.descargaMsgHTTPS(Gestor_Comunicaciones.java:1284)
    at gestores.comunicacion.Gestor_Comunicaciones$RecibeMensajesSSL.run(Gestor_Comunicaciones.java:1852)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.security.InvalidKeyException: The RSA asymmetric cipher only operates with RSA keys; unsupported key found (sun.security.pkcs11.P11Key$P11PrivateKey)
    at com.entrust.toolkit.security.provider.RSA.a(Unknown Source)
    at com.entrust.toolkit.security.provider.RSA.engineGetKeySize(Unknown Source)
    at javax.crypto.Cipher.passCryptoPermCheck(Cipher.java:1052)
    at javax.crypto.Cipher.checkCryptoPerm(Cipher.java:1010)
    at javax.crypto.Cipher.init(Cipher.java:1209)
    at java.security.Signature$CipherAdapter.engineInitSign(Signature.java:1254)
    at java.security.Signature$Delegate.init(Signature.java:1128)
    at java.security.Signature$Delegate.chooseProvider(Signature.java:1085)
    at java.security.Signature$Delegate.engineInitSign(Signature.java:1158)
    at java.security.Signature.initSign(Signature.java:529)
    at sun.security.ssl.RSASignature.engineInitSign(RSASignature.java:125)
    at java.security.Signature$Delegate.engineInitSign(Signature.java:1156)
    at java.security.Signature.initSign(Signature.java:529)
    at sun.security.ssl.HandshakeMessage$CertificateVerify.<init>(HandshakeMessage.java:1556)
    at sun.security.ssl.ClientHandshaker.serverHelloDone(ClientHandshaker.java:1049)
    ... 15 more

It seems that when getting the unextractable private key in the card for signing the challenge in SSL handshake, it fail because PKCS#11 private key is not supported/recognized as valid.

By adding this line into my code I get many lines of logs about SSL.

Properties systemProps = System.getProperties();
systemProps.put("javax.net.debug", "all");          
System.setProperties(systemProps);

At the end I can see this:

SESSION KEYGEN:
PreMaster Secret:
0000: A5 F7 9C 89 A4 B5 B4 66   D4 DC CC 40 45 C8 41 07  .......f...@E.A.
0010: 0E F1 E9 5C 99 36 C8 84   06 B0 6B 95              ...\.6....k.
CONNECTION KEYGEN:
Client Nonce:
0000: 58 07 74 02 CD 8B 65 BB   E0 03 8D 53 95 C4 87 8C  X.t...e....S....
0010: EE 95 B2 92 C1 DF E8 CA   B0 7D E4 AD 16 B7 31 D2  ..............1.
Server Nonce:
0000: 4A 41 AA 83 DD 1D 9C DE   84 4A 56 40 A3 32 F7 53  JA.......JV@.2.S
0010: 18 48 32 BD 7A E2 3A 1D   19 AD 67 6E DD E1 3B 20  .H2.z.:...gn..; 
Master Secret:
0000: 72 58 C0 3D 90 67 2E 3B   B2 AE D4 54 15 AB 18 AA  rX.=.g.;...T....
0010: 95 73 91 9A DA AE EF 3D   77 A1 CD 7A 68 8B 37 56  .s.....=w..zh.7V
0020: 0F 05 64 EB DD 93 AF 6C   C4 C2 8A 75 A7 C2 CA 06  ..d....l...u....
Client MAC write Secret:
0000: 30 2C D3 A0 4C 2D 3F 67   ED B9 64 B8 3B 81 47 0E  0,..L-?g..d.;.G.
0010: D1 7B 75 A9                                        ..u.
Server MAC write Secret:
0000: 4B 22 25 8E 81 D1 55 6D   B9 40 0F 2A A2 26 49 F5  K"%...Um.@.*.&I.
0010: 66 6A 91 AE                                        fj..
Client write key:
0000: 72 BD C0 56 3C 1A E5 61   90 2C A6 AE AA FE B9 71  r..V<..a.,.....q
Server write key:
0000: C6 CE F4 C6 CE A4 E3 55   F6 2D 29 D6 2E 4C CA 7A  .......U.-)..L.z
Client write IV:
0000: 7D DB C8 17 F4 18 72 33   A1 DC 03 D6 2F 87 65 F1  ......r3..../.e.
Server write IV:
0000: 58 B1 4E BE 1A 90 0C B3   0D AE 5C 5B CD 36 74 4D  X.N.......\[.6tM
%% Invalidated:  [Session-3, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA]
Thread-7, SEND TLSv1 ALERT:  fatal, description = handshake_failure
Thread-7, WRITE: TLSv1 Alert, length = 2
[Raw write]: length = 7
0000: 15 03 01 00 02 02 28                               ......(
Thread-7, called closeSocket()
Thread-7, handling exception: javax.net.ssl.SSLHandshakeException: Error signing certificate verify
Finalizer, called close()
Finalizer, called closeInternal(true)

I don't know if it's relevant, but when testing my server site in www.ssllabs.com, I found TLSv1.2 is supported but TLSv1.1 is not. And the algorithm which is picked here in the log is not listed in the preferred list of the server:

So, what can be the reason?

I have changed the $JAVA_HOME/jre/lib/security/java.security, disabling the TLSv1.2 and SSLv3 in the option jdk.tls.disabledAlgorithms=SSLv3, TLSv1.1, leaving only TLSv1.1 because it's the server's choice, to no avail.
The driver I used to load the Provider is from Izenpe website. From what I got in the Console of Eclipse, it's working.

My thought about it:

What fails maybe the Provider SunPKCS11. I say this because the original code works in Windows when we use SunMSCAPI. Some internal mechanism of SunMSCAPI did the work for us to sign the challenge. But now as we must adapt the project to be multi-platformed, in Linux we have no choice but SunPKCS11. The JAR sunpkcs11.jar of Java 6 and Java 8 both failed. (In Java 7 it's not present.)
The original intention is to make (almost) all smart cards available for authentication in my APP, but at the moment we only want Izenpe and DNIe(Spanish electronic ID card, supported by OpenSC) to work.

score 1 · Accepted Answer · answered Nov 04 '16 at 15:29

1

Looks like for the SSL connection by the Java JCE, Entrust Provider is being used.

If you look in the stacktrace:

Caused by: java.security.InvalidKeyException: The RSA asymmetric cipher only operates with RSA keys; unsupported key found (sun.security.pkcs11.P11Key$P11PrivateKey) at com.entrust.toolkit.security.provider.RSA.a(Unknown Source) at com.entrust.toolkit.security.provider.RSA.engineGetKeySize(Unknown Source) at javax.crypto.Cipher.passCryptoPermCheck(Cipher.java:1052)

This error is being thrown by the Entrust JCE Provider. It may be expecting only Java's RSA keys but it is being sent a PKCS11 Key which it may not be able to handle.

So, try removing the 'Entrust' Provider as well.

If that doesn't work, try to pass the SunPKCS11 provider instance to the SSLContext whereever you are using it to initiate your secured connection.

answered Nov 04 '16 at 15:29

always_a_rookie

4,515
1
25
46

Thanks. I just observed that before establishing the connection there is a method to reorder the providers, putting IAIK and Entrust to the last of the list of providers. That's enough? Another thing is, how to pass the SunPKCS11 provider to the SSLContext? – WesternGun Nov 07 '16 at 08:45
If your are able to connect to the `https` url (i.e. the certificate is trusted), don't worry about the SSLContext. Looks like you in your case the certificate is trusted. If you still get the same error after reordering the providers, try removing the provider. You can add them back after you are done with the connection. – always_a_rookie Nov 07 '16 at 17:33
Thanks, I have tried and that basically is the reason, although I have modified many codes and using NSS and I am not so sure, but the same exception is not appearing again. Just doubt that where is this Entrust provider, indeed? – WesternGun Nov 09 '16 at 10:51
It should be from the Entrust's JCE jar. You might be adding the provider using `Security.addProvider(...)` or may be their own api. – always_a_rookie Nov 09 '16 at 12:13

score 0 · Answer 2 · answered Oct 25 '16 at 04:49

while I'm not too versed in PKCS#11 (nor in SSL when I think about it) from your description and log files the following items caught my attention and have me thinking:

Considering the InvalidKeyException thrown above and the highlighted algorithm (TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA), I'm inclined to believe that the key is an elliptic curve key, not RSA (in the screenshot, it states that security provided by TLS_ECDHE equals the one you get by using RSA with 2048 bits). Consequentially, somewhere along the way the wrong Signature Provider (RSA instead of ECDSA) is selected, either programmatically or by a malformed certificate.

Hope that helps.

I checked the card and it shows that the public key is RSA. I think the private key is also RSA? Also, when doing SSL handshake in Windows 10, same card and with Microsoft Windows provider(`Windows-MY`), it works good with `TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA`. So I guess it's not the reason(to be confirmed). — WesternGun, Oct 26 '16 at 11:25

PKCS#11 Smartcard getting error: javax.net.ssl.SSLHandshakeException due to java.security.InvalidKeyException: The RSA asymmetric cipher

Situation

My thought about it:

2 Answers2