Small question regarding an issue faced when mTLS is enabled.
On a Java + Spring Webflux project for both client and server, the client is configured to send requests as such:
public SslContext getSslContext() {
try {
final KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
try (InputStream file = Files.newInputStream(Paths.get(URI.create(keyStorePath)))) {
final KeyStore keyStore = KeyStore.getInstance(keyStoreType);
keyStore.load(file, keyStorePassPhrase.toCharArray());
keyManagerFactory.init(keyStore, keyPassPhrase.toCharArray());
}
final TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
try (InputStream trustStoreFile = Files.newInputStream(Paths.get(URI.create(trustStorePath)))) {
final KeyStore trustStore = KeyStore.getInstance(trustStoreType);
trustStore.load(trustStoreFile, trustStorePassPhrase.toCharArray());
trustManagerFactory.init(trustStore);
}
return SslContextBuilder.forClient().keyManager(keyManagerFactory).trustManager(trustManagerFactory).build();
} catch (CertificateException | NoSuchAlgorithmException | IOException | KeyStoreException | UnrecoverableKeyException e) {
return null;
}
}
//the webflux web client
WebClient.create().mutate().defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE).clientConnector(new ReactorClientHttpConnector(HttpClient.create().wiretap(true).secure(sslContextSpec -> sslContextSpec.sslContext(getSslContext())))).build();
Upon the HTTP call, I am seeing this on client side log:
[or-http-epoll-3] r.netty.http.client.HttpClientConnect : [id: 0x02d7d229, L:/123:44092 ! R:the-service.com/456:19010] The connection observed an error
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:478) ~[netty-codec-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) ~[netty-transport-native-epoll-4.1.58.Final-linux-x86_64.jar!/:4.1.58.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) ~[netty-transport-native-epoll-4.1.58.Final-linux-x86_64.jar!/:4.1.58.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) ~[netty-transport-native-epoll-4.1.58.Final-linux-x86_64.jar!/:4.1.58.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.58.Final.jar!/:4.1.58.Final]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[na:na]
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[na:na]
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:336) ~[na:na]
at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[na:na]
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) ~[na:na]
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:171) ~[na:na]
at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681) ~[na:na]
at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636) ~[na:na]
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454) ~[na:na]
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433) ~[na:na]
at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:637) ~[na:na]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:282) ~[netty-handler-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1387) ~[netty-handler-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1282) ~[netty-handler-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1329) ~[netty-handler-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) ~[netty-codec-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) ~[netty-codec-4.1.58.Final.jar!/:4.1.58.Final]
... 15 common frames omitted
And on server log, I am seeing the following.
s.ApplicationProtocolNegotiationHandler : [id: 0x10b61280, L:/1xx.xxx:19010 - R:/1xxx:45678] Failed to select the application-level protocol:
javax.net.ssl.SSLHandshakeException: Empty client certificate chain
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[na:na]
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[na:na]
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:336) ~[na:na]
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292) ~[na:na]
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:283) ~[na:na]
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1193) ~[na:na]
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1180) ~[na:na]
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[na:na]
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443) ~[na:na]
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074) ~[na:na]
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061) ~[na:na]
at java.base/java.security.AccessController.doPrivileged(Native Method) ~[na:na]
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008) ~[na:na]
at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1557) ~[netty-handler-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1571) ~[netty-handler-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1455) ~[netty-handler-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1282) ~[netty-handler-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1329) ~[netty-handler-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) ~[netty-codec-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) ~[netty-codec-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) ~[netty-transport-native-epoll-4.1.58.Final-linux-x86_64.jar!/:4.1.58.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) ~[netty-transport-native-epoll-4.1.58.Final-linux-x86_64.jar!/:4.1.58.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) ~[netty-transport-native-epoll-4.1.58.Final-linux-x86_64.jar!/:4.1.58.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.58.Final.jar!/:4.1.58.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.58.Final.jar!/:4.1.58.Final]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
A first intuition was to look inside the server trust store, and for sure, I am trusting the root of the certificate chain used in the client keystore.
- Why is it a "bad" certificate?
- Why the client certificate chain is empty? I Build the client keystore with a full chain.
I will be glad if I can understand the root cause of this issue and how to fix it.
Thank you
