Jetty sends different SSL certificate than what's configured in keystore

Question

Jetty 9.3.1.v20150714 configured with a keystore containing 3 certificates: server's certificate, CA cert and GeoTrust root cert.

Using keytool -list -keystore jetty/etc/keystore -storetype pkcs12 shows it has these certs -

Keystore type: PKCS12
Keystore provider: SunJSSE

Your keystore contains 4 entries

jetty, Aug 29, 2015, PrivateKeyEntry, 
Certificate fingerprint (SHA1): A3:...:10
root, Aug 30, 2015, trustedCertEntry, 
Certificate fingerprint (SHA1): DE:28:F4:A4:FF:E5:B9:2F:A3:C5:03:D1:A3:49:A7:F9:96:2A:82:12
ca, Aug 30, 2015, trustedCertEntry, 
Certificate fingerprint (SHA1): 0E:34:14:18:46:E7:42:3D:37:F2:0D:C0:AB:06:C9:BB:D8:43:DC:24
server, Aug 30, 2015, trustedCertEntry, 
Certificate fingerprint (SHA1): 20:...:E3

When server loads, log shows it loads the certificates -

2015-08-30 05:32:47 main ServerConnector [INFO] Started ServerConnector@34849eef{HTTP/1.1,[http/1.1, h2c, h2c-17, h2c-16, h2c-15, h2c-14]}{0.0.0.0:8080}
2015-08-30 05:32:47 main SslContextFactory [DEBUG] Certificate cn alias=jetty cn=example.com in SslContextFactory@7af707e0(file:///usr/share/jetty/jetty-distribution-9.3.1.v20150714/etc/keystore,file:///usr/share/jetty/jetty-distribution-9.3.1.v20150714/etc/keystore)
2015-08-30 05:32:47 main SslContextFactory [DEBUG] Certificate cn alias=root cn=GeoTrust Global CA in SslContextFactory@7af707e0(file:///usr/share/jetty/jetty-distribution-9.3.1.v20150714/etc/keystore,file:///usr/share/jetty/jetty-distribution-9.3.1.v20150714/etc/keystore)
2015-08-30 05:32:47 main SslContextFactory [DEBUG] Certificate SAN alias=server cn=example.com in SslContextFactory@7af707e0(file:///usr/share/jetty/jetty-distribution-9.3.1.v20150714/etc/keystore,file:///usr/share/jetty/jetty-distribution-9.3.1.v20150714/etc/keystore)
2015-08-30 05:32:47 main SslContextFactory [INFO] x509={example.com=server} wild={} alias=null for SslContextFactory@7af707e0(file:///usr/share/jetty/jetty-distribution-9.3.1.v20150714/etc/keystore,file:///usr/share/jetty/jetty-distribution-9.3.1.v20150714/etc/keystore)
2015-08-30 05:32:47 main SslContextFactory [DEBUG] managers=[sun.security.ssl.SunX509KeyManagerImpl@36c54a56] for SslContextFactory@7af707e0(file:///usr/share/jetty/jetty-distribution-9.3.1.v20150714/etc/keystore,file:///usr/share/jetty/jetty-distribution-9.3.1.v20150714/etc/keystore)

But when trying to connect the server using HTTPS, it fails due to bad (self signed) certificate.

Inspecting the returned certificate with openssl s_client -connect example.com:443 returns -

CONNECTED(00000003)
depth=0 C = ..., CN = example.com
verify error:num=18:self signed certificate
verify return:1
depth=0 C = ..., CN = example.com
verify return:1
---
Certificate chain
 0 s:/C=.../CN=example.com
   i:/C=.../CN=example.com
--- 

This shows only 1, self-signed certificate, instead of the ones in the keystore. When trying to send an HTTPS command to the server, it shows in logs -

2015-08-30 06:06:52 qtp1579572132-17 SslContextFactory [DEBUG] Customize 1d4638d[SSLEngine[hostname=12.13.14.15 port=25453] SSL_NULL_WITH_NULL_NULL]
2015-08-30 06:06:52 qtp1579572132-17 SslContextFactory [DEBUG] Enable SNI matching 1d4638d[SSLEngine[hostname=12.13.14.15 port=25453] SSL_NULL_WITH_NULL_NULL]
2015-08-30 06:06:52 qtp1579572132-20 SslContextFactory [DEBUG] matches=type=host_name (0), value=example.com for org.eclipse.jetty.util.ssl.SslContextFactory$AliasSNIMatcher@22497dda
2015-08-30 06:06:52 qtp1579572132-20 SslContextFactory [DEBUG] matched example.com->server
2015-08-30 06:06:52 qtp1579572132-16 HttpParser [WARN] parse exception: java.lang.IllegalStateException: too much data seeking EOF in CLOSE for HttpChannelOverHttp@37b97125{r=1,c=false,a=IDLE,uri=-}

I've made sure it's connecting to the correct server by shutting it down, then got a CONNECTION REFUSED, also tried to remove the keystore file and server failed to load, meaning it was using the correct file.

Where does this self signed, 1-long certificate chain come from? Is it possible the server has two keystores defined?

Update

Replacing the keystore with a different (working) one was shown correctly using openssl s_client, with all 3 certs.

This means that problem lies in the original keystore, as shown above it has 3 public certs, but openssl outputs only 1, self-signed cert.

Here are the commands used to produce the keystore -

keytool -genkeypair -storetype pkcs12 -keyalg RSA -keysize 2048 -keystore keystore.p12 -alias jetty
keytool -certreq -alias jetty -keystore keystore.p12 -storetype pkcs12 -file jetty.csr -keyalg RSA

# send CSR file to a CA for signing. got back 3 CRT files.

keytool -import -alias root -keystore keystore.p12 -storetype pkcs12 -trustcacerts -file root.crt 
keytool -import -alias ca -keystore keystore.p12 -storetype pkcs12 -trustcacerts -file ca.crt
keytool -import -alias server -keystore keystore.p12 -storetype pkcs12 -trustcacerts -file server.crt

Answer 1

Keystore was built incorrectly.

The last entry added to the keystore had the alias server instead of jetty, which is the alias of the private key. Doing this prevented from recognising the certificate reply to the private key.

To fix this, I've removed all certificates from the keystore and added them with the correct aliases.

keytool -import -alias root -keystore keystore.p12 -storetype pkcs12 -trustcacerts -file root.crt 
keytool -import -alias inter -keystore keystore.p12 -storetype pkcs12 -trustcacerts -file ca.crt
keytool -import -alias jetty -keystore keystore.p12 -storetype pkcs12 -trustcacerts -file server.crt

after the last command, keytool will respond with Certificate reply was installed in keystore, indicates it recognises the server certificate response.

I'm not sure if it has an affect, but instead of calling the CA's intermediate certificate ca, I called it inter in the working keystore, I don't think it's required but this worth mentioning in case it does.