Understanding the output of openssl s_client

Question

Ever since our email provider changed their SSL certificate, a POP3 client based on mono refuses to connect to their secure POP server to download emails. Other clients do not have an issue; e.g. Thunderbird and Outlook; neither does most SSL checker sites that are capable of checking odd ports except this one. I have been working with both providers in an attempt to pinpoint the problem, but have finally reached a dead-end with both, since I don't know enough about SSL Certificates to be able to guide either provider to understand where the fault lies.

During the investigation, my attention was drawn to the difference in output of the following two commands (I have removed the certificates from the output for readability):

echo "" | openssl s_client -showcerts -connect pop.gmail.com:995

CONNECTED(00000003)
depth=2 /C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=pop.gmail.com
   i:/C=US/O=Google Inc/CN=Google Internet Authority G2
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
 1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
   i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
 2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
   i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
---
Server certificate
subject=/C=US/ST=California/L=Mountain View/O=Google Inc/CN=pop.gmail.com
issuer=/C=US/O=Google Inc/CN=Google Internet Authority G2
---
No client certificate CA names sent
---
SSL handshake has read 3236 bytes and written 435 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : RC4-SHA
    Session-ID: 745F84194498529B91B7D9194363CBBD23425446CF2BFF3BF5557E3C6606CA06
    Session-ID-ctx:
    Master-Key: DED1AE0A44609F9D6F54626F4370ED96436A561A59F64D66240A277066322DCD2CCB9A6D198C95F4D2B0C7E6781EECD2
    Key-Arg   : None
    Start Time: 1397678434
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---
+OK Gpop ready for requests from 69.3.61.10 c13mb42148040pdj
DONE

echo "" | openssl s_client -showcerts -connect secure.emailsrvr.com:995

CONNECTED(00000003)
depth=2 /C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
Certificate chain
 0 s:/serialNumber=tG0GnsyAUkdX7DEo15ylNBjQJqAWZ/dD/OU=4159320284/OU=See www.rapidssl.com/resources/cps (c)14/OU=Domain Control Validated - RapidSSL(R)/CN=secure.emailsrvr.com
   i:/C=US/O=GeoTrust, Inc./CN=RapidSSL CA
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
 1 s:/C=US/O=GeoTrust, Inc./CN=RapidSSL CA
   i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
 2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
   i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
---
Server certificate
subject=/serialNumber=tG0GnsyAUkdX7DEo15ylNBjQJqAWZ/dD/OU=4159320284/OU=See www.rapidssl.com/resources/cps (c)14/OU=Domain Control Validated - RapidSSL(R)/CN=secure.emailsrvr.com
issuer=/C=US/O=GeoTrust, Inc./CN=RapidSSL CA
---
No client certificate CA names sent
---
SSL handshake has read 3876 bytes and written 319 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: 3F4EE3992B46727BE2C7C3E76A9A6A8D64D66EE843CB1BB17A76AE2E030C7161
    Session-ID-ctx:
    Master-Key: 016209E50432EFE2359DB73AB527AF718152BFE6F88215A9CE40604E8FF2E2A3AC97A175F46DF737596866A8BC8E3F7F
    Key-Arg   : None
    Start Time: 1397678467
    Timeout   : 300 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
---
DONE

I have been trying to understand if this is meaningful, because when the -CApath option is provided, the commands do not produce any errors:

openssl s_client -CApath /etc/ssl/certs -showcerts -connect secure.emailsrvr.com:995

CONNECTED(00000003)
depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
verify return:1
depth=1 C = US, O = "GeoTrust, Inc.", CN = RapidSSL CA
verify return:1
depth=0 serialNumber = tG0GnsyAUkdX7DEo15ylNBjQJqAWZ/dD, OU = 4159320284, OU = See www.rapidssl.com/resources/cps (c)14, OU = Domain Control Validated - RapidSSL(R), CN = secure.emailsrvr.com
verify return:1
...

openssl s_client -CApath /etc/ssl/certs -showcerts -connect pop.gmail.com:995

CONNECTED(00000003)
depth=3 C = US, O = Equifax, OU = Equifax Secure Certificate Authority
verify return:1
depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
verify return:1
depth=1 C = US, O = Google Inc, CN = Google Internet Authority G2
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google Inc, CN = pop.gmail.com
verify return:1
...

I can also use the -CAfile option successfully after downloading the CAfile cert directly from GeoTrust.

Nevertheless, Fog Creek seems to think that problem lies with the cert, because they've tried adding the cert to mono's Trust store without success. I would disagree with them, but (as mentioned above) while most SSL checkers either don't check port 995 or succeed during the attempt, I found this page that produces SSL error 7.

Do I interpret the output correctly to mean that there is nothing wrong with the cert?

I think the "self signed certificate in certificate chain" error is a red herring. A root CA is always self signed, so a server that returns its full certificate chain will always return a self signed certificate. More info [here](http://stackoverflow.com/a/4106224/2454476). — bennettp123, Apr 17 '14 at 06:47
Actually, it seems `openssl s_client` doesn't import any root certs by default. Try this instead: `openssl s_client -connect secure.emailsrvr.com:995 -showcerts -CApath /etc/ssl/certs`, and you'll probably find that the self-signed error disappears. — bennettp123, Apr 17 '14 at 06:53
@bennettp123 I note the output of that command towards the bottom of the question. Am I right to understand the output of both versions of the command to mean that the cert is valid? — jobu1324, Apr 17 '14 at 16:33
yes, according to that output, openssl thinks the cert is valid. Why it's being rejected? I don't know. It might be because the Organization field is not set, but that's just a guess. — bennettp123, Apr 18 '14 at 17:10

score 5 · Answer 1 · edited Mar 17 '17 at 13:14

The answer (as explained in this security.SE post) is that the two GeoTrust Global CA certificate you see in the chain are in fact not the same certificate, one is derived from the other.

Because of CA Cross-signing!

When the GeoTrust Global CA certificate was first created and signed, no computer/browsers/applications would have had it in their trust store.

By having another CA (with a pre-existing reputation and distribution) sign the GeoTrust root CA cert, the resulting certificate (referred to as a "bridge" certificate) can now be verified by the second CA, without the GeoTrust root CA certificate having to be trusted by the client explicitly.

When Google presents the Cross-signed version of the GeoTrust root CA cert, a client that doesn't trust the original can just use the Equifax CA cert to verify GeoTrust - so Equifax acts as a kind of "legacy" trust anchor.

That's why the two server chains are different and yet both valid. But it isn't *necessarily* the reason for the OP's problem with the mono client, without clear data on exactly which roots are and are not installed in the particular mono instance's truststore. — dave_thompson_085, May 03 '15 at 07:54

score 0 · Answer 2 · edited Apr 18 '16 at 14:32

0

While generating and configuring certificates, one should update openssl.cnf file as well (Debian - /etc/ssl/openssl.cnf), to indicate proper path, cert names etc., then you can run command and check them without -CApath option.

And accordingly remote hosts also could check your certificates properly in this case.

Here is the appropriate openssl.cnf section:

####################################################################
[ ca ]

default_ca  = CA_default        # The default ca section

####################################################################
[ CA_default ]

dir     = /etc/ssl

edited Apr 18 '16 at 14:32

techraf

4,243
8
29
44

answered Oct 24 '14 at 17:26

cmpt

87
1
5

1

This is **wrong**. The `default_ca` data in the (any) openssl config file is used *only* for the 'ca' utility to issue and optionally revoke certs, never for verification. The way to change the default verification store (besides recompiling) is with **environment variables** SSL_CERT_{FILE,DIR}. However (1) due to a bug `s_client` doesn't use the default (fix planned as of Apr. 2015) which (2) this OP didn't want to change anyway. – dave_thompson_085 May 03 '15 at 07:51

score 0 · Answer 3 · edited Jun 10 '15 at 20:29

0

Had similar problem with fetchmail when I enabled ssl check for pop.gmail.com.

I downloaded Equifax pem file but it did not work as is, had to run c_rehash ssl/certs which created a symbolic link with hash value, it then just worked.

Alternatively, hash value can be also known by running...

openssl x509 -in Equifax_Secure_Certificate_Authority.pem -fingerprint -subject -issuer -serial -hash -noout | sed  -n /^[0-9]/p

https://www.geotrust.com/resources/root_certificates/certificates/Equifax_Secure_Certificate_Authority.pem

edited Jun 10 '15 at 20:29

sebix

4,313
2
29
47

answered Jun 10 '15 at 20:22

chetangb

145
6

Could you please extend what to do with the linked pem-file? – sebix Jun 10 '15 at 20:29
# ldd /usr/bin/fetchmail | grep ssl libssl.so.10 => /lib64/libssl.so.10 – chetangb Jun 10 '15 at 20:54
what I read sometime back is fetchmail uses openssl libs and it only looks by the hash value of the cert https://productforums.google.com/forum/#!topic/gmail/tqjOmqxpMKY # ldd /usr/bin/fetchmail | grep ssl libssl.so.10 => /lib64/libssl.so.10 yum whatprovides libssl.so.10 openssl-libs-1.0.1e-42.el7.i686 : A general purpose cryptography library with TLS implementation Repo : base Matched from: Provides : libssl.so.10 – chetangb Jun 10 '15 at 21:02
Please extend your answer and explain what the problem may be, you want to achieve. – sebix Jun 11 '15 at 10:23

Understanding the output of openssl s_client

3 Answers3

Because of CA Cross-signing!

Linked