0

in our e-commerce service we are trying to implement mutual TLS (TLS1.2 client certificate validation) between a generic client (I use the openssl utility CLI) and a webapp behind nginx. It appears our openssl s_client request succeed either we supply the client cert or not. What's wrong?

This is the nginx server block:

## start server b2b.ourservice.ourdomain
    server {
        server_name b2b.ourservice.ourdomain ;

        listen 443 proxy_protocol  ssl http2;

        # PEM sha: 13defe88e3695d7cccf1043a2e342433b65783
        ssl_certificate                         /etc/ingress-controller/ssl/default-fake-certificate.pem;
        ssl_certificate_key                     /etc/ingress-controller/ssl/default-fake-certificate.pem;

        ssl_certificate_by_lua_block {
            certificate.call()
        }

        # PEM sha: 85e928be3bafef2342342051944328ddbda
        ssl_client_certificate                  /etc/ingress-controller/ssl/ca-ingress-nginx-client-ca-b2b.ourservice.ourdomain.pem;
        ssl_verify_client                       on;
        ssl_verify_depth                        1;

        location / {

            set $namespace      "ecommerce";
            set $ingress_name   "b2b.ourservice.ourdomain-servicename";
            set $service_name   "ecommerce-ecommerce-servicename-default-service";
            set $service_port   "4000";
            set $location_path  "/";

            rewrite_by_lua_block {
                lua_ingress.rewrite({
                    force_ssl_redirect = true,
                    use_port_in_redirects = false,
                })
                balancer.rewrite()
                plugins.run()
            }

            header_filter_by_lua_block {

                plugins.run()
            }
            body_filter_by_lua_block {

            }

            log_by_lua_block {

                balancer.log()

                monitor.call()

                plugins.run()
            }

            if ($scheme = https) {
                more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains";
            }

            port_in_redirect off;

            set $proxy_upstream_name    "ecommerce-ecommerce-ecommerce-servicename-default-service-4000";
            set $proxy_host             $proxy_upstream_name;

            if ($deny_hVFCsxGzxnnReAIQoIeOZBVvLpIPSBWw) {
                return 403;
            }

            client_max_body_size                    5m;

            proxy_set_header Host                   $best_http_host;

            # Pass the extracted client certificate to the backend

            proxy_set_header ssl-client-cert        $ssl_client_escaped_cert;

            proxy_set_header ssl-client-verify      $ssl_client_verify;
            proxy_set_header ssl-client-subject-dn  $ssl_client_s_dn;
            proxy_set_header ssl-client-issuer-dn   $ssl_client_i_dn;

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;

            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Request-ID           $req_id;
            proxy_set_header X-Real-IP              $the_real_ip;

            proxy_set_header X-Forwarded-For        $the_real_ip;

            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

            proxy_set_header X-Original-URI         $request_uri;

            proxy_set_header X-Scheme               $pass_access_scheme;

            # Pass the original X-Forwarded-For
            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers to proxied server

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      120s;
            proxy_read_timeout                      120s;

            proxy_buffering                         off;
            proxy_buffer_size                       4k;
            proxy_buffers                           4 4k;
            proxy_request_buffering                 on;

            proxy_http_version                      1.1;

            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;

            # In case of errors try the next upstream server before returning an error
            proxy_next_upstream                     error;
            proxy_next_upstream_tries               3;

            if ( $request_method !~ ^(GET|POST|OPTIONS|DELETE|PUT|HEAD)$ ) {
                return 405;
            }
            proxy_ssl_server_name on;
            set $vpn "yes";
            if ($not_vpn_ip) {
                set $vpn "no";
            }

            if ($ingress_name != "") {
                set $service "servicename";
                set $subenv "ecommerce";
                set $service_id "ecommerce-ecommerce-servicename";
                set $real_ingress "true";
            }

            set $cors "false";
            set $cors_done "false";

            proxy_pass https://upstream_balancer;

            proxy_redirect                          off;

        }

    }
    ## end server b2b.ourservice.ourdomain

Our configuration is simple and we don't have any Intermediate Certificates. So, having

ssl_verify_client                       on;
ssl_verify_depth                        1;

should be enough.

Let's verify the SSL connection without appending certificates:

openssl s_client -connect b2b.ourservice.ourdomain-servicename:443 -state -debug -servername b2b.ourservice.ourdomain-servicename
CONNECTED(00000005)
SSL_connect:before/connect initialization
write to 0x7f95d8400120 [0x7f95d881be03] (221 bytes => 221 (0xDD))
0000 - 16 03 01 00 d8 01 00 00-d4 03 03 cd 5e db 91 df   ............^...
00b0 - 08 00 06 00 1d 00 17 00-18 00 23 00 00 00 0d 00   ..........#.....
00c0 - 1c 00 1a 06 01 06 03 ef-ef 05 01 05 03 04 01 04   ................
00d0 - 03 ee ee ed ed 03 01 03-03 02 01 02 03            .............
SSL_connect:SSLv3 write client hello A
read from 0x7f95d8400120 [0x7f95d8817c03] (5 bytes => 5 (0x5))
0000 - 16 03 03 00 41                                    ....A
read from 0x7f95d8400120 [0x7f95d8817c08] (65 bytes => 65 (0x41))
0000 - 02 00 00 3d 03 03 ac 55-5c 00 fc 24 c1 91 b8 7a   ...=...U\..$...z
0010 - ba 93 45 eb ba e7 54 d4-41 4e bb b1 4c 9d 44 4f   ..E...T.AN..L.DO
0020 - 57 4e 47 52 44 01 00 cc-a8 00 00 15 ff 01 00 01   WNGRD...........
0030 - 00 00 00 00 00 00 0b 00-04 03 00 01 02 00 23      ..............#
0041 - <SPACES/NULS>
SSL_connect:SSLv3 read server hello A
read from 0x7f95d8400120 [0x7f95d8817c03] (5 bytes => 5 (0x5))
0000 - 16 03 03 11 92                                    .....
read from 0x7f95d8400120 [0x7f95d8817c08] (4498 bytes => 2763 (0xACB))
0000 - 0b 00 11 8e 00 11 8b 00-08 2e 30 82 08 2a 30 82   ..........0..*0.
0110 - 32 33 32 33 5a 30 63 31-0b 30 09 06 03 55 04 06   2323Z0c1.0...U..
0120 - 13 02 49 54 31 0f 30 0d-06 03 55 04 08 13 06 54   .COUNTRY1.0....C
0130 - 6f 72 69 6e 6f 31 0e 30-0c 06 03 55 04 07 13 05   ityname....U....
0140 - 49 76 72 65 61 31 18 30-16 06 03 55 04 0a 13 0f   Province1.0.....
0150 - 57 69 6e 64 20 54 72 65-20 53 2e 70 2e 41 2e 31   Company..Name...
0160 - 19 30 17 06 03 55 04 03-13 10 62 32 62 2e 73 68   .0...U....b2b.ou
0170 - 6f 70 2e 77 69 6e 64 2e-69 74 30 82 02 22 30 0d   serv.domain0.."0.
01c0 - 93 ff 51 10 ac 15 54 01-d8 fc 05 b1 89 a1 7f 74   ..Q...T........t
01d0 - 83 9a 49 d7 dc 4e 7b 8a-48 6f 8b 45 f6            ..I..N{.Ho.E....
depth=2 C = US, O = "Entrust, Inc.", OU = See www.entrust.net/legal-terms, OU = "(c) 2009 Entrust, Inc. - for authorized use only", CN = Entrust Root Certification Authority - G2
verify return:1
depth=1 C = US, O = "Entrust, Inc.", OU = See www.entrust.net/legal-terms, OU = "(c) 2012 Entrust, Inc. - for authorized use only", CN = Entrust Certification Authority - L1K
verify return:1
depth=0 C = OURCOUNTRY, ST = OURCITY, L = OURPROVINCE, O = OURCOMPANY, CN = b2b.ourservice.ourdomain-servicename
verify return:1
SSL_connect:SSLv3 read server certificate A
read from 0x7f95d8400120 [0x7f95d8817c03] (5 bytes => 5 (0x5))
0000 - 16 03 03 02 2c                                    ....,
read from 0x7f95d8400120 [0x7f95d8817c08] (556 bytes => 556 (0x22C))
0000 - 0c 00 02 28 03 00 1d 20-a3 e5 d0 75 f6 6b 49 e0   ...(... ...u.kI.
0010 - 79 8d 06 3e f1 66 30 cf-51 52 40 d2 bd d4 97 f9   y..>.f0.QR@.....
0020 - f2 b0 c4 07 7f 37 58 74-04 01 02 00 22 72 3b dc   .....7Xt...."r;.
0030 - c1 dd 02 33 cc 57 46 fc-1b b9 b9 dc 64 c9 be 96   ...3.WF.....d...
00c0 - d0 21 30 5a da 42 fc 6b-73 1c 3d 70 58 61 36 4a   .!0Z.B.ks.=pXa6J
0200 - ac 8e 3b c1 54 9d 79 b0-a3 56 4e 9b 55 8e fa 52   ..;.T.y..VN.U..R
0210 - 4e fa 99 c6 86 a8 3c b0-27 e4 33 58 4d ba f3 24   N.....<.'.3XM..$
0220 - b4 b9 0e 8b 57 3c ff c0-62 cc 75 ad               ....W<..b.u.
SSL_connect:SSLv3 read server key exchange A
read from 0x7f95d8400120 [0x7f95d8817c03] (5 bytes => 5 (0x5))
0000 - 16 03 03 00 90                                    .....
read from 0x7f95d8400120 [0x7f95d8817c08] (144 bytes => 144 (0x90))
0040 - 06 13 02 49 54 31 2c 30-2a 06 03 55 04 03 0c 23   ...IT1,0*..U...#
0050 - 62 32 62 2e 73 68 6f 70-2e 77 69 6e 64 2e 69 74   b2b.ourservice.ourdomain-servicename
0060 - 2d 63 6c 69 65 6e 74 5f-63 61 2e 64 65 76 2e 77   -client_ca.dev.o
0070 - 69 64 69 31 0e 30 0c 06-03 55 04 07 0c 05 4d 62   urdomain........
0080 - 6c 61 6e 31 0b 30 09 06-03 55 04 08 0c 02 49 54   city.....COUNTRY
SSL_connect:SSLv3 read server certificate request A
read from 0x7f95d8400120 [0x7f95d8817c03] (5 bytes => 5 (0x5))
0000 - 16 03 03 00 04                                    .....
read from 0x7f95d8400120 [0x7f95d8817c08] (4 bytes => 4 (0x4))
0000 - 0e                                                .
0004 - <SPACES/NULS>
SSL_connect:SSLv3 read server done A
write to 0x7f95d8400120 [0x7f95d8811600] (12 bytes => 12 (0xC))
0000 - 16 03 03 00 07 0b 00 00-03                        .........
000c - <SPACES/NULS>
SSL_connect:SSLv3 write client certificate A
write to 0x7f95d8400120 [0x7f95d8811600] (42 bytes => 42 (0x2A))
0000 - 16 03 03 00 25 10 00 00-21 20 63 75 51 9f 11 e7   ....%...! cuQ...
0010 - 5d 51 de 5a 06 8f 9d 10-5b 47 59 17 b8 56 93 68   ]Q.Z....[GY..V.h
0020 - 25 14 99 8b 5c a9 da ed-34 24                     %...\...4$
SSL_connect:SSLv3 write client key exchange A
write to 0x7f95d8400120 [0x7f95d8811600] (6 bytes => 6 (0x6))
0000 - 14 03 03 00 01 01                                 ......
SSL_connect:SSLv3 write change cipher spec A
write to 0x7f95d8400120 [0x7f95d8811600] (37 bytes => 37 (0x25))
0000 - 16 03 03 00 20 35 5c 7d-7b ef 46 8d c1 71 08 25   .... 5\}{.F..q.%
0010 - da 16 1d ce cc cb c9 90-76 f4 85 be 01 91 b1 65   ........v......e
0020 - 6c 75 31 59 b4                                    lu1Y.
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
read from 0x7f95d8400120 [0x7f95d8817c03] (5 bytes => 5 (0x5))
0000 - 16 03 03 00 ca                                    .....
read from 0x7f95d8400120 [0x7f95d8817c08] (202 bytes => 202 (0xCA))
0000 - 04 00 00 c6 00 00 02 58-00 c0 81 1c 35 a4 f3 15   .......X....5...
0010 - f7 2c 85 06 fe 84 53 ec-45 85 af 0a 9c 63 dc a7   .,....S.E....c..
00b0 - 82 d8 7d 01 6f 07 97 1f-49 ef e3 d4 45 aa f9 20   ..}.o...I...E.. 
00c0 - 92 64 da 1f 0c 3b c5 d9-e5 db                     .d...;....
SSL_connect:SSLv3 read server session ticket A
read from 0x7f95d8400120 [0x7f95d8817c03] (5 bytes => 5 (0x5))
0000 - 14 03 03 00 01                                    .....
read from 0x7f95d8400120 [0x7f95d8817c08] (1 bytes => 1 (0x1))
0000 - 01                                                .
read from 0x7f95d8400120 [0x7f95d8817c03] (5 bytes => 5 (0x5))
0000 - 16 03 03                                          ...
0005 - <SPACES/NULS>
read from 0x7f95d8400120 [0x7f95d8817c08] (32 bytes => 32 (0x20))
0000 - 23 e5 35 d1 f7 bc ba bd-9f 13 a5 5f e1 9a 3d c6   #.5........_..=.
0010 - 85 75 d7 0b 2d d7 f7 e0-e2 27 6e fc 1c 21 49 51   .u..-....'n..!IQ
SSL_connect:SSLv3 read finished A
---
Certificate chain
 0 s:/C=OURCOUNTRY/ST=OURCITY/L=OURPROVINCE/O=OURCOMPANY/CN=b2b.ourservice.ourdomain-servicename
   i:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2012 Entrust, Inc. - for authorized use only/CN=Entrust Certification Authority - L1K
 1 s:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2012 Entrust, Inc. - for authorized use only/CN=Entrust Certification Authority - L1K
   i:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2009 Entrust, Inc. - for authorized use only/CN=Entrust Root Certification Authority - G2
 2 s:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2009 Entrust, Inc. - for authorized use only/CN=Entrust Root Certification Authority - G2
   i:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2009 Entrust, Inc. - for authorized use only/CN=Entrust Root Certification Authority - G2
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIIKjCCBxKgAwIBAgIQOP5SIrZ9PT0AAAAAUOMHezANBgkqhkiG9w0BAQsFADCB
ujELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUVudHJ1c3QsIEluYy4xKDAmBgNVBAsT
dIAga7lZQ3Cmo8PMwWQek59kyYavIap7chrmElKtpU4D4SBZg8c4/apb7s/39sff
bFhAdYGpflhUI9jEsIJKm8TG2OVWgtmyFsP8IUBt
-----END CERTIFICATE-----
subject=/C=OURCOUNTRY/ST=OURCITY/L=OURPROVINCE/O=OURCOMPANY/CN=b2b.ourservice.ourdomain-servicename
issuer=/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2012 Entrust, Inc. - for authorized use only/CN=Entrust Certification Authority - L1K
---
Acceptable client certificate CA names
/C=OURCOUNTRY/CN=b2b.ourservice.ourdomain-servicename-client_ca.dev.oursuffix/L=OURCITY/ST=OURCOUNTRY
Server Temp Key: ECDH, X25519, 253 bits
---
SSL handshake has read 5542 bytes and written 318 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-CHACHA20-POLY1305
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-CHACHA20-POLY1305
    Session-ID: 69958835E239D7BAFF9351355FA424BD27162B95468FFDF751454AA081ED52BB
    Session-ID-ctx: 
    Master-Key: 7C1208DD1DCD7FB50F9CC551BD1DE3301D51E4A7B191E9F403B24E095CF5B04A8317E9EDCD8F57E258870DA034BBF41B
    TLS session ticket lifetime hint: 600 (seconds)
    TLS session ticket:
    0000 - 81 1c 35 a4 f3 15 f7 2c-85 06 fe 84 53 ec 45 85   ..5....,....S.E.
    0010 - af 0a 9c 63 dc a7 20 4a-a6 e5 3c 9d 1d 8a 97 24   ...c.. J..<....$
    0090 - 93 c8 74 cb 5a 35 be 00-47 7a 29 08 8c 70 70 8b   ..t.Z5..Gz)..pp.
    00a0 - d1 16 ed c6 69 6c 82 d8-7d 01 6f 07 97 1f 49 ef   ....il..}.o...I.
    00b0 - e3 d4 45 aa f9 20 92 64-da 1f 0c 3b c5 d9 e5 db   ..E.. .d...;....

    Start Time: 1589375393
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---
read from 0x7f95d8400120 [0x7f95d8817c03] (5 bytes => 5 (0x5))
0000 - 15 03 03 00 12                                    .....
read from 0x7f95d8400120 [0x7f95d8817c08] (18 bytes => 18 (0x12))
0000 - 51 07 60 3c 5a 5b 8a fe-ec 02 1e 7e 56 22 3b 03   Q.`<Z[.....~V";.
0010 - 53 95                                             S.
SSL3 alert read:warning:close notify
closed
write to 0x7f95d8400120 [0x7f95d881be03] (23 bytes => 23 (0x17))
0000 - 15 03 03 00 12 b1 22 78-82 37 cf 9d e0 c4 76 a2   ......"x.7....v.
0010 - 4a c7 ee 75 be 25 c9                              J..u.%.
SSL3 alert write:warning:close notify

Uhm, it appears we are getting far without sending it! Let's see what happens when we submit it instead:

openssl s_client -connect b2b.ourservice.ourdomain:443 -state -debug -servername b2b.ourservice.ourdomain -cert b2b.ourservice.ourdomain-client_ca.pem -key b2b.ourservice.ourdomain-client_ca.key
CONNECTED(00000005)
SSL_connect:before/connect initialization
write to 0x7f884245f650 [0x7f884281a003] (221 bytes => 221 (0xDD))
0000 - 16 03 01 00 d8 01 00 00-d4 03 03 0f a2 7f ba a4   ................
0010 - d5 25 53 10 44 57 6a 9d-d0 32 bb 80 59 cd db ab   .%S.DWj..2..Y...
0020 - 37 60 f5 d4 10 98 9f 32-e9 7b c6 00 00 5c c0 30   7`.....2.{...\.0
0080 - c0 12 c0 08 00 16 00 0a-00 ff 01 00 00 4f 00 00   .............O..
0090 - 00 15 00 13 00 00 10 62-32 62 2e 73 68 6f 70 2e   .......b2b.ourse
00a0 - 77 69 6e 64 2e 69 74 00-0b 00 02 01 00 00 0a 00   ourdomain.........
00b0 - 08 00 06 00 1d 00 17 00-18 00 23 00 00 00 0d 00   ..........#.....
00c0 - 1c 00 1a 06 01 06 03 ef-ef 05 01 05 03 04 01 04   ................
00d0 - 03 ee ee ed ed 03 01 03-03 02 01 02 03            .............
SSL_connect:SSLv3 write client hello A
read from 0x7f884245f650 [0x7f8842815e03] (5 bytes => 5 (0x5))
0000 - 16 03 03 00 41                                    ....A
read from 0x7f884245f650 [0x7f8842815e08] (65 bytes => 65 (0x41))
0000 - 02 00 00 3d 03 03 79 02-18 e9 d3 a0 de 75 ea 73   ...=..y......u.s
0010 - c6 cb 62 d1 8d 92 58 2c-b2 c4 c2 86 88 04 44 4f   ..b...X,......DO
0020 - 57 4e 47 52 44 01 00 cc-a8 00 00 15 ff 01 00 01   WNGRD...........
0030 - 00 00 00 00 00 00 0b 00-04 03 00 01 02 00 23      ..............#
0041 - <SPACES/NULS>
SSL_connect:SSLv3 read server hello A
read from 0x7f884245f650 [0x7f8842815e03] (5 bytes => 5 (0x5))
0000 - 16 03 03 11 92                                    .....
read from 0x7f884245f650 [0x7f8842815e08] (4498 bytes => 2763 (0xACB))
0070 - 77 77 2e 65 6e 74 72 75-73 74 2e 6e 65 74 2f 6c   ww.entrust.net/l
0080 - 65 67 61 6c 2d 74 65 72-6d 73 31 39 30 37 06 03   egal-terms1907..
0090 - 55 04 0b 13 30 28 63 29-20 32 30 31 32 20 45 6e   U...0(c) 2012 En
00a0 - 74 72 75 73 74 2c 20 49-6e 63 2e 20 2d 20 66 6f   trust, Inc. - fo
00b0 - 72 20 61 75 74 68 6f 72-69 7a 65 64 20 75 73 65   r authorized use
00c0 - 20 6f 6e 6c 79 31 2e 30-2c 06 03 55 04 03 13 25    only1.0,..U...%
00d0 - 45 6e 74 72 75 73 74 20-43 65 72 74 69 66 69 63   Entrust Certific
00e0 - 61 74 69 6f 6e 20 41 75-74 68 6f 72 69 74 79 20   ation Authority 
00f0 - 2d 20 4c 31 4b 30 1e 17-0d 31 38 30 35 32 33 30   - L1K0...1805230
0100 - 38 35 33 32 35 5a 17 0d-32 30 30 35 32 33 30 39   85325Z..20052309
0110 - 32 33 32 33 5a 30 63 31-0b 30 09 06 03 55 04 06   2323Z0c1.0...U..
0120 - 13 02 49 54 31 0f 30 0d-06 03 55 04 08 13 06 54   ..IT1.0...U....T
0130 - 6f 72 69 6e 6f 31 0e 30-0c 06 03 55 04 07 13 05   orino1.0...U....
0140 - 49 76 72 65 61 31 18 30-16 06 03 55 04 0a 13 0f   OURPROVINCE1.0..
0150 - 57 69 6e 64 20 54 72 65-20 53 2e 70 2e 41 2e 31   OURCOMPANY1.....

01d0 - 83 9a 49 d7 dc 4e 7b 8a-48 6f 8b 45 f6            ..I..N{.Ho.E.
depth=2 C = US, O = "Entrust, Inc.", OU = See www.entrust.net/legal-terms, OU = "(c) 2009 Entrust, Inc. - for authorized use only", CN = Entrust Root Certification Authority - G2
verify return:1
depth=1 C = US, O = "Entrust, Inc.", OU = See www.entrust.net/legal-terms, OU = "(c) 2012 Entrust, Inc. - for authorized use only", CN = Entrust Certification Authority - L1K
verify return:1
depth=0 C = IT, ST = OURCITY, L = OURPROVINCE, O = OURCOMPANY, CN = b2b.ourservice.ourdomain
verify return:1
SSL_connect:SSLv3 read server certificate A
read from 0x7f884245f650 [0x7f8842815e03] (5 bytes => 5 (0x5))
0000 - 16 03 03 02 2c                                    ....,
read from 0x7f884245f650 [0x7f8842815e08] (556 bytes => 556 (0x22C))
0000 - 0c 00 02 28 03 00 1d 20-70 40 42 24 11 fc de 34   ...(... p@B$...4
0010 - 19 16 67 a8 f9 41 f5 93-f1 de 59 f4 ad f1 2c 08   ..g..A....Y...,.
0200 - 50 57 17 e3 dd af 8e d7-7b da cb 75 f3 b4 36 f6   PW......{..u..6.
0210 - c2 61 e4 2d c0 69 de ed-19 60 60 74 7c 4b e4 86   .a.-.i...``t|K..
0220 - 97 2d 59 f7 4b 59 4b 74-01 0b 5d 8d               .-Y.KYKt..].
SSL_connect:SSLv3 read server key exchange A
read from 0x7f884245f650 [0x7f8842815e03] (5 bytes => 5 (0x5))
0000 - 16 03 03 00 90                                    .....
read from 0x7f884245f650 [0x7f8842815e08] (144 bytes => 144 (0x90))
0000 - 0d 00 00 8c 03 01 02 40-00 28 04 03 05 03 06 03   .......@.(......
0010 - 08 07 08 08 08 09 08 0a-08 0b 08 04 08 05 08 06   ................
0020 - 04 01 05 01 06 01 03 03-03 01 03 02 04 02 05 02   ................
0030 - 06 02 00 5c 00 5a 30 58-31 0b 30 09 06 03 55 04   ...\.Z0X1.0...U.
0040 - 06 13 02 49 54 31 2c 30-2a 06 03 55 04 03 0c 23   ...IT1,0*..U...#
0050 - 62 32 62 2e 73 68 6f 70-2e 77 69 6e 64 2e 69 74   b2b.ourservice.o
0060 - 2d 63 6c 69 65 6e 74 5f-63 61 2e 64 65 76 2e 77   -client_ca.dev.o
0070 - 69 64 69 31 0e 30 0c 06-03 55 04 07 0c 05 4d 69   urdomain.0...U..
0080 - 6c 61 6e 31 0b 30 09 06-03 55 04 08 0c 02 49 54   city.0...U.OURCO
SSL_connect:SSLv3 read server certificate request A
read from 0x7f884245f650 [0x7f8842815e03] (5 bytes => 5 (0x5))
0000 - 16 03 03 00 04                                    .....
read from 0x7f884245f650 [0x7f8842815e08] (4 bytes => 4 (0x4))
0000 - 0e                                                .
0004 - <SPACES/NULS>
SSL_connect:SSLv3 read server done A
write to 0x7f884245f650 [0x7f884281e200] (1558 bytes => 1558 (0x616))
0030 - 0d 01 01 0b 05 00 30 58-31 0b 30 09 06 03 55 04   ......0X1.0...U.
0040 - 06 13 02 49 54 31 2c 30-2a 06 03 55 04 03 0c 23   ...IT1,0*..U...#
0050 - 62 32 62 2e 73 68 6f 70-2e 77 69 6e 64 2e 69 74   b2b.ourservice.o
0060 - 2d 63 6c 69 65 6e 74 5f-63 61 2e 64 65 76 2e 77   -client_ca.dev.o
0070 - 69 64 69 31 0e 30 0c 06-03 55 04 07 0c 05 4d 69   ordomain.0...U..
0080 - 6c 61 6e 31 0b 30 09 06-03 55 04 08 0c 02 49 54   city.0...U....IT
00b0 - 30 58 31 0b 30 09 06 03-55 04 06 13 02 49 54 31   0X1.0...U....IT1
00c0 - 2c 30 2a 06 03 55 04 03-0c 23 62 32 62 2e 73 68   ,0*..U...#b2b.ou
00d0 - 6f 70 2e 77 69 6e 64 2e-69 74 2d 63 6c 69 65 6e   op.ourdomain-cli
00e0 - 74 5f 63 61 2e 64 65 76-2e 77 69 64 69 31 0e 30   t_ca.dev.ourd1.0
00f0 - 0c 06 03 55 04 07 0c 05-4d 69 6c 61 6e 31 0b 30   ...U....OURCITY1
0100 - 09 06 03 55 04 08 0c 02-49 54 30 82 02 22 30 0d   ...U....CO0.."0.
03b0 - 06 03 55 04 03 0c 23 62-32 62 2e 73 68 6f 70 2e   ..U...#b2b.ourse
03c0 - 77 69 6e 64 2e 69 74 2d-63 6c 69 65 6e 74 5f 63   ourdomain-client_c
03d0 - 61 2e 64 65 76 2e 77 69-64 69 31 0e 30 0c 06 03   a.dev.ourd1.0...
03e0 - 55 04 07 0c 05 4d 69 6c-61 6e 31 0b 30 09 06 03   U....OURCITY1.0...
0430 - 4b 4e e2 8f 5c 8a ce 35-87 8c 0d a4 07 d1 66 c9   KN..\..5......f.
0440 - 61 d2 fc ae 3b 6f 22 49-a2 b7 71 97 57 de 4e 42   a...;o"I..q.W.NB

0520 - d9 a2 42 76 07 a2 fc da-93 52 dd 7d 6d d4 09 73   ..Bv.....R.}m..s
0610 - c4 f5 31 35 8f 9c                                 ..15..
SSL_connect:SSLv3 write client certificate A
write to 0x7f884245f650 [0x7f884281e200] (42 bytes => 42 (0x2A))
0000 - 16 03 03 00 25 10 00 00-21 20 8b d7 81 f2 5d 82   ....%...! ....].
0010 - 22 ff 87 89 89 ac f9 84-18 d6 6b 00 67 9b fb e2   ".........k.g...
0020 - 39 ed ea 65 0d 56 14 01-fd 38                     9..e.V...8
SSL_connect:SSLv3 write client key exchange A
write to 0x7f884245f650 [0x7f884281e200] (525 bytes => 525 (0x20D))
0000 - 16 03 03 02 08 0f 00 02-04 04 01 02 00 67 86 26   .............g.&
01e0 - 02 08 32 c5 15 22 5c ee-cc 88 71 e5 37 81 15 0d   ..2.."\...q.7...
01f0 - 65 90 ce da 76 66 ab 48-d8 21 e8 a2 d4 d8 0b 3b   e...vf.H.!.....;
0200 - f0 e1 80 5b 5e a0 71 89-65 52 ca 49 99            ...[^.q.eR.I.
SSL_connect:SSLv3 write certificate verify A
write to 0x7f884245f650 [0x7f884281e200] (6 bytes => 6 (0x6))
0000 - 14 03 03 00 01 01                                 ......
SSL_connect:SSLv3 write change cipher spec A
write to 0x7f884245f650 [0x7f884281e200] (37 bytes => 37 (0x25))
0000 - 16 03 03 00 20 6f c0 fa-44 e0 49 5c 89 fe 99 3b   .... o..D.I\...;
0010 - 94 7a aa b5 c8 55 b1 dd-2c e6 77 14 30 c7 fc 5f   .z...U..,.w.0.._
0020 - d7 c9 2e 37 01                                    ...7.
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
read from 0x7f884245f650 [0x7f8842815e03] (5 bytes => 5 (0x5))
0000 - 16 03 03 06 da                                    .....
read from 0x7f884245f650 [0x7f8842815e08] (1754 bytes => 1754 (0x6DA))
0000 - 04 00 06 d6 00 00 02 58-06 d0 e0 e1 66 06 35 97   .......X....f.5.

06c0 - 44 3c dc de 92 e9 09 73-bf 86 15 89 ea 01 4d bb   D<.....s......M.
06d0 - fd c3 2f 6c 6d b8 82 b5-1f 9d                     ../lm.....
SSL_connect:SSLv3 read server session ticket A
read from 0x7f884245f650 [0x7f8842815e03] (5 bytes => 5 (0x5))
0000 - 14 03 03 00 01                                    .....
read from 0x7f884245f650 [0x7f8842815e08] (1 bytes => 1 (0x1))
0000 - 01                                                .
read from 0x7f884245f650 [0x7f8842815e03] (5 bytes => 5 (0x5))
0000 - 16 03 03                                          ...
0005 - <SPACES/NULS>
read from 0x7f884245f650 [0x7f8842815e08] (32 bytes => 32 (0x20))
0000 - 69 41 e0 7a 5e 5d f4 80-af f7 70 e9 3b 12 ab 22   iA.z^]....p.;.."
0010 - 60 08 56 fb 5c 4a b4 f1-3e 8d 63 c9 38 88 90 e0   `.V.\J..>.c.8...
SSL_connect:SSLv3 read finished A
---
Certificate chain
 0 s:/C=OURCOUNTRY/ST=OURCITY/L=OURPROVINCE/O=OURCOMPANY/CN=b2b.ourservice.ourdomain
   i:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2012 Entrust, Inc. - for authorized use only/CN=Entrust Certification Authority - L1K
 1 s:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2012 Entrust, Inc. - for authorized use only/CN=Entrust Certification Authority - L1K
   i:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2009 Entrust, Inc. - for authorized use only/CN=Entrust Root Certification Authority - G2
 2 s:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2009 Entrust, Inc. - for authorized use only/CN=Entrust Root Certification Authority - G2
   i:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2009 Entrust, Inc. - for authorized use only/CN=Entrust Root Certification Authority - G2
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIIKjCCBxKgAwIBAgIQOP5SIrZ9PT0AAAAAUOMHezANBgkqhkiG9w0BAQsFADCB
ujELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUVudHJ1c3QsIEluYy4xKDAmBgNVBAsT
H1NlZSB3d3cuZW50cnVzdC5uZXQvbGVnYWwtdGVybXMxOTA3BgNVBAsTMChjKSAy
MDEyIEVudHJ1c3QsIEluYy4gLSBmb3IgYXV0aG9yaXplZCB1c2Ugb25seTEuMCwG
A1UEAxMlRW50cnVzdCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eSAtIEwxSzAeFw0x
dIAga7lZQ3Cmo8PMwWQek59kyYavIap7chrmElKtpU4D4SBZg8c4/apb7s/39sff
bFhAdYGpflhUI9jEsIJKm8TG2OVWgtmyFsP8IUBt
-----END CERTIFICATE-----
subject=/C=OURCOUNTRY/ST=OURCITY/L=OURPROVINCE/O=OURCOMPANY/CN=b2b.ourservice.ourdomain
issuer=/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2012 Entrust, Inc. - for authorized use only/CN=Entrust Certification Authority - L1K
---
Acceptable client certificate CA names
/C=OURCOUNTRY/CN=b2b.ourservice.ourdomain-client_ca.dev.ourd/L=OURCITY/ST=IT
Server Temp Key: ECDH, X25519, 253 bits
---
SSL handshake has read 7094 bytes and written 2389 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-CHACHA20-POLY1305
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-CHACHA20-POLY1305
    Session-ID: 7F2A4D6F1C37B1F0F9BB9D667E097162B41D429CDCE5F4AF807957E6468EA339
    Session-ID-ctx: 
    Master-Key: CE164C555337F0E5E79358B1B2B6B49FB8A255BCF48724922CE14035E2DFD8B78297A25146550CF9C99858514BD846A3
    TLS session ticket lifetime hint: 600 (seconds)
    TLS session ticket:
    0000 - e0 e1 66 06 35 97 74 1e-7c e8 9d 54 4b 5f 60 ca   ..f.5.t.|..TK_`.
    0010 - ea cf 85 b0 8c 4b 1e 55-12 6c 18 6e 15 cc b6 89   .....K.U.l.n....

    06a0 - d5 ee 08 54 5b 9d fd cd-29 14 f9 dd b9 0c dd b1   ...T[...).......
    06b0 - 6e 89 3f eb cb ae 44 3c-dc de 92 e9 09 73 bf 86   n.?...D<.....s..
    06c0 - 15 89 ea 01 4d bb fd c3-2f 6c 6d b8 82 b5 1f 9d   ....M.../lm.....

    Start Time: 1589377212
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---
read from 0x7f884245f650 [0x7f8842815e03] (5 bytes => 5 (0x5))
0000 - 15 03 03 00 12                                    .....
read from 0x7f884245f650 [0x7f8842815e08] (18 bytes => 18 (0x12))
0000 - d6 ae a0 e5 98 c2 8d d6-11 19 cd 3e 37 7d c9 d1   ...........>7}..
0010 - de 2e                                             ..
SSL3 alert read:warning:close notify
closed
write to 0x7f884245f650 [0x7f884281a003] (23 bytes => 23 (0x17))
0000 - 15 03 03 00 12 80 8e fd-de 24 e2 61 7a fb 79 ba   .........$.az.y.
0010 - 53 0f fd 51 62 a2 96                              S..Qb..
SSL3 alert write:warning:close notify

From this, we see we are hitting SSL3 alert read:warning:close notify in both cases but we expect the first request to fail due to client certificate setting in nginx. Whis is this happening and how can we fix it ?

lsambo
  • 300
  • 3
  • 21
  • 1
    nginx does not abort the TLS handshake (as some servers do) for a missing or invalid client cert; instead it rejects the (first) request on that connection with an _HTTP_ pseudo-error 496 or 495 (normally mapped to 400 on output). Try sending e.g. `GET / HTTP/1.1(CRLF)Host: whatever(CRLF)(CRLF)` from your first `s_client` . See http://nginx.org/en/docs/http/ngx_http_ssl_module.html#errors . – dave_thompson_085 May 13 '20 at 16:32
  • Thanks dave. Tried `echo -e "GET / HTTP/1.1(CLRF)Host: b2b.ourservice.ourdomain:443(CRLF)(CRLF)" | openssl s_client -ign_eof -connect b2b.ourservice.ourdomain:443 -state -debug -servername b2b.ourservice.ourdomain.it -cert b2b.ourclientca.pem -key b2b.ourclient.key -crlf` and now we get ```HTTP/1.1 400 Bad Request Date: Thu, 14 May 2020 07:25:40 GMT Content-Type: text/html Content-Length: 150 Connection: close 400 Bad Request

    400 Bad Request


    nginx
    ```Can we differentiate them?
    – lsambo May 14 '20 at 07:29
  • To make this up to date, we are trying to debug more from the nginx server side on this specific request. – lsambo May 15 '20 at 16:01
  • Luigi: I didn't mean the actual characters `(CRLF)`, I meant the (Internet-standard) linebreak consisting of a Carriage Return byte and a Line Feed byte. If your `echo` supports `-e` (not all do), use backslash r backslash n in the data, within quotes so shell doesn't remove them, and DON'T use `-crlf` on `s_client` (which adds ANOTHER CR), and also use `-n` on `echo` to prevent an extra LF (without CR) added at the end. Also I think you can configure a custom error-page for 496 (and maybe 495 also). – dave_thompson_085 May 16 '20 at 01:44
  • For _much_ more detail on manually sending HTTPS using s_client, see my recent answer at https://crypto.stackexchange.com/questions/80420/debug-client-certificate-authentication-with-openssl – dave_thompson_085 May 18 '20 at 16:13

0 Answers0