Tls 1.3 client does not report failed handshake when client certificate verification by server failed

Question

I have a C client using OpenSSL that is failing a test when using a certificate that fails validation on the server side during the SSL_do_handshake() call on the server. When the application was using TLS 1.2 The SSL_do_handshake() failure on the server would be reported back to the client when it called SSL_do_handshake() as a failure return value.

When upgrading my application to OpenSSL 1.1.1 and TLS 1.3 I noted that while the validation error is still occurring on the server, it was no longer being reported back to the client.

I'm aware that the handshake protocol got completely re-written as part of TLS 1.3 however it seems like with all of the various callbacks available I should be able somehow on the client side to determine that authentication has failed without having to attempt to write data to the server.

Has anyone else encountered this and can they recommend a path forward?

score 6 · Answer 1 · answered Jun 19 '20 at 08:01

The server and client in both TLSv1.2 and TLSv1.3 consider the handshake to be complete when they have both written a "Finished" message, and received one from the peer. This is what the handshake looks like in TLSv1.2 (taken from RFC5246):

      Client                                               Server

      ClientHello                  -------->
                                                      ServerHello
                                                     Certificate*
                                               ServerKeyExchange*
                                              CertificateRequest*
                                   <--------      ServerHelloDone
      Certificate*
      ClientKeyExchange
      CertificateVerify*
      [ChangeCipherSpec]
      Finished                     -------->
                                               [ChangeCipherSpec]
                                   <--------             Finished
      Application Data             <------->     Application Data

So here you can see that the client sends its Certificate and Finished messages in its second flight of communication with the server. It then waits to receive the ChangeCipherSpec and Finished messages back from the server before it considers the handshake "complete" and it can start sending application data.

This is the equivalent flow for TLSv1.3 taken from RFC8446:

       Client                                           Server

Key  ^ ClientHello
Exch | + key_share*
     | + signature_algorithms*
     | + psk_key_exchange_modes*
     v + pre_shared_key*       -------->
                                                  ServerHello  ^ Key
                                                 + key_share*  | Exch
                                            + pre_shared_key*  v
                                        {EncryptedExtensions}  ^  Server
                                        {CertificateRequest*}  v  Params
                                               {Certificate*}  ^
                                         {CertificateVerify*}  | Auth
                                                   {Finished}  v
                               <--------  [Application Data*]
     ^ {Certificate*}
Auth | {CertificateVerify*}
     v {Finished}              -------->
       [Application Data]      <------->  [Application Data]

One of the advantages of TLSv1.3 is that it speeds up the time taken to complete a handshake. In TLSv1.3 the client receives the "Finished" message from the server before it sends its Certificate and Finished messages back. By the time the client sends its "Finished" message, it has already received the "Finished" and so the handshake has completed and it can immediately start sending application data.

This of course means that the client won't know whether the server has accepted the certificate or not until it next reads data from the server. If it has been rejected then the next thing the client will read will be a failure alert (otherwise it will be normal application data).

I'm aware that the handshake protocol got completely re-written as part of TLS 1.3 however it seems like with all of the various callbacks available I should be able somehow on the client side to determine that authentication has failed without having to attempt to write data to the server.

It's not writing data to the server that is important - it is reading data. Only then will you know whether the server has sent an alert or just normal application data. Until that data has been read there are no callbacks available in OpenSSL that will tell you this - because OpenSSL itself does not know due to the underlying protocol.

Thank you for the explanation. That makes sense. As I noted previously my client code was relying just on the function SSL_do_handshake() returning an error to detect if the server had any certificate error. I have added this additional check in the client: `char a; if (recv(&a, 1, MSG_PEEK) == 0) return false; }` However even though I know the server has indicated a handshake error, I see that recv() call in the client succeed where given what you have described I would expect it to fail. — David Ritter, Jun 19 '20 at 15:09
If the server fails it sends an alert to the client. This is a TLS level error message. recv() just works at the TCP level so it will successfully see the arrival of "something". It is up to OpenSSL to interpret it, decide it is an error alert and only then will the client also fail. — Matt Caswell, Jun 19 '20 at 16:22
I switched from `recv()` to `SSL_read()`, it also indicates a successful read. I apologize for being dense, how should I be checking with OpenSSL to interpret what came back and determine if it is an error alert? — David Ritter, Jun 19 '20 at 16:39
If you have received an alert then SSL_read() should return <=0. You are then supposed to use SSL_get_error() to determine whether the error is fatal or not (an alert will give an SSL_ERROR_SSL result). If SSL_read() is successful then you have successfully managed to read application data from the server (that's what SSL_read() does). If its sent you app data then the server hasn't failed. — Matt Caswell, Jun 19 '20 at 18:12
Oh okay. Sorry I knew about `SSL_get_error()` I wasn't seeing any error returned so I figured there must have been some other part of the API I missed. I'll have to go over my code again and see why the server `SSL_do_handshake()` failure isn't triggering a read error on the client side. Thank you for your help. — David Ritter, Jun 19 '20 at 18:21

Tls 1.3 client does not report failed handshake when client certificate verification by server failed

1 Answers1