TLS 1.2 handhsake failing with Handshake failure error

Question

I am trying to establish a TLSV1.2 connection to a Java-based MQTT broker (Active MQ).
My client has the private key in an HSM module and hence inaccessible to me.
x509 cert is available.
CA is self-signed.

Cert details:
Signature Algorithm: ecdsa-with-SHA256
Public Key Algorithm: id-ecPublicKey (256bit)
curve: prime256v1

Behavior observed:

The server is throwing the following error:

43 2022-04-01 23:22:26.772084 serverip clientip TLSv1.2 73 Alert (Level: Fatal, Description: Handshake Failure)

This is happening right after Client key exchange

41 2022-04-01 23:22:26.739193 clientip serverip TLSv1.2 303 Client Key Exchange, Certificate Verify, Change Cipher Spec, Encrypted Handshake Message

What I have tried so far:

I tried establishing a connection through curl to the same broker using an x509 cert signed by the same CA. This is being accepted. Hence I have concluded this is not a bad certificate issue.

Note: I am using --insecure in the curl command, I am okay with this as client side auth is something I am not concerned about, given that server is throwing an error.

Observations/Assumptions:

The handshake failure is coming from the server after:

Client Hello  
Server Hello  
Server Hello, Certificate, Server Key Exchange, Certificate Request, ServerHelloDone
Certificate
Client Key Exchange, Certificate Verify, Change Cipher Spec, Encrypted Handshake Message  
Alert (Level: Fatal, Description: Handshake Failure) ( from server)

By this I am concluding there is no issue with the cipher suite /protocol compatibility.

The only difference I see between the X509 cert generated by HSM and the cert I generated for verification is in the common name format.
Wireshark says:

HSM generated cert CN is DirectoryString: teletexString (0)
Manually generated cert CN is utf-8
The CN is the same, the format as shown by wireshark differs.

I have never looked at a handshake so in depth and know just basics of a TLS handshake.

The server is a managed Kubernetes deployment and hence really difficult currently to associate logs with clients.

But, based on observation , I see this message (activemq-netty-threads)","message":"AMQ222208: SSL handshake failed for client. java.io.IOException: Sequence tag error."

I think this is the corresponding broker logs.
Any way forward/ opinions appreciated.

Results of parsing x509 certs

HSM cert CN=A_00000066,OU=AA,O=Organisation Limited,L=Place,ST=State,C=C
FS certs

1.2.840.113549.1.9.1=#160a726f6f74406174686572,CN=A_00000066,OU=AA,O=O,L=Place,ST=State,C=C

Adding relevant Server logs

SSL logs on the server:

"throwable" : {
  java.security.SignatureException: Invalid encoding for signature
    at java.base/sun.security.util.ECUtil.decodeSignature(ECUtil.java:279)
    at jdk.crypto.ec/sun.security.ec.ECDSASignature.engineVerify(ECDSASignature.java:477)
    at java.base/java.security.Signature$Delegate.engineVerify(Signature.java:1247)
    at java.base/java.security.Signature.verify(Signature.java:675)
    at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyMessage.<init>(CertificateVerify.java:651)
    at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyConsumer.consume(CertificateVerify.java:771)
    at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
    at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074)
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:689)
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)
    at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1542)
    at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1556)
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1440)
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267)
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
  Caused by: java.io.IOException: Sequence tag error
    at java.base/sun.security.util.DerInputStream.getSequence(DerInputStream.java:336)
    at java.base/sun.security.util.ECUtil.decodeSignature(ECUtil.java:255)
    ... 32 more}

)
javax.net.ssl|WARNING|32|Thread-4 (activemq-netty-threads)|2022-04-06 10:01:32.559 UTC|SSLEngineOutputRecord.java:168|outbound has closed, ignore outbound application data

Update 2:

The signing is handed over to the rustls library. There is a custom function that uses the HSM private key to sign the digest as required.

example signature: 0349003046022100838bde8a902f9ebb18cdd9bc5af263dc978a670d95770c11e2e8d29e3c7b2c28022100d345fa7245fb34c8cf710958da80a638c598c44e2cbd724571dfd9e9ade95008

Actual Client encrypted handshake message after which failure happens:

Consuming ECDHE ClientKeyExchange handshake message (
"ECDH ClientKeyExchange": {
  "ecdh public": {
    0000: 04 EB B8 76 96 C5 E0 C6   20 73 F0 4C AB 93 F1 A6  ...v.... s.L....
    0010: E9 6C 64 B0 BB 72 64 A4   74 75 26 4B E2 79 C0 26  .ld..rd.tu&K.y.&
    0020: 42 C8 C8 8F D4 C5 CA EC   22 DA B5 3B 03 E8 E8 19  B......."..;....
    0030: 28 28 EF C6 9D EE 80 3A   CD A1 60 2B 62 83 52 8F  ((.....:..`+b.R.
    0040: 23 B4 5B 46 1F 76 86 00   0D DF F3 1E 6B 86 01 A4  #.[F.v......k...
    0050: 64 09 C9 80 0A 03 C6 EE   A4 AA 36 05 F4 45 7A 91  d.........6..Ez.
    0060: A5                                                 .
  },
}

Update 2: Updated stack trace

javax.net.ssl|ERROR|41|Thread-16 (activemq-netty-threads)|2022-04-11 10:39:02.983 UTC|TransportContext.java:312|Fatal (HANDSHAKE_FAILURE): Invalid CertificateVerify signature (
"throwable" : {
  javax.net.ssl.SSLHandshakeException: Invalid CertificateVerify signature
      at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
      at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
      at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:307)
      at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263)
      at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:254)
      at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyMessage.<init>(CertificateVerify.java:978)
      at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyConsumer.consume(CertificateVerify.java:1125)
      at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
      at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
      at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074)
      at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
      at java.base/java.security.AccessController.doPrivileged(AccessController.java:689)
      at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)
      at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1542)
      at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1556)
      at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1440)
      at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267)
      at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314)
      at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
      at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
      at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
      at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
      at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
      at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
      at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
      at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
      at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
      at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
      at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)}

Have you tried adding the Public Key to your java truststore ? — Sai Prasad Sabeson, Apr 02 '22 at 19:38
@SaiPrasadSabeson yes it is added. Also curl is going through. If the issues was an invalid cert , curl would also fail. — Abhilash Gopalakrishna, Apr 02 '22 at 19:59
Have you tried your setup against a local instance of ActiveMQ Artemis that would be simpler to configure and inspect? — Justin Bertram, Apr 02 '22 at 20:10
Can you temporarily turn on SSL logging on the server side? That will give you the server's reasons for breaking off the negotiation. See https://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/ReadDebug.html — Stephen C, Apr 03 '22 at 01:07
"Sequence tag error" speaks to an ASN.1 parsing error, so it could be worth trying to parse the HSM certificate in a standalone Java program. BTW do you have the full stack trace for that IOException? — Peter Dettman, Apr 03 '22 at 07:14
@PeterDettman unfortunately no, but I will try to get all the above logs and update the question — Abhilash Gopalakrishna, Apr 03 '22 at 07:43
@PeterDettman I tried parsing the x509 certs through a standalone java app, have updated the results in the question if it helps. — Abhilash Gopalakrishna, Apr 03 '22 at 09:03
Your 'parsing' app shows only the subject name. That's not the cert, that's a small fraction of it -- probably less than a tenth, maybe less than a twentieth. Get at least the suspect (HSM) cert in a file if it isn't already and try `keytool -printcert -file $filename` which will display _all_ the fixed fields and at least most extensions. However (@PeterDettman) this might be instead the encoding of the protocol signature in the CertVerify message, which should be DER (a sequence of two integers) but an HSM might do as P1363; the javax.net.debug trace would show that (as would the stack) — dave_thompson_085, Apr 03 '22 at 09:47
@SaiPrasadSabeson There is no such thing as adding a public key to a truststore. Do you mean adding the *certificate?* — user207421, Apr 03 '22 at 09:52
@Robert using Java 11 on server side. Client is not Java, it's a rust client. — Abhilash Gopalakrishna, Apr 03 '22 at 10:43
@dave_thompson_085 got it, I have the HSM cert in a file and able to see the cert as well. The public key is ecdsa sha256. The CN appears same when I use openssl to display the cert, but only wireshark shows the teletextstring and utf-8 encoding difference. — Abhilash Gopalakrishna, Apr 03 '22 at 10:44
I think @dave_thompson_085 may be onto something; what is the rust client software (rustls?) and what code are you using to generate the client authentication signature from your HSM? Also, is this done in a callback from the TLS software somehow? - give details, please. — Peter Dettman, Apr 03 '22 at 11:21
@dave_thompson_085 your hunch looks to be right, have added detailed server logs, I am not really sure what this implies, I understand I am using ecdsa , but not sure what the sequence tag error means. — Abhilash Gopalakrishna, Apr 06 '22 at 21:44

score 3 · Accepted Answer · answered Apr 07 '22 at 04:29

3

The signing is handed over to the rustls library. There is a custom function that uses the HSM private key to sign the digest as required.
example signature: 0349003046022100838bde8a902f9ebb18cdd9bc5af263dc978a670d95770c11e2e8d29e3c7b2c28022100d345fa7245fb34c8cf710958da80a638c598c44e2cbd724571dfd9e9ade95008

Assuming you are displaying hex for data that is actually binary, that is almost correct. Dehexed it is actually a valid (or at least validly formatted) Ecdsa-Sig for a 256-bit curve such as P-256, embedded in a (ASN.1) BITSTRING. Specifically 03 49 00 is the tag and length for a BITSTRING containing 72 bytes with no unused/extra bits. Those 72 bytes consist of 30 46 02 21 (33bytes) 02 21 (33bytes) which is the correct encoding of: a constructed SEQUENCE (tag 0x10 plus 0x20) with value length 70 bytes, containing two primitive INTEGER items (tag 0x02) each with value length 33 bytes.

Whatever code is building the CertificateVerify message should remove the first 3 bytes and use the rest. It needs to handle varying lengths; at most the DER signature will be 72 bytes as in this example, but often it will be 71 or 70 bytes instead, and occasionally but rarely even less. Compare openssl rust crate: ECDSA signature size is not 64 bytes? and (as linked there) how to specify signature length for java.security.Signature sign method also (my) Java's BouncyCastle doesn't always verify OpenSSL ECDSA signature .

(PS: the ClientKeyExchange message is irrelevant here. Client signature is always in the CertVerify message. For server below TLS1.3, the signature does go in ServerKeyExchange, but that's not relevant to your case.)

answered Apr 07 '22 at 04:29

dave_thompson_085

34,712
6
50
70

thank you for the detailed references and wonderful explanation, really helped a lot. What I understand is, the signature is failing as its 71 bytes and not 72, which is enforced to stricter der encoding rules. Also the first 3 bytes indicate/explain the signature itself and hence not needed in Certificate Verify message. Even after removing the first 3 bytes I am getting an invalid error, now the attempt is to generate a 72 byte long Sig. Correct me if I'm wrong.Also can you point me to some references that explains encoding during exchange in detail. Thanks. – Abhilash Gopalakrishna Apr 10 '22 at 07:11
No, 71 bytes is not wrong and it shouldn't be failing for that reason, and you shouldn't try to change it to 72 somehow. As I said and the links explain, these (ECDSA-256bit DER) signatures **vary** in length. In fact more of them are 71 bytes than 72 bytes (roughly 50% vs 25%). Are you still getting an error about the _encoding_ or _format_ of the signature, or about its _value_? Those are completely different problems. ... – dave_thompson_085 Apr 11 '22 at 15:27
1

... The signature generation for 1.2 CertVerify is in RFC5246 7.4.8, with the general encoding for all 1.2 signatures referenced in in 4.7 'digitally-signed' and the ECDSA-specific encoding in RFC4492 5.8 which refers to DER in X.690, however you may find easier the [wikipedia article on BER/DER/CER](https://en.wikipedia.org/wiki/X.690) or the [LetsEncrypt introduction to DER](https://letsencrypt.org/docs/a-warm-welcome-to-asn1-and-der/). – dave_thompson_085 Apr 11 '22 at 15:31
yes I am still getting an invalid certificate verify signature. No encoding errors. I was under the impression this is because of stricter encoding standards enforced where anything <72 bytes is not accepted. Which I now understand is wrong. The encoding is right I belive but still getting an invalid signature. Stack trace does not really help though. I have added the new modified stack trace to the question. – Abhilash Gopalakrishna Apr 11 '22 at 16:36
1

The new exception is vaguer, but _most likely_ indicates wrong/mismatched value rather than any encoding issue. This could be an error in the client stack determining the data to be signed, or in the client HSM doing the signature (probably least likely, as it will probably have been tested), or use of a key that doesn't match the cert, or conceivably different/wrong data (covered by the signature) could be passed to the server. I don't know anything about rust so I can't help with it, but if I were in this situation I would try to get the rust client to use a software key instead.... – dave_thompson_085 Apr 12 '22 at 04:50
1

... And/or maybe try to get the HSM to do RSA instead; that can provide some debugging data ECDSA can't (the padded but not modexped hash). Good luck. – dave_thompson_085 Apr 12 '22 at 04:52

TLS 1.2 handhsake failing with Handshake failure error

1 Answers1