2

I am working with IBM on trying to find the cause for some handshake errors we are receiving randomly when connecting with certain endpoints. This is happening on an IBM i system using the GSKit SSL APIs. The IBM is acting as the client.

The error we are receiving during the handshake is 415 (Bad Peer).

In the trace we show this response from the remote server:

TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Illegal Parameter)
    Content Type: Alert (21)
    Version: TLS 1.2 (0x0303)
    Length: 2
    Alert Message
        Level: Fatal (2)
        Description: Illegal Parameter (47)

If we remove TLS 1.3 from the communications, the errors do not occur. If we add it back in, then the errors pop up communicating with Google and GoDaddy.

This only seemed to start happening recently after new SSL Certificates were installed at the endpoints. And, as I said, it's random. If I encounter the error, I can make the same request again without any error.

We weren't sure if it was because of how certain servers on their farms are configured differently from others, or if updates are propagating through them... but it's very odd.

We have isolated 2 ciphers that seem to be the issue. When removed, the errors seem to stop. They are:

*ECDHE_RSA_AES_256_GCM_SHA384 - this one seems to cause problems with Google

*AES_256_GCM_SHA384 - this one seems to cause problems with GoDaddy

The tech at IBM said that if we could somehow get in touch with someone at either place and recreate the issue, it would be a great help to figure out what is going on. Just hoping we can make that happen... if anyone knows anyone on the SSL support side of GoDaddy or Google that would be willing to work with myself and IBM to resolve this, it would be great.

Or, if anyone knows why it's happening, we're all ears. Right now the only option we have is turning off TLS v1.3 or removing these two ciphers. (and there may be more that affect other endpoints that we just don't know about yet).

TIA!

UPDATE

According to my tech, they are not following the RFC properly:

RFC 5246 (the last sentence):

The cipher suite list, passed from the client to the server in the
   ClientHello message, contains the combinations of cryptographic
   algorithms supported by the client in order of the client's
   preference (favorite choice first).  Each cipher suite defines a key
   exchange algorithm, a bulk encryption algorithm (including secret key
   length), a MAC algorithm, and a PRF.  The server will select a cipher
   suite or, if no acceptable choices are presented, return a handshake
   failure alert and close the connection.  If the list contains cipher
   suites the server does not recognize, support, or wish to use, the
   server MUST ignore those cipher suites, and process the remaining
   ones as usual.

UPDATE 2 Here is a handshake that fails: TLSv1.2 Record Layer: Handshake Protocol: Client Hello

Content Type: Handshake (22)

Version: TLS 1.2 (0x0303)

Length: 549

Handshake Protocol: Client Hello

    Handshake Type: Client Hello (1)

    Length: 545

    Version: TLS 1.2 (0x0303)

    Random: 16ab36ccafcb18074cf7ce0296745cb87a4ef732402fbdc273790f082d4844d1

    Session ID Length: 0

    Cipher Suites Length: 18

    Cipher Suites (9 suites)

        Cipher Suite: TLS_AES_128_GCM_SHA256 (0x1301)

        Cipher Suite: TLS_AES_256_GCM_SHA384 (0x1302)

        Cipher Suite: TLS_CHACHA20_POLY1305_SHA256 (0x1303)

        Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)

        Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)

        Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)

        Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)

        Cipher Suite: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca9)

        Cipher Suite: TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca8)

    Compression Methods Length: 1

    Compression Methods (1 method)

    Extensions Length: 486

    Extension: supported_versions (len=9)

        Type: supported_versions (43)

        Length: 9

        Supported Versions length: 8

        Supported Version: TLS 1.3 (0x0304)

        Supported Version: TLS 1.2 (0x0303)

        Supported Version: TLS 1.1 (0x0302)

        Supported Version: TLS 1.0 (0x0301)

    Extension: renegotiation_info (len=1)

        Type: renegotiation_info (65281)

        Length: 1

        Renegotiation Info extension

    Extension: ec_point_formats (len=2)

        Type: ec_point_formats (11)

        Length: 2

        EC point formats Length: 1

        Elliptic curves point formats (1)

            EC point format: uncompressed (0)

    Extension: signature_algorithms (len=22)

        Type: signature_algorithms (13)

        Length: 22

        Signature Hash Algorithms Length: 20

        Signature Hash Algorithms (10 algorithms)

            Signature Algorithm: ecdsa_secp521r1_sha512 (0x0603)

            Signature Algorithm: ecdsa_secp384r1_sha384 (0x0503)

            Signature Algorithm: ecdsa_secp256r1_sha256 (0x0403)

            Signature Algorithm: rsa_pss_rsae_sha512 (0x0806)

            Signature Algorithm: rsa_pss_rsae_sha384 (0x0805)

            Signature Algorithm: rsa_pss_rsae_sha256 (0x0804)

            Signature Algorithm: rsa_pkcs1_sha512 (0x0601)

            Signature Algorithm: rsa_pkcs1_sha384 (0x0501)

            Signature Algorithm: rsa_pkcs1_sha256 (0x0401)

            Signature Algorithm: rsa_pkcs1_sha1 (0x0201)

    Extension: signature_algorithms_cert (len=20)

        Type: signature_algorithms_cert (50)

        Length: 20

        Signature Hash Algorithms Length: 18

        Signature Hash Algorithms (9 algorithms)

            Signature Algorithm: ecdsa_secp521r1_sha512 (0x0603)

            Signature Algorithm: ecdsa_secp384r1_sha384 (0x0503)

            Signature Algorithm: ecdsa_secp256r1_sha256 (0x0403)

            Signature Algorithm: rsa_pss_rsae_sha512 (0x0806)

            Signature Algorithm: rsa_pss_rsae_sha384 (0x0805)

            Signature Algorithm: rsa_pss_rsae_sha256 (0x0804)

            Signature Algorithm: rsa_pkcs1_sha512 (0x0601)

            Signature Algorithm: rsa_pkcs1_sha384 (0x0501)

            Signature Algorithm: rsa_pkcs1_sha256 (0x0401)

    Extension: supported_groups (len=12)

        Type: supported_groups (10)

        Length: 12

        Supported Groups List Length: 10

        Supported Groups (5 groups)

            Supported Group: secp256r1 (0x0017)

            Supported Group: secp384r1 (0x0018)

            Supported Group: x25519 (0x001d)

            Supported Group: secp521r1 (0x0019)

            Supported Group: x448 (0x001e)

    Extension: key_share (len=71)

        Type: key_share (51)

        Length: 71

        Key Share extension

            Client Key Share Length: 69

            Key Share Entry: Group: secp256r1, Key Exchange length: 65

                Group: secp256r1 (23)

                Key Exchange Length: 65

                Key Exchange: 047fbc26145d5067052eff17103285a7bdc30952cdbcea601491a6a08eca7d424484a9cb…

    Extension: server_name (len=20)

        Type: server_name (0)

        Length: 20

        Server Name Indication extension

    Extension: extended_master_secret (len=0)

        Type: extended_master_secret (23)

        Length: 0

    Extension: psk_key_exchange_modes (len=2)

        Type: psk_key_exchange_modes (45)

        Length: 2

        PSK Key Exchange Modes Length: 1

        PSK Key Exchange Mode: PSK with (EC)DHE key establishment (psk_dhe_ke) (1)

    Extension: pre_shared_key (len=283)

        Type: pre_shared_key (41)

        Length: 283

        Pre-Shared Key extension

            Identities Length: 230

            PSK Identity (length: 224)

                Identity Length: 224

                Identity: 000017719425bdded0ac21c8cd61318334f131527b4ef48f21ba8116523cf71681ae51fa…

                Obfuscated Ticket Age: 68965835

            PSK Binders length: 49

            PSK Binders

The server then responds with

TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Illegal Parameter)

    Content Type: Alert (21)

    Version: TLS 1.2 (0x0303)

    Length: 2

    Alert Message

        Level: Fatal (2)

        Description: Illegal Parameter (47)
bvstone
  • 587
  • 2
  • 6
  • 17
  • This is a complex issue that needs further assistance. To follow up you can [create an issue](https://developers.google.com/issue-tracker/guides/create-issue-ui#create_an_issue), or reach the Support Team through [paid support](https://cloud.google.com/support). – Rogelio Monter Nov 23 '21 at 22:36

2 Answers2

0

IBM released an APAR with PTF for V7R3 and V7R4. The APAR number is MA49492

Regards Michael

  • This does not provide an answer to the question. Once you have sufficient [reputation](https://stackoverflow.com/help/whats-reputation) you will be able to [comment on any post](https://stackoverflow.com/help/privileges/comment); instead, [provide answers that don't require clarification from the asker](https://meta.stackexchange.com/questions/214173/why-do-i-need-50-reputation-to-comment-what-can-i-do-instead). - [From Review](/review/late-answers/31243158) – MD. RAKIB HASAN Mar 14 '22 at 09:10
-1

You can review encryption protocols that Google Front End supports when communicating with clients in this whitepaper.

In this whitepaper, you will find more detail on encryption in transit for Google Cloud, including Google Cloud Platform and Google Workspace.

From table 1, you can check that the cipher ECDHE_RSA_AES_256_GCM_SHA384, which is having trouble connecting with Google, is not listed as compatible.

I have found this documentation from IBM that lists the supported CipherSpecs (I’m not quite sure if this applies to your IBM product). Also, there’s a troubleshooting TLS guide from IBM.

It is important to understand this information is not a comprehensive source for troubleshooting information, but rather a guide to aid in common problem resolution.

UPDATE

It's unclear if IBM's GSKit SSL APIs are generating the error, or if it is allowing the user to send invalid packets and then reporting the error.

Unsupported TLS 1.2 ciphers would be ignored in the handshake as stated here, but I believe there is just 5 ciphers in TLS 1.3 (see RFC 8446 - Cipher Suites and OpenSSL - Ciphersuites), and ECDHE_RSA_AES_256_GCM_SHA384 is not in them, so it would be strange if IBM's product allowed an invalid cipher to be requested.

The 1.3 cipher specs work fine with Google servers, like:

echo | openssl s_client -tls1_3 -connect SERVER:443 -ciphersuites TLS_CHACHA20_POLY1305_SHA256 | grep "New, TLSv1.3, Cipher"

As mentioned above, ECDHE_RSA_AES_256_GCM_SHA384 is not in the list of TLS 1.2 cipher specifications.

UPDATE 2

From the whitepaper, I noted:

TLS in the GFE is implemented with BoringSSL. Table 1 shows the encryption protocols that GFE supports when communicating with clients.

Protocols Authentication Key exchange Encryption Hash Functions
TLS 1.34 RSA 2048 Curve25519 AES-128-GCM SHA384
TLS 1.2 ECDSA P-256 P-256 (NIST secp256r1) AES-256-GCM SHA256

The supported values above don't match with the Cipher Suite list provided shown on the handshake packet capture, note the different SHA values:

TLS_AES_128_GCM_SHA256 (0x1301) 
TLS_AES_256_GCM_SHA384 (0x1302)

The values used are AES128 with SHA256, and AES256 with SHA384 but BoringSSL supports AES128 with SHA384 for TLS 1.3 and AES256 with SHA256 for TLS 1.2.

Looking at some more IBM docs for z/OS, I'm seeing a discrepancy with 0x1302 which showcases AES256 with SHA256. For TLS V1.3, the following is allowed in the evaluated configurations:

Ciphers:
1301 - TLS_AES_128_GCM_SHA256
1302 - TLS_AES_256_GCM_SHA256

How are the SSL certificates used? Is it possible to share the link for those docs?

Throubleshoot

I would recommend to try the following:

  1. Find out what is causing the randomness. Maybe there could be some propagation delay on your IBM system or the SSL distribution management.
  2. Try changing 1302 from TLS_AES_256_GCM_SHA384 to TLS_AES_256_GCM_SHA256 and verify if matching z/OS docs works
  3. Try changing 1301 from TLS_AES_128_GCM_SHA256 to TLS_AES_128_GCM_SHA384 and verify if matching BoringSSL works
Rogelio Monter
  • 1,084
  • 7
  • 18
  • Thanks, Rogelio. Why would it cause an issue, then. Wouldn't Google just ignore that cipher and use another from the list? Why throw an error? – bvstone Oct 18 '21 at 12:40
  • @bvstone I've made an update on the answer – Rogelio Monter Oct 20 '21 at 22:34
  • updated original question with what seems to be the issue, ie the server isn't following the RFC properly. – bvstone Oct 21 '21 at 16:43
  • Could you please share a packet capture to compare "OK" vs "NG" scenarios? To compare which ciphers are being sent in each one – Rogelio Monter Oct 28 '21 at 18:41
  • Added capture for the error. As stated in original post, certain ciphers seem to trigger errors for certain endpoints. *ECDHE_RSA_AES_256_GCM_SHA384 - this one seems to cause problems with Google *AES_256_GCM_SHA384 - this one seems to cause problems with GoDaddy. – bvstone Oct 29 '21 at 14:43
  • The support I am working with at IBM seems to think that they are not following the RFC posted. They find one cipher they don't like and just throw an error instead of ignoring it and using a different cipher. I think that's proven by removing the ciphers and testing again. IBM tends to follow RFCs to the letter (unlike others). – bvstone Oct 29 '21 at 14:45
  • Hi @bvstone I made an update on the answer from the packet capture you've shared and other observations – Rogelio Monter Nov 16 '21 at 23:51
  • Thanks @rogelio Monter. I am not sure you are quit understanding. I am the client in the negotiations. I get the error when the two ciphers are in the list. If I remove them, negotiation works fine. I doubt GFE only supports 2 ciphers in total (from your table), one for each TLS version. Otherwise when removing the two from my list (that seem to cause errors from the server side) the handshake would fail as well with no matching cipher. Google seems to see one of these "bad" ciphers and just kills the connection instead of using a different cipher from the list. – bvstone Nov 17 '21 at 14:27