I am working with IBM on trying to find the cause for some handshake errors we are receiving randomly when connecting with certain endpoints. This is happening on an IBM i system using the GSKit SSL APIs. The IBM is acting as the client.
The error we are receiving during the handshake is 415 (Bad Peer).
In the trace we show this response from the remote server:
TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Illegal Parameter)
Content Type: Alert (21)
Version: TLS 1.2 (0x0303)
Length: 2
Alert Message
Level: Fatal (2)
Description: Illegal Parameter (47)
If we remove TLS 1.3 from the communications, the errors do not occur. If we add it back in, then the errors pop up communicating with Google and GoDaddy.
This only seemed to start happening recently after new SSL Certificates were installed at the endpoints. And, as I said, it's random. If I encounter the error, I can make the same request again without any error.
We weren't sure if it was because of how certain servers on their farms are configured differently from others, or if updates are propagating through them... but it's very odd.
We have isolated 2 ciphers that seem to be the issue. When removed, the errors seem to stop. They are:
*ECDHE_RSA_AES_256_GCM_SHA384 - this one seems to cause problems with Google
*AES_256_GCM_SHA384 - this one seems to cause problems with GoDaddy
The tech at IBM said that if we could somehow get in touch with someone at either place and recreate the issue, it would be a great help to figure out what is going on. Just hoping we can make that happen... if anyone knows anyone on the SSL support side of GoDaddy or Google that would be willing to work with myself and IBM to resolve this, it would be great.
Or, if anyone knows why it's happening, we're all ears. Right now the only option we have is turning off TLS v1.3 or removing these two ciphers. (and there may be more that affect other endpoints that we just don't know about yet).
TIA!
UPDATE
According to my tech, they are not following the RFC properly:
RFC 5246 (the last sentence):
The cipher suite list, passed from the client to the server in the
ClientHello message, contains the combinations of cryptographic
algorithms supported by the client in order of the client's
preference (favorite choice first). Each cipher suite defines a key
exchange algorithm, a bulk encryption algorithm (including secret key
length), a MAC algorithm, and a PRF. The server will select a cipher
suite or, if no acceptable choices are presented, return a handshake
failure alert and close the connection. If the list contains cipher
suites the server does not recognize, support, or wish to use, the
server MUST ignore those cipher suites, and process the remaining
ones as usual.
UPDATE 2 Here is a handshake that fails: TLSv1.2 Record Layer: Handshake Protocol: Client Hello
Content Type: Handshake (22)
Version: TLS 1.2 (0x0303)
Length: 549
Handshake Protocol: Client Hello
Handshake Type: Client Hello (1)
Length: 545
Version: TLS 1.2 (0x0303)
Random: 16ab36ccafcb18074cf7ce0296745cb87a4ef732402fbdc273790f082d4844d1
Session ID Length: 0
Cipher Suites Length: 18
Cipher Suites (9 suites)
Cipher Suite: TLS_AES_128_GCM_SHA256 (0x1301)
Cipher Suite: TLS_AES_256_GCM_SHA384 (0x1302)
Cipher Suite: TLS_CHACHA20_POLY1305_SHA256 (0x1303)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca9)
Cipher Suite: TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca8)
Compression Methods Length: 1
Compression Methods (1 method)
Extensions Length: 486
Extension: supported_versions (len=9)
Type: supported_versions (43)
Length: 9
Supported Versions length: 8
Supported Version: TLS 1.3 (0x0304)
Supported Version: TLS 1.2 (0x0303)
Supported Version: TLS 1.1 (0x0302)
Supported Version: TLS 1.0 (0x0301)
Extension: renegotiation_info (len=1)
Type: renegotiation_info (65281)
Length: 1
Renegotiation Info extension
Extension: ec_point_formats (len=2)
Type: ec_point_formats (11)
Length: 2
EC point formats Length: 1
Elliptic curves point formats (1)
EC point format: uncompressed (0)
Extension: signature_algorithms (len=22)
Type: signature_algorithms (13)
Length: 22
Signature Hash Algorithms Length: 20
Signature Hash Algorithms (10 algorithms)
Signature Algorithm: ecdsa_secp521r1_sha512 (0x0603)
Signature Algorithm: ecdsa_secp384r1_sha384 (0x0503)
Signature Algorithm: ecdsa_secp256r1_sha256 (0x0403)
Signature Algorithm: rsa_pss_rsae_sha512 (0x0806)
Signature Algorithm: rsa_pss_rsae_sha384 (0x0805)
Signature Algorithm: rsa_pss_rsae_sha256 (0x0804)
Signature Algorithm: rsa_pkcs1_sha512 (0x0601)
Signature Algorithm: rsa_pkcs1_sha384 (0x0501)
Signature Algorithm: rsa_pkcs1_sha256 (0x0401)
Signature Algorithm: rsa_pkcs1_sha1 (0x0201)
Extension: signature_algorithms_cert (len=20)
Type: signature_algorithms_cert (50)
Length: 20
Signature Hash Algorithms Length: 18
Signature Hash Algorithms (9 algorithms)
Signature Algorithm: ecdsa_secp521r1_sha512 (0x0603)
Signature Algorithm: ecdsa_secp384r1_sha384 (0x0503)
Signature Algorithm: ecdsa_secp256r1_sha256 (0x0403)
Signature Algorithm: rsa_pss_rsae_sha512 (0x0806)
Signature Algorithm: rsa_pss_rsae_sha384 (0x0805)
Signature Algorithm: rsa_pss_rsae_sha256 (0x0804)
Signature Algorithm: rsa_pkcs1_sha512 (0x0601)
Signature Algorithm: rsa_pkcs1_sha384 (0x0501)
Signature Algorithm: rsa_pkcs1_sha256 (0x0401)
Extension: supported_groups (len=12)
Type: supported_groups (10)
Length: 12
Supported Groups List Length: 10
Supported Groups (5 groups)
Supported Group: secp256r1 (0x0017)
Supported Group: secp384r1 (0x0018)
Supported Group: x25519 (0x001d)
Supported Group: secp521r1 (0x0019)
Supported Group: x448 (0x001e)
Extension: key_share (len=71)
Type: key_share (51)
Length: 71
Key Share extension
Client Key Share Length: 69
Key Share Entry: Group: secp256r1, Key Exchange length: 65
Group: secp256r1 (23)
Key Exchange Length: 65
Key Exchange: 047fbc26145d5067052eff17103285a7bdc30952cdbcea601491a6a08eca7d424484a9cb…
Extension: server_name (len=20)
Type: server_name (0)
Length: 20
Server Name Indication extension
Extension: extended_master_secret (len=0)
Type: extended_master_secret (23)
Length: 0
Extension: psk_key_exchange_modes (len=2)
Type: psk_key_exchange_modes (45)
Length: 2
PSK Key Exchange Modes Length: 1
PSK Key Exchange Mode: PSK with (EC)DHE key establishment (psk_dhe_ke) (1)
Extension: pre_shared_key (len=283)
Type: pre_shared_key (41)
Length: 283
Pre-Shared Key extension
Identities Length: 230
PSK Identity (length: 224)
Identity Length: 224
Identity: 000017719425bdded0ac21c8cd61318334f131527b4ef48f21ba8116523cf71681ae51fa…
Obfuscated Ticket Age: 68965835
PSK Binders length: 49
PSK Binders
The server then responds with
TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Illegal Parameter)
Content Type: Alert (21)
Version: TLS 1.2 (0x0303)
Length: 2
Alert Message
Level: Fatal (2)
Description: Illegal Parameter (47)