We have an HTTP client based on libuv and libopenssl for TLS, written in C++ - with it's own TCP wrapper around lubuv and HTTP parser.
Usually it just works and our servers make millions of successful HTTPS requests to social networks with it. And there has never been an issue with single connection requests and it ends in a "Connection: Close" state. But, albeit very rarely, for keep-alive connections sometimes strange ssl errors occurs:
either
error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for modulus
or error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed
It is a really rare issue, once per million, I believe. The error is returned from SSL_read when new data is received from the server. It never happens at the first response, usually there are more than ten request-response exchanges in the connection before the error occurs.
We have a tcpdump of such exchanges and keylogs captured using the SSL_CTX_set_keylog_callback interface. Wireshark opens and decrypts the dump using keylog as pre-master keyfile without any problem and contains a normal HTTP response, "200 OK". For our HTTP client tho the last packet produces an error. By inspecting the exchange we have not noticed any signs of an issue or data corruption. That fact makes me assume that the data is ok but our OpenSSL usage may not be entirely correct.
It is no problem to mock a server that sends exactly the same responses using the tcpdump data. But I need to use the pre-master key from the keylog on the client side. I cannot find any function that sets keys to SSL_CTX* or SSL*. Is there one?
I also modified libopenssl to use constant keys.
I've put memcpy(s->session->master_key, overriden_secret, 48) in ssl_generate_master_secret and tls13_generate_secret. Also memcpy(s->s3->client_random, overriden_random, 32); in tls_construct_client_hello and tls_early_post_process_client_hello. It doesn't work and produces an ssl error at handshake phase with the error error:1416C095:SSL routines:tls_process_finished:digest check failed. The "Client Hello" produced by this patched libopenssl is always different, this probably means I haven't replaced all keys. There is also something in s->tmp structure that I don't quite understand for me to be able to replace all occurrences and values.
It seems that without replaying the exchange under the same condition it is practically impossible to find the problem.
Anyways: I would either like to reproduce this situation by replaying the tcpdump captured for the HTTP client - which would required me to set the key at the correct locations or int the correct way. Or learn about the true nature of the issue.
