ProxyRemote leads to "AH00898: Error during SSL Handshake with remote server ...", works on other server

Question

Goal

Set up a Virtual Host as Reverse Proxy, which also acts as Forward Proxy to another "Remote" Proxy for a specific URL pattern.

Issue

I have 2 servers (actually 2 separate machines), both having the same config but only one server is able to forward the requests.

I've searched the whole web and did even more experiments than described below (but seem irrelevant to mention here), so I'm very very thankful for any idea/experiment you come up with!

Config

Dump

Acc. to apache2ctl -DDUMP_CONFIG | grep -vE "^[ ]*#[ ]*[0-9]+:$" > apache_dump.conf the apache config of both servers is identical.

Virtual Host

<VirtualHost *:80>

[...]

SSLProxyEngine on
ProxyRemote "https://booking-service.com/" "http://remote-proxy:3128"

<Location /booking>
    ProxyPass https://booking-service.com/api
    ProxyPassReverse https://booking-service.com/api
    ProxyPreserveHost Off
    RequestHeader set X-Api-Key "..."
    RequestHeader unset Cookie
    RequestHeader unset Authorization
</Location>

</VirtualHost>

Modules

Here an excerpt of activated modules which IMHO might be relevant:

[...]
http_module (static)
[...]
ssl_module (shared)
[...]
proxy_module (shared)
proxy_http_module (shared)
proxy_ftp_module (shared)
proxy_ajp_module (shared)
proxy_wstunnel_module (shared)
proxy_balancer_module (shared)
[...]

Regular Error Logs

(IPs and host names obfuscated)

[proxy:trace2] [pid 21616:tid 140692231767808] proxy_util.c(3016): HTTPS: fam 2 socket created to connect to booking-service.com
[proxy:debug] [pid 21616:tid 140692231767808] proxy_util.c(3050): AH02824: HTTPS: connection established with 192.18.191.131:3128 (booking-service.com)
[proxy:debug] [pid 21616:tid 140692231767808] proxy_util.c(2677): AH00948: CONNECT: sending the CONNECT request for booking-service.com:443 to the remote proxy 192.18.191.131:3128 (remote-proxy.net)
[proxy:debug] [pid 21616:tid 140692231767808] proxy_util.c(2731): AH00949: send_http_connect: response from the forward proxy: HTTP/1.1 200 Connection established\r\n\r\n
[proxy:debug] [pid 21616:tid 140692231767808] proxy_util.c(3218): AH00962: HTTPS: connection complete to 192.18.191.131:3128 (remote-proxy.net)
[proxy:error] [pid 21616:tid 140692231767808] (20014)Internal error (specific information not available): [client 111.222.33.444:20435] AH01084: pass request body failed to 192.18.191.131:3128 (remote-proxy.net)
[proxy:error] [pid 21616:tid 140692231767808] [client 111.222.33.444:20435] AH00898: Error during SSL Handshake with remote server returned by /booking/test-request
[proxy_http:error] [pid 21616:tid 140692231767808] [client 111.222.33.444:20435] AH01097: pass request body failed to 192.18.191.131:3128 (remote-proxy.net) from 111.222.33.444 ()
[proxy:debug] [pid 21616:tid 140692231767808] proxy_util.c(2334): AH00943: HTTPS: has released connection for (booking-service.com)

Error Logs With ssl:trace7

I get the following for both web servers:

[ssl:trace3] [pid 25298:tid 140395937773312] ssl_engine_kernel.c(2180): [remote 192.18.191.131:3128] OpenSSL: Handshake: start
[...]
[ssl:trace3] [pid 25298:tid 140395937773312] ssl_engine_kernel.c(2189): [remote 192.18.191.131:3128] OpenSSL: Loop: before/connect initialization
[ssl:trace4] [pid 25298:tid 140395937773312] ssl_engine_io.c(2214): [remote 192.18.191.131:3128] OpenSSL: write 517/517 bytes to BIO#7fb05c009ba0 [mem: 7fb05c011070] (BIO dump follows)
[ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2137): [remote 192.18.191.131:3128] +-------------------------------------------------------------------------+
[ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2175): [remote 192.18.191.131:3128] | 0000: 16 03 01 02 00 01 00 01-fc 03 03 c5 c2 b9 30 65  ..............0e |
[...]
[ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2175): [remote 192.18.191.131:3128] | 0140: 03 00 0f 00 01 01 00 15-00 bb                    ..........       |
[ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2179): [remote 192.18.191.131:3128] | 0517 - <SPACES/NULS>
[ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2181): [remote 192.18.191.131:3128] +-------------------------------------------------------------------------+
[ssl:trace3] [pid 25298:tid 140395937773312] ssl_engine_kernel.c(2189): [remote 192.18.191.131:3128] OpenSSL: Loop: SSLv2/v3 write client hello A
[ssl:trace4] [pid 25298:tid 140395937773312] ssl_engine_io.c(2214): [remote 192.18.191.131:3128] OpenSSL: read 7/7 bytes from BIO#7fb05c00cc70 [mem: 7fb05c0165d0] (BIO dump follows)
[ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2137): [remote 192.18.191.131:3128] +-------------------------------------------------------------------------+
[ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2175): [remote 192.18.191.131:3128] | 0000: 16 03 03 00 41 02                                ....A.           |
[ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2179): [remote 192.18.191.131:3128] | 0007 - <SPACES/NULS>
[ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2181): [remote 192.18.191.131:3128] +-------------------------------------------------------------------------+
[ssl:trace4] [pid 25298:tid 140395937773312] ssl_engine_io.c(2214): [remote 192.18.191.131:3128] OpenSSL: read 63/63 bytes from BIO#7fb05c00cc70 [mem: 7fb05c0165da] (BIO dump follows)
[ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2137): [remote 192.18.191.131:3128] +-------------------------------------------------------------------------+
[ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2175): [remote 192.18.191.131:3128] | 0000: 00 3d 03 03 f8 86 f8 5b-c5 71 0e 3f d6 fb 37 1d  .=.....[.q.?..7. |
[...]
[ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2175): [remote 192.18.191.131:3128] | 0030: 00 00 00 00 0b 00 04 03-00 01 02 00 23           ............#    |
[ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2179): [remote 192.18.191.131:3128] | 0063 - <SPACES/NULS>
[ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2181): [remote 192.18.191.131:3128] +-------------------------------------------------------------------------+
[ssl:trace3] [pid 25298:tid 140395937773312] ssl_engine_kernel.c(2189): [remote 192.18.191.131:3128] OpenSSL: Loop: unknown state
[ssl:trace4] [pid 25298:tid 140395937773312] ssl_engine_io.c(2214): [remote 192.18.191.131:3128] OpenSSL: read 5/5 bytes from BIO#7fb05c00cc70 [mem: 7fb05c026da3] (BIO dump follows)
[ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2137): [remote 192.18.191.131:3128] +-------------------------------------------------------------------------+
[ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2175): [remote 192.18.191.131:3128] | 0000: 16 03 03 0d ce                                   .....            |
[ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2181): [remote 192.18.191.131:3128] +-------------------------------------------------------------------------+
[ssl:trace4] [pid 25298:tid 140395937773312] ssl_engine_io.c(2214): [remote 192.18.191.131:3128] OpenSSL: read 3534/3534 bytes from BIO#7fb05c00cc70 [mem: 7fb05c026da8] (BIO dump follows)
[ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2137): [remote 192.18.191.131:3128] +-------------------------------------------------------------------------+
[ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2175): [remote 192.18.191.131:3128] | 0000: 0b 00 0d ca 00 0d c7 00-07 12 30 82 07 0e 30 82  ..........0...0. |
[...]
[ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2175): [remote 192.18.191.131:3128] | 0dc0: ca 5b e0 d5 f6 6c 23 9d-20 29 55 cd 3a c5        .[...l#. )U.:.   |
[ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2181): [remote 192.18.191.131:3128] +-------------------------------------------------------------------------+

The BAD web server does abruptly stop here, there is no "Certificate Verification" and no "Handshake: done", i.e. no further ssl:... log entries related to this client request.

In contrast, the GOOD web server does the following:

[ssl:debug] [pid 9659:tid 140475999237888] ssl_engine_kernel.c(1738): [remote 192.18.191.131:3128] AH02275: Certificate Verification, depth 1, CRL checking mode: none (0) [subject: CN=...]
[ssl:debug] [pid 9659:tid 140475999237888] ssl_engine_kernel.c(1738): [remote 192.18.191.131:3128] AH02275: Certificate Verification, depth 0, CRL checking mode: none (0) [subject: CN=...]
[...]
[ssl:trace3] [pid 9659:tid 140475999237888] ssl_engine_kernel.c(2184): [remote 192.18.191.131:3128] OpenSSL: Handshake: done
[...]

Failed Experiments

What I tried so far:

Adding the following settings (even if the other web server works w/o them, i know... I'm desperate :D):

SSLProxyVerify none
SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off
SSLProxyCheckPeerExpire off
SSLProxyProtocol all -SSLv2 -SSLv3 -TLSv1
SSLProxyCACertificateFile /etc/ssl/certs/<the-ca-cert>.crt [afaik should be considered anyway b/c in /etc/ssl/certs]

Restarting the apache2.service
Restarting the whole linux machine
Request with curl: works!
- curl --request POST 'https://booking-service.com/api/test-request' --header 'Content-Type: application/json' --header 'X-Api-Key: ...' --proxy 'http://remote-proxy.net:3128' --data '@/tmp/request-body.txt' -iv
Debug with openssl: looks good & is the same for both servers
- openssl-1_1 s_client -connect booking-service.com -proxy remote-proxy.net:3128 -state -debug

App Versions (identical on both servers)

Linux: lsb_release -a:

LSB Version:    n/a
Distributor ID: SUSE
Description:    SUSE Linux Enterprise Server 12 SP5
Release:        12.5
Codename:       n/a

Apache: httpd -v:

Server version: Apache/2.4.38 (Linux/SUSE)
Server built:   2019-02-08 01:59:10.000000000 +0000

OpenSSL: openssl version:

OpenSSL 1.0.2p-fips  14 Aug 2018

score 0 · Answer 1 · answered Aug 16 '21 at 20:57

0

I would check that the TLS/SSL certificates are the same in all server.

By default httpd is generating a self signed certificate in each installation.

It is imperative that all servers/hosts in TLS communication cluster share identical certificates in order to trust each other.

I suggest to copy private key, and public certificate from host #1 to other hosts.

answered Aug 16 '21 at 20:57

Dudi Boy

4,551
1
15
30

Thx for the idea! But certs are the same on both servers. Checked the relevant cert directories & also analyzed/confirmed with curl & openssl (just added this info to the question as well). So if curl works, I assume that in apache there is a specific setting defined which enforces mod_ssl to use openssl in a way that curl does not - but then again I remember that the apache config dumps of both web servers are equal, so there shouldn't be any differences in apache settings!? – von_court Aug 18 '21 at 08:21