-1

Problem

1 of ~150 server clients (in different locations and with different network setups) is not redirected through my apache service. I need to know where the problem is but can't figure it out.

All clients access a virtual host and send the same requests against the proxy:

  <VirtualHost *:80>
  ServerName update.***.tld
  ServerAdmin mail@company.tld

  CustomLog /var/log/apache2/update.***.tld_access.log combined
  ErrorLog  /var/log/apache2/update.***.tld_error.log

  # redirect all http request to https
  RewriteEngine on
  Options +FollowSymLinks
  RewriteCond %{SERVER_PORT} !^443$
  RewriteRule ^.*$ https://%{SERVER_NAME}%{REQUEST_URI} [L,R]
  </VirtualHost>

  <VirtualHost *:443>
  ServerName update.***.tld

  LogLevel warn
  SSLEngine on
  SSLCertificateFile /etc/ssl/certs/wildcard.***.tld.cert
  SSLCertificateKeyFile /etc/ssl/private/wildcard.***.tld.key
  SSLCertificateChainFile /etc/ssl/certs/wildcard.***.tld.combined.cert

  CustomLog /var/log/apache2/update.***.tld_access.log combined
  ErrorLog  /var/log/apache2/update.***.tld_error.log

  TimeOut 3600
  KeepAlive On

  AddDefaultCharset UTF-8

  SSLProxyEngine on
  ProxyPreserveHost Off
  SetEnv force-proxy-request-1.0 1
  SetEnv proxy-nokeepalive 1
  ProxyTimeout 15

  ProxyRequests Off
  ProxyPass /         https://***-***-prod.aws.tld/
  ProxyPassReverse /  https://***-***-prod-prod.aws.tld/

  <Proxy *>
     AddDefaultCharset UTF-8
     Require all granted
  </Proxy>
  </VirtualHost>

Debug

  • Changed the client to a static IP setup with Google DNS server
  • Checked the log files
    • It seems that all requests from that special client are not redirected correctly
    • The requests from that client are logged in default_access.log and dont reach my custom log so I guess the forwarding is not working but why it is for 150 other clients ..
    • Default access log: clientipaddress - - [23/Jul/2020:20:23:17 +0200] "GET /api/agent/ping HTTP/1.1" 400 301 "-" "-"
    • When I send a wget from the client to my proxy on Port 443 it is correctly forwarded to the virtual host and logged in my custom log
    • I have checked the tcpdump and noticed that the client tries to send to the correct server with the correct port (443)

tcpdump on corrupt client:

tcpdump -i eth0 -vvv host update.***.tld  > dump

https://gist.github.com/herz0g/e02ef883688c904667164a175955ecc0

Conclusion

I guess it is a problem on the customer side network otherwise it would not work for 150 other clients but I am not sure how to prove that or what could be debugged further.

Stuggi
  • 3,506
  • 4
  • 19
  • 36
Herzog
  • 11
  • 3
  • There seems to be some traffic missing from `tcpdump`. The complete connection is not shown. Try using `host ` rather than `net`. – Michael Hampton Jul 23 '20 at 20:45
  • "is not redirected through my apache service. " What does it mean exactly? What happens exactly? Your gist of tcpdump is not easily readable, can't you first see what happens at the HTTP(S) level by making the client use wget or curl? PS: If you gave real names people could help you better. – Patrick Mevzek Jul 23 '20 at 21:31
  • @MichaelHampton I used net because the n option is not working with host and I wanted to see the IP the host was trying to connect. But I see that it makes sense to capture with the host option. I changed the tcpdump query and captured with the host option against the URL the client agent will call. Added a complete capture of a client run against the proxy in my post. – Herzog Jul 23 '20 at 21:32
  • If you see your client access in your logs, it means it reaches your server. If your server serves it 400 instead of a redirection it means an error in your server and its configuration. This is an HTTP level problem and looking at the TCP level won't be of a lot of help. You can crank up logging level in Apache, among other thing to try capture exactly the HTTP request and replay it. – Patrick Mevzek Jul 23 '20 at 21:32
  • That capture looks complete, and perfectly normal. – Michael Hampton Jul 23 '20 at 21:36
  • @PatrickMevzek, like i wrote in my post I already tested the access with wget and this action is reaching my apache virtual host. I see the request in my virtual host log. On the client is an agent installed which tries to reach the exact same url I use with wget but this action is only shown in my default access log, not into my virtual host log. So I guess it is not forwarded correctly, at least I would suggest that in combination with the 301 error I see in the log. I checked the tcp level because I wanted to make sure that the address and port the client agent tries to reach are correct. – Herzog Jul 23 '20 at 21:50
  • 1
    Again: if it reaches your Apache and there is an HTTP reply even if not the one you expect, it is not a TCP problem, but an HTTP configuration one. Adding more logs (see LogLevel) and trying to replay locally the HTTP request done by the faulty client should give you the explanation of the trouble. – Patrick Mevzek Jul 23 '20 at 23:15

2 Answers2

1

You actually partly answer your own question:

when I send a wget from the client to my proxy on Port 443 it is correctly forwarded to the virtual host and logged in my custom log

The incorrect Apache redirect is the issue you have, because you're relying on the explicit port number, and that's not very common.

RewriteCond %{SERVER_PORT} !^443$

This suggest client doesn't have port number in the URI of the API call and when it's not present it fails.

Try to change the redirect configuration in Apache to suggestion in this answer

Geeky Masters
  • 729
  • 3
  • 8
  • "This suggest client doesn't have port number in the URI" Which is like 99.9999999% of cases as 443 is the default HTTPS one, so (almost) no one uses URLs like `https://www.example.com:443/`. – Patrick Mevzek Jul 23 '20 at 23:11
  • All Agents on all clients send the same request to the same url with the same port - because the configuration file for my agents is identical for host/port configuration at all of them. If my rewrite rule would be wrong no agent would be redirected - this is not the case. I changed my configuration like described in your answer without any effect. – Herzog Jul 24 '20 at 10:15
  • if it didn't fix your issue, I'd check the SSL/TLS libs on client's end. What sort of device is sending the request (IOT?). I'd try to create a "blind" alias client can call and put it on port 80 if it won't work it's probably network related. If it will it's possibly the encryption, search in your logs if you can see anything in your apache log like "AH####" might give you a hint.Double check with client if they can access the url i.e. via phone or different internet connection, might be firewall/routing issue. Capture their request via wireshark, "follow the stream" of the request. – Geeky Masters Jul 24 '20 at 12:46
1

As mentioned by @PatrickMevzek this wasn't a network or DNS issue. I checked the logs with debug level enabled and noticed this error:

[Fri Jul 24 12:33:11.463639 2020] [ssl:info] [pid 9792:tid 139651482162944] [client clientip:26294] AH01964: Connection to child 441 established (server default.virtual.host:443)
[Fri Jul 24 12:33:11.463917 2020] [ssl:debug] [pid 9792:tid 139651482162944] ssl_engine_kernel.c(2096): [client clientip:26294] AH02043: SSL virtual host for servername update.***.tld found
[Fri Jul 24 12:33:11.612839 2020] [core:debug] [pid 9792:tid 139651482162944] protocol.c(1158): [client clientip:26294] AH02427: Request header value is malformed: TOKEN ******\r
[Fri Jul 24 12:33:11.612873 2020] [core:debug] [pid 9792:tid 139651482162944] protocol.c(1318): [client clientip:26294] AH00567: request failed: error reading the headers

Seems that my config file on the client was screwed up for any reason but the content within the file was the same like for my other agents, also the rights and the owner was correct. I deleted the file and created a new one. Everything is working now.

Thanks for pointing me into the correct direction.

Herzog
  • 11
  • 3