We are a small company that has an Android app backed by 3 servers running Proxmox. All the connections use HTTPS.
The problem we are experiencing is the following: connections made from outside the LAN timeout ~95% of the time.
We tried to expose all kind of services to our public IP, including HAProxy, Apache, Nginx, IIS and a custom-made Java program; all these software have the same issue. When possible, we tested them on Debian, Windows Server 2016 and Windows 10 in a physical and virtual environment.
We normally receive about ~1000 requests/second, but in the last week or so, only ~30 requests/second are handled, according to the logs of every software. The other requests timeout.
When we test this from our client, approximately 95% of the requests timeout after 20 seconds. When we try to connect to the servers from outside the LAN on any port or protocol (for example SSH) the same issue still stands, so the problem, apparently, is not limited to HTTPS.
We checked the DNS and there are no problems.
The only test that successfully "handled" all the ~1000 requests/second was this Java program that immediately closed the socket when a request was made. In this case, we could see from our client that every request immediately gave the expected error, without timing out. When we introduced a small 1-second delay before closing the connection, to simulate a request being processed, the above problem arose.
We tried literally everything we know in order to solve the problem, but nothing seems to have changed the situation.
What do you suggest to fix or debug the problem? What could the cause be?
Thanks a lot for your help, Marco
