We have a server setup containing 2 webservers, a RDS MySQL cluster and a loadbalancer, on AWS. The webservers are running PHP and Nginx, and contains multiple services each.
We have a service managing an entity let's call it service A We then have a service managing authentication, service B
If we put load on Service A, without authentication it manages 400~ requests per seconds, and then nginx starts dropping connections because of too high load. ( This is what we want ).
However, we need Service A to contant service B to get Authenticated.
If we add this call and try to put load on Service A again, we now manage to get ~20 requests per seconds, before the services stops responding.
All the requests after that point is backlogged, and we can tell in the logs, that Service B just times out after 900 seconds for each request, and then Service A times out as well.
(In some instances, PHP-FPM crashes and the services instantly responds with 502 bad gateway).
We have isolated the problem to being something in the vommunication between service A and B. We use Guzzle (A Curl based communication framework) to send the requests, and we make sure that both services are on localhost, and the curl request is made to localhost.
We can tell that the DB server uses about 1.5% CPU ( Everything is in query cache ), and the Loadbalancer uses about 1% CPU.
The only servers which are under pressure are the webservers.
We're quite lost on this one, so any pointers...
