I have been dealing with an issue on one of my servers for quite a while now. In short, the server receives massive spikes in load - up to 50 - but the site gets almost no traffic. I know this can be down to poor coding standards, large PHP processes and badly formed MySQL statements however the site runs normally 90% of the time.
I have checked the access logs during the spikes and more often than not its some unknown site crawler thats trying to load a million pages at once and causing issue.
Recently however I have been getting access logs that show something similar to:
127.0.0.1 - - [13/Aug/2014:08:39:55 -0700] "GET /index.php HTTP/1.1" 200 20579 "-" "-"
where there is no user agent string. Which I wouldn't normally care about apart from this unknown user agent is requesting 100s of pages within a few seconds, which start with HTTP 200s then 499 errors and as the server overloads gets 502 and then 503 errors. So i can't block it in robots.txt or any other blacklisting because i have nothing there to blacklist/disallow.
The stack is a standard ubuntu, nginx, php-fpm, MySQL with Varnish caching.
Does anyone know whats going on here?
