2

As of recently when I look at my web statistics through AWStats, I see several things that concern me. The first is 'Unknown robot' listed under the 'Robots/spiders visitors' tab. The second, and most concerning line is 'A PHP script' under the same AWStats robots tab. I run content on my site that should not be fetched by other site's PHP scripts. Is there any way to log this in Apache logs? In other words, how can I tell if the script is being called by a PHP script (through logs or PHP functions)? Lastly, an image is listed below that shows what I'm describing. As you can see, hits from a normal bot - Googlebot - number in the hundreds whereas the hits from the 'Unknown robots' number roughly 700 thousand.

Community
  • 1
  • 1
B. Schmidt
  • 85
  • 5
  • I'm having the same issue with my website. 'A PHP script' has been leeching away most of my bandwidth (I only get about 1GB a month). I can tolerate Google, Yahoo, Bing and even Baidu as they help me with SEO, but I can't tolerate other lechers ! I have currently setup a trap and the "bad robots" are already falling for it. Just make sure you declare your trap 'not allowed' in the robots.txt so that good crawlers obey them. Any help regarding the 'A PHP script' would be highly appreciated ! – thethakuri Aug 08 '15 at 08:58

1 Answers1

1

There's no 100% way to do it, as no matter what kind of script is connecting to your site, it can make it look like a browser so you will never know.

The only thing crossing my mind — they call "PHP script" requests that contain something specific in $_SERVER['HTTP_USER_AGENT'], for example user agents starting with PHP/, like PHP/5.2.9.

Oleg Dubas
  • 2,320
  • 1
  • 10
  • 24