0

I am tired by bingbot, howto fight against this robot. it create too session.

I do that in ht access : opinion ?

#bingbot create too session
RewriteCond %{HTTP_USER_AGENT}  ^bingbot/.*         [OR,NC]
RewriteCond %{HTTP_USER_AGENT} bingbot [NC]
RewriteRule ^(.*)$ http://go.away/                      [L]
luator
  • 4,769
  • 3
  • 30
  • 51
Kurama
  • 37
  • 1
  • 16

1 Answers1

1

.htaccess

RewriteEngine On

RewriteCond %{HTTP_USER_AGENT} (bingbot) [NC]
RewriteRule .* - [R=403,L]

Or use robots.txt

User-agent: bingbot
Disallow: /
drj
  • 533
  • 2
  • 16
  • doesn't work. I delete my session, my stats and 2 mn after I have 32 bingbot – Kurama Mar 23 '15 at 15:16
  • It might take some time for bots to see your robots.txt To test the .htaccess, use an user-agent switcher. Here's a list of user agents for spiders: http://www.useragentstring.com/pages/Crawlerlist/ I'll play with this a little bit this evening. – drj Mar 24 '15 at 01:26
  • I tested the htaccess code with the following tool and got a 403 error. https://www.xml-sitemaps.com/se-bot-simulator.html It is possible that your logs are still getting hit with this, if that's what you're using to test. Bing will likely stop crawling if it's getting a constant 403. – drj Mar 24 '15 at 01:30
  • If you want to prevent bing from hitting apache at all, use a firewall like IP Tables. This is an interesting article: http://www.computersolutions.cn/blog/2012/05/msn-bing-crawler-spider-madness/ – drj Mar 24 '15 at 01:33
  • http://stackoverflow.com/questions/19209551/how-can-i-block-some-special-user-agents-via-iptables <- blocking by user-agent using iptables – drj Mar 24 '15 at 01:34
  • Read this regarding bing bot and robots.txt: http://blogs.bing.com/webmaster/2012/05/03/to-crawl-or-not-to-crawl-that-is-bingbots-question/ – drj Mar 24 '15 at 01:35
  • I try this : User-agent: bingbot Disallow: marketplace/advanced_search.php Disallow: marketplace/advanced_search_result.php – Kurama Mar 25 '15 at 01:47