How do I prevent Bing from swamping my site with traffic irregularly?

Question

Bingbot will hit my site pretty hard for a couple of hours each day, and will be extremely light for the rest of the time.

I'd either like to smooth out its crawls, reduce its rate limit, or block it altogether. It doesn't really send through any real visitors.

Is there a way I can smooth its crawling, or rate limit it?

You already added robots.txt as a tag. Does that not do the trick for you? — EboMike, Feb 08 '11 at 01:24

score 23 · Answer 1 · answered Feb 12 '11 at 20:49

Their webmaster blog says that they support adding a crawl-delay parameter to your robots.txt file to throttle the bingbot.

User-agent: msnbot
Crawl-delay: 1

There's a bit more explanation in the webmaster FAQ PDF

These other links might be helpful as well:
http://www.bing.com/toolbox/webmasters
http://www.bing.com/community/webmaster/f/12252/t/651373.aspx

score 1 · Answer 2 · answered Apr 27 '11 at 07:01

You can limit the number of connections from the crawler to f.i. 5 by setting IPTables like this (requires root access to the firewall):

The article at 2bits.com

the setting of IPTables: iptables -I INPUT -p tcp -m connlimit --connlimit-above 5 -j REJECT

This limits connections from each IP address to no more than 5 simultaneous connections. This sort of "rations" connections, and prevents crawlers from hitting the site simultaneously.

score 0 · Answer 3 · answered Feb 08 '11 at 01:14

0

You can ban his IP using HTACCESS.

order allow,deny
deny from 192.168.44.201
deny from 224.39.163.12
deny from 172.16.7.92
allow from all

More about that you may find here: Blog about bot blocking

answered Feb 08 '11 at 01:14

Tomáš Zato

50,171
52
268
778

3

this has a bad side-effect: it will lower your pagerank – knocte Apr 19 '12 at 11:34

How do I prevent Bing from swamping my site with traffic irregularly?

3 Answers3