how to ban crawler 360Spider with robots.txt or .htaccess?

Question

I've got a problems because of 360Spider: this bot makes too many requests per second to my VPS and slows it down (the CPU-usage becomes 10-70%, but usually i have 1-2%). I looked into httpd logs and saw there such lines:

182.118.25.209 - - [06/Sep/2012:19:39:08 +0300] "GET /slovar/znachenie-slova/42957-polovity.html HTTP/1.1" 200 96809 "http://www.hrinchenko.com/slovar/znachenie-slova/42957-polovity.html" "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider
182.118.25.208 - - [06/Sep/2012:19:39:08 +0300] "GET /slovar/znachenie-slova/52614-rospryskaty.html HTTP/1.1" 200 100239 "http://www.hrinchenko.com/slovar/znachenie-slova/52614-rospryskaty.html" "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider

etc.

How can I block this spider completely via robots.txt? Now my robots.txt looks like this:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/

User-agent: YoudaoBot
Disallow: /

User-agent: sogou spider
Disallow: /

I've added lines:

User-agent: 360Spider
Disallow: /

but that does not seem to work. How to block this angry bot?

If you offer to block it via .htaccess, so mind that it looks now like this:

# Turn on URL rewriting
RewriteEngine On

# Installation directory
RewriteBase /

SetEnvIfNoCase Referer ^360Spider$ block_them
Deny from env=block_them

# Protect hidden files from being viewed
<Files .*>
    Order Deny,Allow
    Deny From All
</Files>

# Protect application and system files from being viewed
RewriteRule ^(?:application|modules|system)\b.* index.php/$0 [L]

# Allow any files or directories that exist to be displayed directly
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d

# Rewrite all other URLs to index.php/URL
RewriteRule .* index.php/$0 [PT]

And, in spite of presence of

SetEnvIfNoCase Referer ^360Spider$ block_them
Deny from env=block_them

this bot still tries to kill my VPS and is logged in access logs.

use `.htaccess` or `nginx` rewrite module, `robots.txt` do not block anything its just list of rules — Alexander Larikov, Sep 06 '12 at 17:07

score 8 · Answer 1 · answered Jan 26 '13 at 00:47

In your .htaccess file simply add the following :

RewriteCond %{REMOTE_ADDR} ^(182\.118\.2)

RewriteRule ^.*$ http://182.118.25.209/take_a_hike_moron [R=301,L]

This will catch ALL the bots being launched from the 182.118.2xx.xxx range and send them back to themself...

The crappy 360 bot is being fired from servers in China... so as long as you don't mind saying bye bye to crappy Chinese traffic from that IP range, this will guaranteed make those puppies disappear from reaching any files on your web site.

The following two lines in your .htaccess file will also pick it off simply by it being stupid enough to proudly put 360spider in its user agent string. This could be handy for when they use other IP ranges then the 182.118.2xx.xxx

RewriteCond %{HTTP_USER_AGENT} .*(360Spider) [NC]

RewriteRule ^.*$ http://182.118.25.209/take_a_hike_moron [R=301,L]

And yes... I hate them too !

They seem to use 101.226.16{8,9} too. – Stephen Paulger Jun 05 '13 at 08:21 — Stephen Paulger, Jun 05 '13 at 08:21

score 7 · Accepted Answer · edited Apr 13 '17 at 12:13

7

Your robots.txt seems right. Some bots just ignore it (malicious bots crawl from any IP address from any botnet of hundreds to millions of infected devices from all around the globe), in this case you can limit the number of requests per second using mod_security module for apache 2.X

Config example here: http://blog.cherouvim.com/simple-dos-protection-with-mod_security/

[EDIT] On linux, iptables also allows restricting tcp:port connections per (x) second(s) per ip, providing conntrack capabilities are enabled on your kernel. See: https://serverfault.com/questions/378357/iptables-dos-limit-for-all-ports

edited Apr 13 '17 at 12:13

Community

1
1

answered Sep 06 '12 at 17:23

NotGaeL

8,344
5
40
70

You can also use .htaccess as seen in other answers, but take on account there will always be other IPs and referers/agents trying to mess with your server, and a lot of damage can be done before you reconfigure to stop them... – NotGaeL Sep 06 '12 at 17:29
Yes, this was the answer for my second question I had no time to ask :) Thanx :) – kovpack Sep 06 '12 at 19:25

score 4 · Answer 3 · edited Dec 28 '14 at 10:23

4

You can put following rules into your .htaccess file

RewriteEngine On  
RewriteBase /  
SetEnvIfNoCase Referer 360Spider$ block_them
Deny from env=block_them

Note: Apache module mod_setenvif should be enabled in your server configuration

edited Dec 28 '14 at 10:23

guido

18,864
6
70
95

answered Sep 06 '12 at 17:20

Alexander Larikov

2,328
15
15

This seems to be what I need, but for some reason it does not work. I've added `SetEnvIfNoCase Referer ^360Spider$ block_them Deny from env=block_them`, but still see that spider in logs. By the way, I have also block ` Order Deny,Allow Deny From All ` and also some rewriting rules after it. – kovpack Sep 06 '12 at 19:44
1

`SetEnvIfNoCase user-agent 360Spider$ block_them`. "360Spider" is at the end of the user-agent header. – Hendrik Brummermann Sep 25 '12 at 09:44

score 3 · Answer 4 · answered Sep 06 '12 at 17:16

3

The person running the crawler might be ignoring robots.txt. You could block them via IP

order deny, allow
deny from 216.86.192.196

in .htaccess

answered Sep 06 '12 at 17:16

d4v3y0rk

345
1
9

Spider uses 30 or even more different IPs :( – kovpack Sep 06 '12 at 20:53
That was just part of the log :) – kovpack Sep 07 '12 at 18:14

score 0 · Answer 5 · answered Dec 25 '12 at 23:12

0

SetEnvIfNoCase User-agent 360Spider blocked

answered Dec 25 '12 at 23:12

Doug Wilson

1

score 0 · Answer 6 · answered Jan 15 '13 at 04:52

I have lines in my .htaccess file like this to block bad bots:

RewriteEngine On
RewriteCond %{ENV:bad} 1
RewriteCond %{REQUEST_URI} !/forbidden.php
RewriteRule (.*) - [R=402,L]

SetEnvIf Remote_Addr "^38\.99\." bad=1
SetEnvIf Remote_Addr "^210\.195\.45\." bad=1
SetEnvIf Remote_Addr "^207\.189\." bad=1
SetEnvIf Remote_Addr "^69\.84\.207\." bad=1

# ...

SetEnvIf Remote_Addr "^221\.204\." bad=1
SetEnvIf User-agent "360Spider" bad=1

It will send the status code 402 Payment Required to all blacklisted IPs / user-agents.

You can put anything that you want displayed to the bot in forbidden.php.

It's quite effective.

score 0 · Answer 7 · answered Feb 18 '13 at 00:48

0

I just had to block 360Spider. Solved with StreamCatcher on IIS (IIS7), which fortunately was already installed so only a small configuration change was needed. Details at http://needs-be.blogspot.com/2013/02/how-to-block-spider360.html

answered Feb 18 '13 at 00:48

user424855

233
1
2
8

score 0 · Answer 8 · answered Jul 24 '14 at 21:19

I use the following, and it helps alot! Check the HTTP_USER_AGENT for bad bots

<IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteBase /

    RewriteCond %{REQUEST_URI} !^/robots\.txt$
    RewriteCond %{REQUEST_URI} !^/error\.html$

    RewriteCond %{HTTP_USER_AGENT} EasouSpider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} YisouSpider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} Sogou\ web\ spider [NC]
    RewriteCond %{HTTP_USER_AGENT} 360Spider [NC,OR]
    RewriteRule ^.*$ - [F,L]
</IfModule>

<Location />
    <IfModule mod_setenvif.c>
        SetEnvIfNoCase User-Agent "EasouSpider" bad_bot
        SetEnvIfNoCase User-Agent "YisouSpider" bad_bot
        SetEnvIfNoCase User-Agent "LinksCrawler" bad_bot

        Order Allow,Deny
        Allow from All
        Deny from env=bad_bot
    </IfModule>
</Location>

how to ban crawler 360Spider with robots.txt or .htaccess?

8 Answers8