Website is being targeted by mail bots

Question

I have a small website. When I perform a netstat is shows a lot of traffic from .p.mail.

I think this is some kind of mail bot, trying to harvest email addresses from my website. How can I prevent this?

I added this to my htaccess RewriteCond %{HTTP_REFERER} \.(ru|kz|in|mail)(/|$) [NC] RewriteRule .* - [F] but it doesn't seem to help. I'm running an Apache server with a PHP app

tcp        0     64 128.199.152.125:ssh     254.96.96.58.stat:49174 ESTABLISHED
tcp6       1      0 128.199.152.125:http    fetcher9-7.p.mail:52455 CLOSE_WAIT
tcp6       1      0 128.199.152.125:http    crawl-66-249-71-7:39927 CLOSE_WAIT
tcp6       0      0 128.199.152.125:http    fetcher9-5.p.mail:48034 ESTABLISHED
tcp6       1      0 128.199.152.125:http    fetcher9-6.p.mail:38781 CLOSE_WAIT
tcp6       1      0 128.199.152.125:http    fetcher9-3.p.mail:49137 CLOSE_WAIT
tcp6       1      0 128.199.152.125:http    fetcher9.mail.ru:46906  CLOSE_WAIT
tcp6       1      0 128.199.152.125:http    fetcher9-3.p.mail:49102 CLOSE_WAIT
tcp6       0      0 128.199.152.125:http    fetcher9-4.p.mail:60833 ESTABLISHED
tcp6       1      0 128.199.152.125:http    fetcher9-1.p.mail:58404 CLOSE_WAIT
tcp6       1      0 128.199.152.125:http    fetcher9-3.p.mail:38515 CLOSE_WAIT
tcp6       1      0 128.199.152.125:http    crawl-66-249-71-9:65419 CLOSE_WAIT
tcp6       1      0 128.199.152.125:http    fetcher9-4.p.mail:39761 CLOSE_WAIT
tcp6       0      0 128.199.152.125:http    fetcher9-3.p.mail:46664 ESTABLISHED
tcp6       1      0 128.199.152.125:http    fetcher9-5.p.mail:57961 CLOSE_WAIT
tcp6       1      0 128.199.152.125:http    fetcher9-2.p.mail:58029 CLOSE_WAIT
tcp6       1      0 128.199.152.125:http    fetcher9-6.p.mail:53075 CLOSE_WAIT
tcp6       1      0 128.199.152.125:http    fetcher9.mail.ru:47363  CLOSE_WAIT
tcp6       1      0 128.199.152.125:http    fetcher9-4.p.mail:52394 CLOSE_WAIT
tcp6       1      0 128.199.152.125:http    fetcher9.mail.ru:54476  CLOSE_WAIT
tcp6       1      0 128.199.152.125:http    fetcher9.mail.ru:36110  CLOSE_WAIT
tcp6       1      0 128.199.152.125:http    fetcher9-2.p.mail:55155 CLOSE_WAIT
tcp6       1      0 128.199.152.125:http    fetcher9-7.p.mail:59306 CLOSE_WAIT
tcp6       1      0 128.199.152.125:http    fetcher9-2.p.mail:36667 CLOSE_WAIT
tcp6       0      0 128.199.152.125:http    fetcher9-5.p.mail:51968 ESTABLISHED
tcp6       0      0 128.199.152.125:http    fetcher9-4.p.mail:41478 ESTABLISHED
tcp6       1      0 128.199.152.125:http    fetcher9-5.p.mail:60032 CLOSE_WAIT
tcp6       1      0 128.199.152.125:http    fetcher9-2.p.mail:44335 CLOSE_WAIT
tcp6       1      0 128.199.152.125:http    fetcher9-6.p.mail:57922 CLOSE_WAIT
tcp6       1      0 128.199.152.125:http    fetcher9-1.p.mail:59718 CLOSE_WAIT
tcp6       1      0 128.199.152.125:http    fetcher9-3.p.mail:47470 CLOSE_WAIT
tcp6       0      0 128.199.152.125:http    fetcher9-6.p.mail:59941 ESTABLISHED
tcp6       1      0 128.199.152.125:http    fetcher9-1.p.mail:54604 CLOSE_WAIT
tcp6       0      0 128.199.152.125:http    fetcher9.mail.ru:48307  ESTABLISHED
tcp6       1      0 128.199.152.125:http    fetcher9-6.p.mail:47410 CLOSE_WAIT
tcp6       1      0 128.199.152.125:http    fetcher9-2.p.mail:52740 CLOSE_WAIT
tcp6       0      0 128.199.152.125:http    fetcher9.mail.ru:48957  ESTABLISHED
tcp6       0      0 128.199.152.125:http    fetcher9-6.p.mail:55988 ESTABLISHED
tcp6       0      0 128.199.152.125:http    fetcher9-6.p.mail:45431 ESTABLISHED
tcp6       0      0 128.199.152.125:http    crawl-66-249-71-1:54299 ESTABLISHED
tcp6       1      0 128.199.152.125:http    fetcher9-1.p.mail:44075 CLOSE_WAIT
tcp6       0      0 128.199.152.125:http    fetcher9-7.p.mail:51332 ESTABLISHED
tcp6       1      0 128.199.152.125:http    fetcher9-6.p.mail:40081 CLOSE_WAIT
tcp6       1      0 128.199.152.125:http    fetcher9-2.p.mail:47806 CLOSE_WAIT
tcp6       1      0 128.199.152.125:http    fetcher9-5.p.mail:40396 CLOSE_WAIT
tcp6       1      0 128.199.152.125:http    baiduspider-180-7:53078 CLOSE_WAIT
tcp6       1      0 128.199.152.125:http    fetcher9-1.p.mail:46357 CLOSE_WAIT

That's mail.ru, a well known search engine in Russia. They do seem to be crawling rather fast, though. — Michael Hampton, Dec 22 '14 at 15:42
....which begs the question...what is in your robots.txt file — symcbean, Dec 23 '14 at 12:55

Cerveser · Answer 1 · 2014-12-22T15:53:06.377

1

You could use:

RewriteCond %{HTTP_USER_AGENT} ^p\.mail [NC,OR]
RewriteRule ^.* - [F,L]

or block the IP address, but like Michael said, maybe it's a search engine.

edited Dec 22 '14 at 15:53

answered Dec 22 '14 at 15:47

Cerveser

234
1
7

tried using htaccess it didnt work for me, iptables did – Inktown Dec 23 '14 at 12:50

score 0 · Answer 2 · answered Dec 23 '14 at 12:49

I ended up using iptables to block the ips.

From the root linux shell I typed

iptables -A INPUT -s  217.69.133.13  -j DROP
iptables -A INPUT -s  217.69.133.12  -j DROP
iptables -A INPUT -s  217.69.133.10  -j DROP
iptables -A INPUT -s  217.69.133.11  -j DROP
iptables -A INPUT -s  217.69.133.14  -j DROP
iptables -A INPUT -s  217.69.133.15  -j DROP
iptables -A INPUT -s  217.69.133.16  -j DROP
iptables -A INPUT -s  217.69.133.17  -j DROP
iptables -A INPUT -s  217.69.133.18  -j DROP
iptables -A INPUT -s  fetcher9.mail.ru  -j DROP

And it stopped

Website is being targeted by mail bots

2 Answers2