Why do insurance and banking domains send bot traffic to my site?

Question

I know how to detect and filter bot traffic, but I'm wondering if anyone knows why I am seeing tons of bot traffic from insurance and banking domains? E.g., bankofamerica.com, allstate.com and others are hitting my site upwards of 100 times per day. Note that I am talking about domain associated with the user's IP, not the traffic referral domain.

The site is a large ecommerce retailer and the banking and insurance bot traffic started after it moved to the Demandware ecommerce platform.

@Layke, edited to add context around domain--I mean the domain associated with the user's IP address, not the referring domain from a traffic perspective. — Michelle, Mar 11 '15 at 15:55

score 0 · Answer 1 · answered Mar 12 '15 at 10:09

0

Which kind of traffic are they sending? It would be great to have a HTTP request as example, try sniff the traffic to your website with a tcpsniffer such as tcpdump or ngrep.

answered Mar 12 '15 at 10:09

Cristian - ScrapeSentry.com

36
3

Thanks, @Cristian, I waited a few months for any other insights, but this is the most helpful solution so far. – Michelle Sep 08 '15 at 19:31

SGD · Answer 2 · 2015-03-12T19:33:09.067

0

It could be that those are simply users which are bind a corporate proxy and thus appear as one IP. How do you know it is bot traffic?

edited Mar 12 '15 at 19:33

answered Mar 12 '15 at 12:59

SGD

1,676
14
18

I'm looking at pageviews per session, and seeing that all sessions from these IPs are only one pageview. In other words, they hit one page on the site then a new session begins and they hit once, then a new session begins and they hit once, etc for hundreds of sessions. – Michelle Apr 07 '15 at 16:23
1

Can you see the user-agent for those requests? – SGD Apr 07 '15 at 16:25

score 0 · Answer 3 · answered Mar 15 '15 at 09:50

0

You can prevent those request hitting your server by implementing DOS ( denial of service) on the application level and DDOS on your network level if you have the cisco firewall installed in your infrastructure.

answered Mar 15 '15 at 09:50

Deepak Rohilla

97
6

Thanks, Deepak; I'm not concerned about how to block the traffic, I'm mostly wondering why I would see so much bot activity from insurance and banking domains--maybe there is a business reason why they ping sites? – Michelle Apr 07 '15 at 16:24

score 0 · Answer 4 · answered Sep 28 '15 at 13:23

If you still have this problem, contact Demandware support by raising a customer support ticket in their SalesForce portal. There is not much that you could by configuration of your Demandware site from the Business Manager. Chances are that the traffic was always there, just that it was filtered by the firewall of your previous e-commerce provider.

score 0 · Accepted Answer · answered Aug 14 '16 at 14:49

0

Turns out this was caused by bots spoofing legitimate domains

answered Aug 14 '16 at 14:49

Michelle

7
4

Why do insurance and banking domains send bot traffic to my site?

5 Answers5