1

I rent a server somewhere out in Canada or so that I'm using to host a website of mine. That website has close to 400,000 pages that I wanted to index today. For that, I wrote a crawler a while back (see JCrawler on Stackoverflow.com). Now, I'm greedy and didn't want it to take too long so I ran multiple threads resulting in some 60+ requests per second from my IP. A couple minutes later, my server locked me out. I can still FTP into it, but I can't HTTP it.

As server administrator or user, do you have any idea how servers usually handle these situations? Is it common to place a permanent or temporary ban on the IP or what is typically done?

Naturally, I'll re-run my software with fewer requests once I'm back on.

Community
  • 1
Jan K.
  • 113
  • 5

2 Answers2

1

Take a look at mod_evasive. It is probably what locked you out based on your description.

user6738237482
  • 1,510
  • 12
  • 7
0

What did lock you out?

It is your server, so you should know / check with your provider before coming here. Can be some auto firewall on the server that you are not aware of (nitpick: says a lot about your ability as administrator) or some provider firewall thingy that locks you out (in which case it is actually not the server doing anything).

TomTom
  • 51,649
  • 7
  • 54
  • 136
  • Judging by your comment, I guess I'm glad I'm not the administrator. I just pay for hosting. I finally gave in and called them (PITA because I'm currently in Europe; I wanted to avoid an international call), so let's make this a general kind of question. I was unable to access the server via HTTP request. FTP worked fine and any other IP was able to access the server. – Jan K. Jun 05 '10 at 18:23
  • Then syou should not have said "I rent a server" which implies a physical box that you administrate ;) – TomTom Jun 06 '10 at 03:48