0

This is the best I can describe my problem...

I am running a search engine scraper essentially on Windows 2008 Server. Constantly accessing millions of random domain names and following their links basically.

After a few hours of serious scraping... the server will become non responsive in terms of HTTP requests. The only solution is the restart the server.

The URL http://google.com does not respond. Even an http://localhost/ request to a running WAMP instance does not respond. Although I am still able to connect remotely to the machine somehow... so internet services are indeed working.

I have tried disabling the firewall already. I have tried clearing the DNS cache as well. The only solution is a restart, and then HTTP requests work immediately.

Is there some setting I can alter or error list I can view for these types of errors on Windows 2008 Server?

EDIT: Upon further inspection, it looks like my maximum available ports are being maxed out. cports tells me I have 15,000 "System" ports (every single one from 49,158 to 65,535) is stuck in a "Fin Wait 1" state. Any ideas?

A script of some sort that forcibly removes all fin wait 1 connections regularly via task scheduler perhaps?

darkAsPitch
  • 1,931
  • 4
  • 25
  • 42

1 Answers1

0

It sounds like you have managed to seriously overload IIS.

Have you adjusted the settings for your application pool to account for this huge throughput ?

Of course, you don't say whether IIS itself is used in making external web requests - I do hope you understand that these have nothing to do with one another in principle.

adaptr
  • 16,576
  • 23
  • 34
  • the OP says he is running WAMP - which might be part of the problem. – Jim B Dec 22 '11 at 15:32
  • It's an obscure reference at best. – adaptr Dec 22 '11 at 15:41
  • WAMP is an obscure reference? – Jim B Dec 22 '11 at 20:56
  • Upon further inspection, it looks like my maximum available ports are being maxed out. cports tells me I have 15,000 "System" ports (every single one from 49,158 to 65,535) is stuck in a "Fin Wait 1" state. Any ideas? – darkAsPitch Dec 22 '11 at 22:34
  • @JimB: it is if it is referenced halfway down the end of a sentence that talks about HTTP *client* services, and doesn't have anything to do with a web server at all. He could still be using IIS for the client part. – adaptr Dec 23 '11 at 08:15