0

I have a windows service that crawls web pages using multi-threading, occasionally I am getting connections resets due to the TCP window becoming full. I suspect it has something to do with the page faults that are occurring around the time of the connection resets. I think the delay caused by the page fault in a thread closes the network connection. Is it possible to somehow cause page faults as the web pages are being crawled so I can prove my theory?

Imran Azad
  • 1,008
  • 2
  • 12
  • 30
  • 1
    To provoke page faults, you could create a list of objects. If you store 1 MByte StringBuilder objects in the list, you can easily calculate the number of objects needed to fill your virtual address space. Then iterate through the list and search the strings. This should spawn paging activity. You might also want to use task manager, process monitor or process explorer to identify resource hungry processes. – Axel Kemper Jan 09 '15 at 23:13
  • @AlexKemper, unfortunately string builder doesn't seem to be available for .net 4.0. Any other alternatives I could use? – Imran Azad Jan 09 '15 at 23:57
  • 1
    FWIW, under normal conditions, a connection timeout should be an eon compared to the time it takes to process a page fault. – 500 - Internal Server Error Jan 10 '15 at 00:03
  • Rather than using StringBuilder you could use an own class with an array member variable of 256*1024 integers (= total size 1 MByte). – Axel Kemper Jan 10 '15 at 00:14
  • 1
    What Axel just said, but StringBuilder is definitely also available for .NET 4.0. – 500 - Internal Server Error Jan 10 '15 at 00:16
  • @500-InternalServerError Ah sorry my bad. – Imran Azad Jan 10 '15 at 00:20
  • Why do you think that the TCP window becoming full has anything to do with page faults? – John Saunders Jan 10 '15 at 01:07
  • @JohnSaunders The reason being that my understanding is that when a page fault occurs the thread that experiences the page fault is halted, I suspect the time it's halted for causes the TCP window to become full because data is being sent faster than it can be read. – Imran Azad Jan 10 '15 at 15:46
  • Only a hard page fault would take long enough to cause issues, and the window shouldn't become that full that fast in any case. What period of time are you suspecting can cause the window to become full? Also, TCP is a fairly robust protocol. The fact that the window becomes full should not cause a problem. Data would simply no longer be sent - only ACK packets. – John Saunders Jan 10 '15 at 21:10

0 Answers0