C# Create 10 parallel web requests at a time

Asked May 26 '12 at 12:28

Active May 26 '12 at 12:35

Viewed 92 times

Possible Duplicate:
Multithreading a large number of web requests in c#

I am working on a scraper and it has to scrape about 12000 pages from a website. Right now I only get 1500 pages an hour which is too slow.

I wanted to know how I can request 10 pages at once rather than 1?

private void backgroundWorker1_DoWork(object sender,
    DoWorkEventArgs e)
{ 
    //Page IDs - Start request loops
    for (int i = 0; i < 12000; i++)
    {
        Get(url + i);
    }
}

An example on the code will be great

Thanks!

edited May 23 '17 at 10:34

Community

asked May 26 '12 at 12:28

Zbone

1

checkout out the parallel answer here: http://stackoverflow.com/questions/4277844/multithreading-a-large-number-of-web-requests-in-c-sharp – bryanmac May 26 '12 at 12:34
Use `Parallel.For`: http://msdn.microsoft.com/en-us/library/dd783539.aspx. – Steven May 26 '12 at 12:35
It's great but now i get Probable I/O race condition detected while copying memory on my stremwriter... – Zbone May 26 '12 at 13:13
Post the code where use the streamwriter – paparazzo May 26 '12 at 13:25

C# Create 10 parallel web requests at a time

0 Answers0