Checking URLs in multiple Threads

Question

I'm trying to validate a List of URLs using the following Code

class MyClient : WebClient
{
    public bool HeadOnly { get; set; }

    protected override WebRequest GetWebRequest(Uri address)
    {
        WebRequest req = base.GetWebRequest(address);

        if (HeadOnly && req.Method == "GET")
        {
            req.Method = "HEAD";
        }
        return req;
    }
}

private static Boolean CheckURL(string url)
{
    using (MyClient myclient = new MyClient())
    {
        try
        {
            myclient.HeadOnly = true;
            // fine, no content downloaded
            string s1 = myclient.DownloadString(url);
            return true;
        }
        catch (Exception error)
        {
            return false;
        }
    }
}

This works well, but for some websites, it involves a waiting time to get the results. How can I split the URL List to 2 or maybe 4 parts and Validate each part using a separate thread and report progress to single progress bar?

Please advice.

Update:

I'm validating the URl's using a Backgroundworker

private void URLValidator_DoWork(object sender, DoWorkEventArgs e)
{
    foreach (var x in urllist)
    {         
        Boolean valid = CheckURL(x);
    }
}

Where is the part of your code, where you iterate over a list of URLs? — Romano Zumbé, Jul 11 '17 at 07:39
see if [this](https://stackoverflow.com/questions/11582211/c-sharp-multi-threaded-app-structure) helps — Masoud Andalibi, Jul 11 '17 at 07:45

score 3 · Accepted Answer · answered Jul 11 '17 at 07:52

3

You could use Parallel.ForEach for this:

using System.Threading.Tasks;

private void URLValidator_DoWork(object sender, DoWorkEventArgs e)
{
    Parallel.ForEach(urllist, (url) =>
    {
        Boolean valid = CheckURL(x);
        // Do something with the result or save it to a List/Dictionary or ...
    });
}

answered Jul 11 '17 at 07:52

Romano Zumbé

7,893
4
33
55

Thanks a lot,this works well.But to prevent UI Freezing i had to put this code within a background worker. – max Jul 11 '17 at 08:49
@max Task.Run will be simpler syntax – Lei Yang Jul 11 '17 at 09:07

Checking URLs in multiple Threads

1 Answers1