multithreading program with webrequests c#

Question

I want to make multithreading asynchronous download manager. But i have problems with multithreading. One thread correctly works, but when I create second thread - works nothing. I assume that a problem with synchronism of webrequest. I read this answer Multithreading a large number of web requests in c#, but I didn't understand completely. Now question: How can I modify a code to use a multithreading(Thread, Threadpool).

class DownloadableContent:

{
    private string url { get; set; }
    private string path { get; set; }
    private Stream streamResponse { get; set; }
    private Stream streamLocal { get; set; }
    private HttpWebRequest webRequest { get; set; }
    private HttpWebResponse webResponse { get; set; }

    public DownloadableContent(string url, string path)
    {
        this.url = url;
        this.path = path;
    }

    public void Download()
    {
        using (WebClient wcDownload = new WebClient())
        {
            try
            {
                webRequest = (HttpWebRequest)WebRequest.Create(url);                 
                webRequest.Credentials = CredentialCache.DefaultCredentials;
                webResponse = (HttpWebResponse)webRequest.GetResponse();
                Int64 fileSize = webResponse.ContentLength;
                streamResponse = wcDownload.OpenRead(url);
                streamLocal = new FileStream(path, FileMode.Create, FileAccess.Write, FileShare.None);
                byte[] downBuffer = new byte[2048];     
                int bytesSize = 0;
                while ((bytesSize = streamResponse.Read(downBuffer, 0, downBuffer.Length)) > 0)
                {
                    streamLocal.Write(downBuffer, 0, bytesSize);
                }
            }
            finally
            {
                streamResponse.Close();
                streamLocal.Close();
            }
        }
    }       
}

And class main:

DownloadableContent file = new DownloadableContent("url", @"path");  
Thread thread = new Thread(file.Download);
thread.Start();

Your code is very confusing. You're using `WebClient` *and* `HttpWebRequest`. As a result, you're making two requests for every URL. You should remove the `webRequest` and `webResponse`, and just use `WebClient`. — Jim Mischel, Sep 26 '14 at 16:23
Very likely the reason you're having trouble is that you're not disposing of the `webRequest` and `webResponse` objects that you're creating. I know from experience that this often causes things to stop working after just a few requests. Since you're not using them, just remove them from your code. — Jim Mischel, Sep 26 '14 at 16:25

Pedro Augusto Freitas de Souza · Answer 1 · 2014-09-26T13:50:22.423

The best advice that I can give for you, is to use TPL. It's one good library from Microsoft to manager threads. I had used this for a similar problems of yours in my code, basically I had to download 8000 urls, in the begin the normal process was taking 30 minutes. After I used this library, the same process was finished in 30 seconds.

TPL LINK

Task Parallel Library (TPL)

Please, get a look in the example:

BookStore- GitHub

dariogriffo · Accepted Answer · 2014-09-26T13:21:17.337

It is worth for you to read Async programming then take a look at this http://msdn.microsoft.com/en-us/library/System.Net.WebClient_methods(v=vs.110).aspx You have method to download thing asynchronously. Also take a look at TPL and avoid threads http://msdn.microsoft.com/en-us/library/dd460717(v=vs.110).aspx

Probably reading a little more will help you avoid lots of headaches.

This is an easy example

private static void Main(string[] args)
{
    var urlsAndPaths = new Dictionary<string, string>();
    urlsAndPaths.Add("http://i.forbesimg.com/media/lists/people/lionel-messi_416x416.jpg","messi.jpg");
    urlsAndPaths.Add("http://sizzlingsuperstars.com/wp-content/uploads/2014/07/Cristiano-Ronaldo-2-480x309.jpg", "cristiano.jpg");            
    foreach (var kvp in urlsAndPaths)
    {
        var wc = new WebClient();
        wc.DownloadFileAsync(new Uri(kvp.Key),kvp.Value);
    }
    Console.ReadKey();
}

thanks for answer, looks correctly, but program condition is "New Download in New Thread" (training task) using Thread and Threadpool. (two differents realizations) — Kuchur Andrei, Sep 26 '14 at 13:24

Cameron Woodall · Answer 3 · 2014-09-26T13:36:58.480

Depending on which version of the .NET framework you are using, you could take advantage of the Task class. You could do something like this

    foreach (var uri in myCollection)
    {
        Task.Factory.StartNew(() =>
        {
            try
            {
                using (System.Net.WebClient client = new System.Net.WebClient())
                {
                    client.DownloadFileCompleted += (o, args) =>
                    {
                        //Do something with the download
                    };
                    client.DownloadFileAsync(uri);

                }
            }
            catch (Exception ex)
            {
                //Do something
            }
        });
    }

Todd Menier · Answer 4 · 2014-09-26T13:59:33.270

0

Are you on .NET 4.5? The "new way" to do this is with Task.WhenAll, which gets your downloads going asynchronously and in parallel but allows the framework to decide if/when work should be scheduled to the thread pool.

var client = new System.Net.WebClient();
await Task.WhenAll(urls.Select(url => {
    var path = ?? // build local path based on url?
    return client.DownloadFileAsync(url, path);
});

edited Sep 26 '14 at 13:59

answered Sep 26 '14 at 13:53

Todd Menier

37,557
17
150
173

multithreading program with webrequests c#

4 Answers4