Downloading too fast and locking up the server - relying on WebClient.DownloadFileCompleted to tell us when DownloadFileAsync is done

Question

Background:

Each year we run and then archive all the reports from a particular ASP.Net application. The archive provides a ‘snapshot’ of what the system’s data looked like at a particular time of year. We’ve built a GUI using .Net Forms that uses System.Net.WebClient to call the report server and download the reports to a particular directory. This has worked well in the past.

This year we are including the Excel files in our archive. The Excel files are created by an .aspx page (Windows Sever 2003, IIS6, .Net 4.0) that takes in a querystring, then returns an Excel file. This works great for 100 or so Excel files, but after that we start having problems. We archive approximately 300,000 files each year.

Mechanics:

We use WebClient.DownloadFileAsync to pull down our files, so as not to lock the UI thread. We rely on the WebClient.DownloadFileCompleted event to tell us when each file is done downloading. When DownloadFileCompleted is raised we then start downloading the next file.

Problem:

We’re locking up the webserver. Each file is only taking a fraction of a second to download and after approximately 167 files the webserver locks up (pages timeout) and the archive process pauses for several minutes. Then the archive process downloads another 100 files or so and stalls again for several minutes. This continues for several hours until the archive process starts to crawl with one file every minute or so.

It appears that IIS6 is running out of threads, but how can we stop this from happening?

The following is a slimmed down version of the code we’re running. I’ve stripped out logging and other items not related to our problem. Does anyone have any tips?

public class DownloadExample
{
    private WebClient _WebClient = new WebClient();

    public string DownloadDirectory { get; set; }
    public List<Report> ReportList { get; set; }

    /// <summary>
    /// Constructor - sets all the attributes needed to access the report server, download, and archive the reports
    /// </summary>
    /// <param name="userName">Username</param>
    /// <param name="userPassword">Password for the user's domain username</param>
    /// <param name="userDomain">Domain of the username</param>
    /// <param name="downloadDirectory">Network path where the files will be archived</param>
    public DownloadExample(string userName, string userPassword, string userDomain, string downloadDirectory, List<Report> reportList)
    {
        DownloadDirectory = downloadDirectory;
        _WebClient.Credentials = new NetworkCredential(userName, userPassword, userDomain);
        _WebClient.DownloadFileCompleted += new System.ComponentModel.AsyncCompletedEventHandler(WebClient_DownloadFileCompleted);
        ReportList = reportList;
    }

    /// <summary>
    /// Kicks off the archive process
    /// </summary>
    public void StartDownloading()
    {
        if (ReportList.Count > 0)
        {
            Report rpt = ReportList[0];
            DoDownload(rpt.URL, CreateFileName(rpt), rpt.ReportTitle, rpt.ReportFormatType);
        }
    }

    /// <summary>
    /// Run the report and then download it to the archive directory
    /// </summary>
    /// <param name="url">URL of the Report</param>
    /// <param name="fileName">File name used to name the report file once it is downloaded</param>
    /// <param name="folderName">Name of the folder where the report will be downloaded to</param> 
    /// <param name="reportFormatType">Type of report being run, PDF or Excel</param>
    private bool DoDownload(string url, string fileName, string folderName, ReportFormatTypes reportFormatType)
    {
        bool isSuccess = false;

        string folderPath = DownloadDirectory + "\\" + folderName;
        DirectoryInfo dir = new DirectoryInfo(folderPath);
        if (!dir.Exists)
        {
            dir.Create();
            dir = null;
            dir = new DirectoryInfo(folderPath);
        }

        if (dir.Exists)
        {
            string path = folderPath + "\\" + fileName + ".xls";
            System.Uri uri = new Uri(url);
            try
            {
                _WebClient.DownloadFileAsync(uri, path);
            }
            catch (Exception exp)
            {
                //log error
            }
            FileInfo file = new FileInfo(path);
            isSuccess = file.Exists;
        }
        return isSuccess;
    }

    /// <summary>
    /// This event is fired after a file is downloaded
    /// After each file is downloaded, we remove the downloaded file from the list, 
    /// then download the next file.
    /// </summary>
    void WebClient_DownloadFileCompleted(object sender, System.ComponentModel.AsyncCompletedEventArgs e)
    {
        //Remove the report that was just run
        ReportList.RemoveAt(0);
        if (ReportList.Count > 0)
        {
            //Download the next report
            Report rpt = ReportList[0];
            DoDownload(rpt.URL, CreateFileName(rpt), rpt.ReportTitle, rpt.ReportFormatType);
        }
    }

    /// <summary>
    /// Does a bunch of stuff to create the file name...
    /// </summary>
    string CreateFileName(Report rpt)
    {
        return rpt.FileName;
    }
}