24

I need to download a large file (2 GB) over HTTP in a C# console application. Problem is, after about 1.2 GB, the application runs out of memory.

Here's the code I'm using:

WebClient request = new WebClient();
request.Credentials = new NetworkCredential(username, password);
byte[] fileData = request.DownloadData(baseURL + fName);

As you can see... I'm reading the file directly into memory. I'm pretty sure I could solve this if I were to read the data back from HTTP in chunks and write it to a file on disk.

How could I do this?

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Nick Cartwright
  • 8,334
  • 15
  • 45
  • 56

6 Answers6

39

If you use WebClient.DownloadFile you could save it directly into a file.

Alex Peck
  • 4,603
  • 1
  • 33
  • 37
37

The WebClient class is the one for simplified scenarios. Once you get past simple scenarios (and you have), you'll have to fall back a bit and use WebRequest.

With WebRequest, you'll have access to the response stream, and you'll be able to loop over it, reading a bit and writing a bit, until you're done.

From the Microsoft documentation:

We don't recommend that you use WebRequest or its derived classes for new development. Instead, use the System.Net.Http.HttpClient class.

Source: learn.microsoft.com/WebRequest

Example:

public void MyDownloadFile(Uri url, string outputFilePath)
{
    const int BUFFER_SIZE = 16 * 1024;
    using (var outputFileStream = File.Create(outputFilePath, BUFFER_SIZE))
    {
        var req = WebRequest.Create(url);
        using (var response = req.GetResponse())
        {
            using (var responseStream = response.GetResponseStream())
            {
                var buffer = new byte[BUFFER_SIZE];
                int bytesRead;
                do
                {
                    bytesRead = responseStream.Read(buffer, 0, BUFFER_SIZE);
                    outputFileStream.Write(buffer, 0, bytesRead);
                } while (bytesRead > 0);
            }
        }
    }
}

Note that if WebClient.DownloadFile works, then I'd call it the best solution. I wrote the above before the "DownloadFile" answer was posted. I also wrote it way too early in the morning, so a grain of salt (and testing) may be required.

John Saunders
  • 160,644
  • 26
  • 247
  • 397
  • Thanks for your detailed answer and code snippet! This will be useful in cases when I want to process the data as it arrives! – Nick Cartwright Jul 03 '09 at 13:43
  • what about exception handling or retry mechanism in this code? network disconnect etc. – Zain Shaikh Jun 18 '13 at 07:11
  • 1
    In most cases, the best exception handling is none at all. If you are in a situation where your network is very unreliable, then you may need to add retry logic. I live in the United States, so I suppose I'm spoiled by good network connections, _usually_. When they don't work, things are so bad that retry is not a useful option. – John Saunders Jun 18 '13 at 07:23
  • I am wondering why you chose a buffer size of 16 * 1024. When I tried to increase the size, it seems to still use smaller chunks. Was there any reasoning behind your choice. Just curious. – kns98 Apr 30 '15 at 20:09
  • There was no reason for the choice. – John Saunders Apr 30 '15 at 20:58
  • 1
    iss it possible to use HttpClient here instead of WebRequest – mtkachenko Mar 09 '16 at 18:03
9

You need to get the response stream and then read in blocks, writing each block to a file to allow memory to be reused.

As you have written it, the whole response, all 2GB, needs to be in memory. Even on a 64bit system that will hit the 2GB limit for a single .NET object.

Update: easier option. Get WebClient to do the work for you: with its DownloadFile method which will put the data directly into a file.

Richard
  • 106,783
  • 21
  • 203
  • 265
3

WebClient.OpenRead returns a Stream, just use Read to loop over the contents, so the data is not buffered in memory but can be written in blocks to a file.

Whuppa
  • 81
  • 3
2

i would use something like this

Sadegh
  • 6,654
  • 4
  • 34
  • 44
0

The connection can be interrupted, so it is better to download the file in small chunks.

Akka streams can help download file in small chunks from a System.IO.Stream using multithreading. https://getakka.net/articles/intro/what-is-akka.html

The Download method will append the bytes to the file starting with long fileStart. If the file does not exist, fileStart value must be 0.

using Akka.Actor;
using Akka.IO;
using Akka.Streams;
using Akka.Streams.Dsl;
using Akka.Streams.IO;

private static Sink<ByteString, Task<IOResult>> FileSink(string filename)
{
    return Flow.Create<ByteString>()
        .ToMaterialized(FileIO.ToFile(new FileInfo(filename), FileMode.Append), Keep.Right);
}

private async Task Download(string path, Uri uri, long fileStart)
{
    using (var system = ActorSystem.Create("system"))
    using (var materializer = system.Materializer())
    {
       HttpWebRequest request = WebRequest.Create(uri) as HttpWebRequest;
       request.AddRange(fileStart);

       using (WebResponse response = request.GetResponse())
       {
           Stream stream = response.GetResponseStream();

           await StreamConverters.FromInputStream(() => stream, chunkSize: 1024)
               .RunWith(FileSink(path), materializer);
       }
    }
}
qqus
  • 483
  • 5
  • 11