0

enter image description hereI am trying to create dynamically created, downloadable ZIP files. The files are created correctly on the server, however when they are sent to the client using "Response.OutputStream" the file becomes corrupted. This only seems to happen when the zip file is over 4GB. Does anyone have any idea why this is?

The exact code I am using is:

       string path = @"C:\temp\vid";

        Response.BufferOutput = false; // Disable Buffer Output to start the download immediately

        // Set custom headers to force browser to download the file instad of trying to open it
        Response.ContentType = "application/x-zip-compressed";
        Response.AppendHeader("content-disposition", "attachment; filename=Archive.zip");

        ZipOutputStream zipOutputStream = new ZipOutputStream(Response.OutputStream, 20000);
        zipOutputStream.SetLevel(0); // No compression
        zipOutputStream.UseZip64 = UseZip64.On;//Forces Zip64 to be used
        zipOutputStream.IsStreamOwner = true;

        try
        {
            foreach (string file in Directory.GetFiles(path, "*.*", SearchOption.AllDirectories))
            {
                using (var fs = System.IO.File.Open(file, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
                {
                    ZipEntry entry = new ZipEntry(ZipEntry.CleanName(Path.GetFileName(file))); //Create zip compatible name
                    zipOutputStream.PutNextEntry(entry); //Adds entry to list

                    fs.CopyTo(zipOutputStream);

                    zipOutputStream.CloseEntry();
                    zipOutputStream.Flush();

                    Response.Flush();
                    Response.Clear();
                }
            }

            zipOutputStream.Finish();
            zipOutputStream.Close();
            Response.End();
            Response.Flush();

        }
        catch
        {
            Debug.WriteLine("Connection Closed or error");
        }

        return new HttpStatusCodeResult(HttpStatusCode.OK);
Christopher Vickers
  • 1,773
  • 1
  • 14
  • 18
  • 1
    Well, for starters, sending a 4GB file over HTTP might not be the best approach. Which OS (specifically, which file system) are you using? – 3Dave May 24 '18 at 23:13
  • Its Server 2016. Are you suggesting using FTP to download it? The problem is that FTP uses different ports so less likely to work with Firewalls. The benefit of using HTTP is that all the authentication is already taken care of. – Christopher Vickers May 24 '18 at 23:17
  • 1
    Or maybe break it up into multiple files. I'm not certain why you have a `Response.Clear()` within your `using(...)` block, either. Once you've done a `Flush`, all the output should be sent (or buffered to send by the server). The next question would be, how much RAM does the server have available? (The application, not the machine.) And hopefully there aren't multiple clients trying to do this at the same time,. – 3Dave May 24 '18 at 23:20
  • The whole idea is to package thousands of files so by splitting it down into smaller ones im adding extra work. The Clear was me trying to clear the OutputStream because the data being downloaded is corrupt. It's kind of like the OutputStream reached a 32bit limit and sends error messages afterwards. – Christopher Vickers May 24 '18 at 23:24
  • 1
    There's a reason sites like, say, Microsoft.com use download managers for massive files. Using this approach, if 10 people are downloading the file at the same time, you'll need *at least* 40GB of RAM (in reality, far more than that) available. This is not a great solution for files of this size. – 3Dave May 24 '18 at 23:25
  • You don't need the RAM because of the way ZIP files are created, assuming you don't have compression. It literally packages the files and sends them one after another, storing only the file length and CRC data which is stuck on the end. This works great until it gets to the 32bit limit, which is where im at now. – Christopher Vickers May 24 '18 at 23:28
  • 1
    It might be worse than @Dav3 mentioned (or not). I have not researched the internals of ZipOutputStream or Response.outputStream, but its possible that the entire stream needs to be allocated contiguously in memory. Something to research. – Sam Axe May 24 '18 at 23:29
  • 1
    The server still needs to buffer the data to be sent, and depending on the download speed, it could very well build the vast majority of the archive in memory, and have to hold it, while it's being dripped down to the client. Calling `Response.Flush()` doesn't magically eliminate bandwidth limitations. `Flush()` is not a blocking call - you're still shoving stuff into memory, and IIS is happily hanging on to that data while trying to pass it on to the client. – 3Dave May 24 '18 at 23:30
  • I'd love to see a profiler log showing memory consumption while you're doing this. – 3Dave May 24 '18 at 23:32
  • Have a look at https://stackoverflow.com/questions/28235804/compress-large-file-using-sharpziplib-causing-out-of-memory-exception, which shows a way to get around this. And, look at the 'corrupted' file in a hex editor (Notepad++ also works) and verify that the end of the file isn't full of zeros, and that it is of the expected length. It's also possible that the browser is "hanging up" if it doesn't have a valid `content-length` header, which isn't possible using this method. – 3Dave May 24 '18 at 23:34
  • Thanks for the help guys. – Christopher Vickers May 24 '18 at 23:38
  • I was incorrect- `Flush()` *is* blocking, which validates your memory consumption statement. Hmmm. – 3Dave May 24 '18 at 23:41
  • Someone asked to see the memory consumption. This was a 3.9GB file that assembled a number of files. Garbage Collection hasn't started but once it does RAM usage drops rapidly. I checked the end of the file and there is definitely content, however it is not the correct type. The end of ZIP files normally has groups of zeros but the downloaded ZIP has none – Christopher Vickers May 24 '18 at 23:49

0 Answers0