3

I was trying to copy a 5 GB ISO file onto a 32 GB flash drive with 29 GB of free space.

Windows 7 refused to let me drag-and-drop the file onto the flash drive, reporting the file was too large for the destination file system.

I eventually learned this was because the drive was formatted as FAT32 instead of NTFS, but not before I wrote this routine to copy the file over:

private void copyFile(string from, string to) {
  bool ok = true;
  using (StreamReader sr = new StreamReader(from, true)) {
    using (StreamWriter sw = new StreamWriter(to, false, sr.CurrentEncoding)) {
      int FOUR_K = 4048;
      char[] buf = new char[FOUR_K];
      try {
        while (-1 < sr.Peek()) {
          int len = sr.Read(buf, 0, FOUR_K);
          sw.Write(buf, 0, len);
          sw.Flush();
        }
      } 
      catch (Exception err) {
        ok = false;
        throw err;
      }
    }
  }
  if (ok) {
    Console.WriteLine("Done!");
  }
}

I let it run for about an hour, and it only got to 270 MB in file size.

What's going on?

What in my code is causing my file to take so long?

Is it my choice of the FOUR_K variable size?

[UPDATE]

I have two ISO files: Win8-32bit at ~3 GB and Win8-64bit-Developer at ~5 GB. Using drag-and-drop, Windows Explorer copied the 3 GB file to my 32 GB flash drive in about three minutes.

Using Marc Gravell's technique, I went at this yet again:

[STAThread]
static void Main(string[] args) {
  using (OpenFileDialog dlg = new OpenFileDialog()) {
    dlg.Title = "Select File";
    if (dlg.ShowDialog() == DialogResult.OK) {
      using (FolderBrowserDialog fdg = new FolderBrowserDialog()) {
        fdg.Description = "Select Destination";
        if (fdg.ShowDialog() == DialogResult.OK) {
          DateTime start = DateTime.Now;
          Console.WriteLine("Started at {0:g}.\nWorking...", start);
          using (FileStream fin = File.Open(dlg.FileName, FileMode.Open)) {
            using (FileStream fout = new FileStream(Path.Combine(fdg.SelectedPath, dlg.SafeFileName), FileMode.Create)) {
              try {
                fin.CopyTo(fout);
              } catch (Exception err) {
                Console.WriteLine("An Error Occurred:");
                Console.WriteLine(err.Message);
              }
            }
          }
          DateTime end = DateTime.Now;
          TimeSpan span = (end - start);
          Console.WriteLine("Process Ended at {0}.\nThe total minutes passed = {1}.", end, span.TotalMinutes);
          Console.WriteLine("Press Any Key.");
          Console.ReadKey();
        }
      }
    }
  }
}

Using the FileStream instances above, the program ran for about 8 hours, copied exactly 4,194,300 KB and then threw an Out Of Memory Exception.

Community
  • 1
  • 1
  • 2
    For one thing, do **not** use a `StreamWriter : TextWriter` to handle binary files... Your ISO would have become invalid. – H H Feb 01 '12 at 20:22
  • Do you base the file size on the likey corrupted output file, or by an internal counter from you program? It would be interesting if you reran it but kept track of a how many blocks you'd written with a variable. – Joshua Honig Feb 01 '12 at 20:23
  • To make this a meaningful question. you should compare it with another method, and with another disk(-type). How did you determine the 270 k? – H H Feb 01 '12 at 20:24
  • @jmh_gr and Henk: I can see the file in Windows Explorer while my code runs. It is slowly ...slowly getting bigger. –  Feb 01 '12 at 20:27
  • As noted: 1, don't flush every time, 2, 4k is 4096, 3, don't create a new buffer every time, you can reuse buf each time, 4, i'd use a much larger buffer than 4K, perhaps 4k*1024, and StreamWriter is the wrong writer here. – Joe Feb 01 '12 at 20:29
  • Consider running it without the sw.Flush() and with a much larger size than 4k. You're effectively trying to outguess any optimizations the drive and OS will make on your behalf as it is now. – GWLlosa Feb 01 '12 at 20:21
  • Even with the 4K block and flush after every block, it doesnt make sense that it would do 270K in an HOUR. That means the total number of iterations for the while loop was ~68. – Abdul Hfuda Feb 01 '12 at 20:26
  • @Xeno : that 270k is also in doubt. – H H Feb 01 '12 at 20:27
  • @Henk Holterman: The word 'consider' was meant as more of a polite linguistic construct rather than a lack of confidence that the Flush() is causing performance problems. Even with that happening, it shouldn't have been THAT slow, but removing the Flush() to a position after the loop WILL cause a speedup. – GWLlosa Feb 01 '12 at 20:28
  • What file size would you recommend I try? (FYI: I know it will not work now because it is Binary and StreamReader/Writer is Text) –  Feb 01 '12 at 20:32
  • 2
    @jp2code, I suppose main problem lies in using `StreamReader/Writer` since they try to encode/decode the input bytes using some encoding(AFAIK defaults to uft-8) (corruption of data is mentioned before) – L.B Feb 01 '12 at 20:43

3 Answers3

8

I don't know about the performance issue (sounds very odd), but there's no reason the use a StreamReader / StreamWriter here at all, since you could just copy at the binary level. Indeed, an ISO image is not text, so reading it into char data is very likely to corrupt things. For info, even if you don't want to use File.Copy, all you need here is:

using(var inFile = File.OpenRead(source))
using(var outFile = File.Create(destination))
{
    inFile.CopyTo(outFile);
}

To get 270k in an hour you need to try hard (unless the IO is basically dead); my guess is it threw an error somewhere.

Marc Gravell
  • 1,026,079
  • 266
  • 2,566
  • 2,900
  • Hey Mark. As I suspected, I was running into the 4GB filesize limit. That is why I was reading in chunks. Would `CopyTo` be able to get around a 4GB limit? –  Feb 01 '12 at 20:30
  • 1
    @jp2code `CopyTo` is just a `Stream` `Read`/`Write` loop, so "yes probably". You can tell it the buffer-size to use if you want (via an overload). – Marc Gravell Feb 01 '12 at 20:32
  • Looks like this is just going to be slow. Using your code, it has been running for 4 minutes and it is only up to 114MB. (oops. Just noticed my original question should say 270MB, not 270KB) –  Feb 01 '12 at 20:50
  • 2
    @jp2code right... so in 8 minutes it will have done roughly what you did previously in an hour... that's progress. What is the flash device? some are simply very slow! If we approximate "small" with "old", then it will also be slow... I've seen some crazy slow devices, and some crazy fast. – Marc Gravell Feb 01 '12 at 20:53
  • I have 2 ISO files: Win8-32bit at ~3GB and Win8-64bit-Developer at ~5GB. Using Drag-N-Drop, Windows Explorer copied the 3GB file to my 32GB flash drive in about 3 minutes. Using the `FileStream` instances above, the program ran for about 8 hours, copied exactly `4,194,300 KB`, then threw an *Out Of Memory* Exception. –  Feb 02 '12 at 15:16
  • The only way around this problem is to format the flash drive as NTFS; FAT32 has a 4GB limit, and any copying method is going to fail if it results in a file > 4GB on a FAT32 drive. – Joe Feb 03 '12 at 17:33
3

I suspect it's the Flush call every time.

If you are using .NET 4, Stream now has a CopyTo method:

using (var input = File.OpenRead(fromFile))
{
  using (var output = File.OpenWrite(toFile))
  {
    input.CopyTo(output);
  }
}
Michael Edenfield
  • 28,070
  • 4
  • 86
  • 117
  • 1
    true, in practice I would use the appropriate 'using' blocks, I was just trying to demonstrate the method. I'll update the answer to be better :) – Michael Edenfield Feb 01 '12 at 20:33
  • Thanks :) It's just that, given the number of people who will just copy-paste code from an answer, I think it's never a good idea to have bad code in an example. – porges Feb 01 '12 at 20:39
2

If you are trying to copy a binary file, why do you use StreamReader/StreamWriter? Try something like this:

using (FileStream source = File.Open(@"c:\Source\data.iso", FileMode.Open))
{
    FileStream destination = new FileStream(@"F:\Desitnation\data.iso",FileMode.OpenOrCreate));

    source.CopyTo(destination);
    destination.Close();
}
Mithrandir
  • 24,869
  • 6
  • 50
  • 66