Improve speed of splitting file

Question

I am using this code to extract a chunk from file

// info is FileInfo object pointing to file
var percentSplit = info.Length * 50 / 100; // extract 50% of file
var bytes = new byte[percentSplit];
var fileStream = File.OpenRead(fileName);
fileStream.Read(bytes, 0, bytes.Length);
fileStream.Dispose();
File.WriteAllBytes(splitName, bytes);

Is there any way to speed up this process?

Currently for a 530 MB file it takes around 4 - 5 seconds. Can this time be improved?

Answer 1

There are several cases of you question, but none of them is language relevant.

Following are something to concern

• What is the file system of source/destination file?
• Do you want to keep original source file?
• Are they lie on the same drive?

In c#, you almost do not have a method could be faster than File.Copy which invokes CopyFile of WINAPI internally. Because of the percentage is fifty, however, following code might not be faster. It copies whole file and then set the length of the destination file

var info=new FileInfo(fileName);
var percentSplit=info.Length*50/100; // extract 50% of file

File.Copy(info.FullName, splitName);
using(var outStream=File.OpenWrite(splitName))
    outStream.SetLength(percentSplit);

Further, if

1. you don't keep original source after file splitted
2. destination drive is the same as source
3. your are not using a crypto/compression enabled file system

then, the best thing you can do, is don't copy files at all. For example, if your source file lies on FAT or FAT32 file system, what you can do is

1. create new dir entry(entries) for newly splitted parts of file
2. let the entry(entries) point(s) to the cluster of target part(s)
3. set correct file size for each entry
4. check for cross-link and avoid that

If your file system was NTFS, you might need to spend a long time to study the spec.

Good luck!

Answer 2

var percentSplit = (int)(info.Length * 50 / 100); // extract 50% of file
var buffer = new byte[8192];
using (Stream input = File.OpenRead(info.FullName))
using (Stream output = File.OpenWrite(splitName))
{
    int bytesRead = 1;
    while (percentSplit > 0 && bytesRead > 0)
    {
        bytesRead = input.Read(buffer, 0, Math.Min(percentSplit, buffer.Length));
        output.Write(buffer, 0, bytesRead);
        percentSplit -= bytesRead;
    }
    output.Flush();
}

The flush may not be needed but it doesn't hurt, this was quite interesting, changing the loop to a do-while rather than a while had a big hit on performance. I suppose the IL is not as fast. My pc was running the original code in 4-6 secs, the attached code seemed to be running at about 1 second.

Answer 3

I get better results when reading/writing by chunks of a few megabytes. The performances changes also depending on the size of the chunk.

FileInfo info = new FileInfo(@"C:\source.bin");
FileStream f = File.OpenRead(info.FullName);
BinaryReader br = new BinaryReader(f);

FileStream t = File.OpenWrite(@"C:\split.bin");
BinaryWriter bw = new BinaryWriter(t);

long count = 0;
long split = info.Length * 50 / 100;
long chunk = 8000000;

DateTime start = DateTime.Now;

while (count < split)
{
    if (count + chunk > split)
    {
        chunk = split - count;
    }

    bw.Write(br.ReadBytes((int)chunk));
    count += chunk;
}

Console.WriteLine(DateTime.Now - start);