Best way to write byte[] into binary file

Question

I have a 1GB in server A, I must use Read() method to get it to local. It is splitted into packages (16517 package). Each package is 65536 bytes include header.

Current, I use BinaryWriter to write each package into file directly. But it took me over 5 minutes (323642 ms).

I try to join some packages into byte[] of 10MB, but it does not reduce time(317397 ms).

Which way is the best to write byte[] into binary file?

Update

const ulong Memory_10MB = 10485760; // 10MB
do {
    byte[] packetData = Read(&totalSize, /*Other parameters*/ ,&hasFollowing);
    if (package == null) {
        // Process error
        return;
    }
    dataSize += (ulong)packetData.Length;
    if (dataSize >= Memory_10MB)
    {
        if (binaryWriter == null)
        {
            path = Path.GetTempFileName();
            fileStream = new FileStream(path, FileMode.Append, FileAccess.Write, FileShare.Read);
            binaryWriter = new BinaryWriter(fileStream);
            if (responseData.Length > 5)
            {
                binaryWriter.Write(responseData.Skip(5).ToArray());
            }
        }           
        binaryWriter.Write(packetData);
    }
    else
    {
        responseData = Utilities.ConcatArrays(responseData, packetData);
    }

} while (dataSize < totalSize);
// Process after got data

If you just copy the file using Windows Explorer how long does it take? Whatever it is I wouldn't expect you could improve on it much. — Adam Benson, Mar 03 '17 at 11:34
First, my output file is under `Temp` folder. I copy it into another drive. It take about 3 seconds — GSP, Mar 03 '17 at 11:37
your benchmarks with 64k and 10M chunk sizes show that the per-packet overhead is low. how much of the time can be attributed to network lag? do you open/seek/close between packages? — Cee McSharpface, Mar 03 '17 at 11:39
What time does it take to copy the entire file in explorer or to read the entire file into memory? — Andrii Litvinov, Mar 03 '17 at 11:55
@AndriiLitvinov What do you mean? Is it use File.Copy or File.Move? File.Move took me about 3 seconds — GSP, Mar 03 '17 at 11:56
Given this code, how can you distinguish between the time for `Read()` and that for `Write()` ? I suspect the Read() is responsible for most of those 5 minutes. — H H, Mar 03 '17 at 11:58
@I would try to drag file in explorer and see what it takes. Alternatively try to read entire file into `responseData` before writing anything to disk. — Andrii Litvinov, Mar 03 '17 at 11:58
responseData.Skip(5).ToArray() allocates a big array. Wouldn't it be better to do Write(responseData, 5, responsedata.Length-5)? (Assuming that responseData is a byte array). Edit: I only noticed now that it's not part of the loop, so performance gain is low. — Palle Due, Mar 03 '17 at 12:00

H H · Answer 1 · 2017-03-06T10:01:52.727

1

Which way is the best to write byte[] into binary file?

Directly to a FileStream. The BinaryWriter just adds a little overhead.

The package size should (could) be optimized for your hardware but in general the default value should be more than enough.

edited Mar 06 '17 at 10:01

answered Mar 03 '17 at 12:00

H H

263,252
30
330
514

If I join some packages into byte[] then write it into file. Which is better? And how much size of byte[] is better to write into file? – GSP Mar 03 '17 at 12:07
1

I don't know the exact numbers, but it starts at 8k and 16k so your 64k blocks should be fine. You already tried this (10M) and saw no improvement so don't bother. – H H Mar 03 '17 at 12:49

Best way to write byte[] into binary file

1 Answers1