Faster processing of large files

Question

//string filename = "123.txt";
foreach (var files in Directory.GetFiles(pathToDir, "*.*", SearchOption.TopDirectoryOnly))
using (var fsIn = new FileStream(files, FileMode.Open, FileAccess.Read))
using (var fsOut = new FileStream($"{files}.crypt", FileMode.CreateNew, FileAccess.Write))
using (var aes = Aes.Create())
using (var enc = aes.CreateEncryptor(new byte[16] /* key */, new byte[16] /* vector */))
using (var cs = new CryptoStream(fsIn, enc, CryptoStreamMode.Write))
    cs.CopyTo(fsOut);

It takes a long time to process large files, read that there is a MemoryMappedFile method that can process large files, tell me how you can use it in the code above ??

How long does it take? Which part of the code is taking time? Does processing the files in parallel help? Does `EnumerateFiles` help? — mjwills, Sep 07 '20 at 00:38

score 2 · Answer 1 · edited Feb 06 '21 at 15:49

A MemoryMappedFile is not really going to do much for you unless you need it all in memory at once.

Your bottle-neck is likely IO not the CPU, as such I would look to playing with the BufferSize and setting SequentialScan on the FileStreams which may help reading and writing in larger chunks to the internal buffer (especially for modern SSDs), and better optimizing the OS file cache (respectively)

FileStream(String, FileMode, FileAccess, FileShare, Int32, FileOptions)

bufferSize

A positive Int32 value greater than 0 indicating the buffer size. The default buffer size is 4096.

FileOptions Enum

SequentialScan

Indicates that the file is to be accessed sequentially from beginning to end. The system can use this as a hint to optimize file caching. If an application moves the file pointer for random access, optimum caching may not occur; however, correct operation is still guaranteed. Specifying this flag can increase performance in some cases.

Also note, that reading and writing files and processing them in parallel may work to a certain extent (though usually won't all that much if you are waiting on IO), it really depends on how well your device controller deals with queue-depth and how much bottle-neck is caused by the CPU. But once again you will need to play with it and its degrees of parallelism for your situation.

Enigmativity · Answer 2 · 2020-09-07T01:21:10.373

1

Can you please try Nugetting "System.Reactive"? Add in the reference to System.Reactive.linq and then try this:

var query =
    Directory
        .EnumerateFiles(pathToDir, "*.*", SearchOption.TopDirectoryOnly)
        .ToObservable()
        .SelectMany(files =>
            Observable.Using(() => new FileStream(files, FileMode.Open, FileAccess.Read), fsIn =>
                Observable.Using(() => new FileStream($"{files}.crypt", FileMode.CreateNew, FileAccess.Write), fsOut =>
                    Observable.Using(() => Aes.Create(), aes =>
                        Observable.Using(() => aes.CreateEncryptor(rgbKey: new byte[16], rgbIV: new byte[16]), enc =>
                            Observable.Using(() => new CryptoStream(fsIn, enc, CryptoStreamMode.Read), cs => Observable.FromAsync(() => cs.CopyToAsync(fsOut))))))));
                            
query.ToArray().Wait();

If that runs faster then you have some benefit from parallel processing. If not then you are IO bound and running parallel isn't going to help.

In any case, I don't see where memory mapped files will help.

edited Sep 07 '20 at 01:21

answered Sep 07 '20 at 01:06

Enigmativity

113,464
11
89
172

Nice work. If you have a test set up, just curious what happens if you set a larger buffer size (if ssd, 10k or something) and the FileModeOptions to sequential (with your parallel test) – TheGeneral Sep 07 '20 at 01:21
@MichaelRandall - I tried changing the settings but couldn't get anything other than 16 & 16 to run - also could find where to set the mode to sequential? – Enigmativity Sep 07 '20 at 01:24
sequential can be set in this constructor in the end options https://learn.microsoft.com/en-us/dotnet/api/system.io.filestream.-ctor?view=netcore-3.1#System_IO_FileStream__ctor_System_String_System_IO_FileMode_System_IO_FileAccess_System_IO_FileShare_System_Int32_System_IO_FileOptions_ all good though, was just curious – TheGeneral Sep 07 '20 at 01:27

Faster processing of large files

2 Answers2