C# Out of Memory while handling and encrypting large files

Question

Im trying to encrypt a large file (Camtasia.exe) with the AES encryption. Now for some reason I get a "Out of Memory" Exception. Im really new to this and I don't know how I could possibly fix that. This is my code

I use this to call my encryption method.

bytes = File.ReadAllBytes("Camtasia.exe");
Cryptography.Encryption.EncryptAES(System.Text.Encoding.Default.GetString(bytes), encryptionKey);

This is the AES encryption itself

public static string EncryptAES(string content, string password)
    {
        byte[] bytes = Encoding.UTF8.GetBytes(content);

        using (SymmetricAlgorithm crypt = Aes.Create())
        using (HashAlgorithm hash = MD5.Create())
        using (MemoryStream memoryStream = new MemoryStream())
        {
            crypt.Key = hash.ComputeHash(Encoding.UTF8.GetBytes(password));
            // This is really only needed before you call CreateEncryptor the second time,
            // since it starts out random.  But it's here just to show it exists.
            crypt.GenerateIV();

            using (CryptoStream cryptoStream = new CryptoStream(
                memoryStream, crypt.CreateEncryptor(), CryptoStreamMode.Write))
            {
                cryptoStream.Write(bytes, 0, bytes.Length);
            }

            string base64IV = Convert.ToBase64String(crypt.IV);
            string base64Ciphertext = Convert.ToBase64String(memoryStream.ToArray());

            return base64IV + "!" + base64Ciphertext;
        }
    }

Here is the error again that I get when calling the function "EncryptAES" at the top. I would be glad if someone could explain how this happens and how to solve it https://i.stack.imgur.com/Vp9ev.jpg

[decrease-memory-consumption-when-encrypting-files-c-sharp](https://stackoverflow.com/questions/25404097/decrease-memory-consumption-when-encrypting-files-c-sharp) — Ryan Wilson, May 11 '21 at 16:45
Where exactly does it bomb? Is it when trying to read all the bytes? — insane_developer, May 11 '21 at 16:55
If it didn't fail there, it'll probably fail when it tries to turn it into a string, and then back into a second (different) byte array (all in memory) — canton7, May 11 '21 at 16:59
It fails when calling "Cryptography.Encryption.EncryptAES(System.Text.Encoding.Default.GetString(bytes), encryptionKey)" — Elvis Detters, May 11 '21 at 17:08
Camtasia.exe is **not** a text-encoded file, passing it through a text encoding/decoding filter is going to render the contents corrupt. — Lasse V. Karlsen, May 11 '21 at 20:46
You're base-64 encoding an executable? Got an extra terabyte of RAM in there? — 3Dave, May 11 '21 at 22:08

canton7 · Accepted Answer · 2021-05-11T20:44:02.627

You're reading the entire exe into memory, interpreting it as a UTF-16 string (??!), turning that back into UTF-8 bytes, and encrypting those. This converting to/from a string is horifically broken. An executable file is not a human-readable string, and even if it was, you're in a real muddle as to which encoding you're using. I think you can drop the whole string thing.

You're also reading the entire thing into memory (several times in fact, because of the whole string thing), which is wasteful. You don't need to do that: you can encrypt it bit-by-bit. To do this, use a Stream.

Something like this should work (untested): at least it gets the general concept across. We set up a series of streams which lets us read the data out of the input file bit-by-bit, and write them out to the output file bit-by-bit.

// The file we're reading from
using var inputStream = File.OpenRead("Camtasia.exe");

// The file we're writing to
using var outputStream = File.OpenWrite("EncryptedFile.txt");

using var HashAlgorithm hash = MD5.Create();
using var aes = Aes.Create();

aes.Key = hash.ComputeHash(Encoding.UTF8.GetBytes(password));

// Turn the IV into a base64 string, add "!", encode as UTF-8, and write to the file
string base64IV = Convert.ToBase64String(aes.IV) + "!";
byte[] base64IVBytes = Encoding.UTF8.GetBytes(base64IV);
outputStream.Write(base64IVBytes, 0, base64IVBytes.Length);

// Create a stream which, when we write bytes to it, turns those into
// base64 characters and writes them to outputStream
using var base64Stream = new CryptoStream(outputStream, new ToBase64Transform(), CryptoStreamMode.Write);

// Create a stream which, when we write bytes to it, encrypts them and sends them to
// base64Stream
using var encryptStream = new CryptoStream(base64Stream, aes.CreateEncryptor(), CryptoStreamMode.Write);

// Copy the entirety of our input file into encryptStream. This will encrypt them and
// push them into base64Stream, which will base64-encode them and push them into
// outputStream
inputStream.CopyTo(encryptStream);

Note, that using MD5 to derive key bytes isn't best practice. Use Rfc2898DeriveBytes.

Also note that you don't necessarily need to base64-encode the encrypted result before writing it to a file -- you can just write the encrypted bytes straight out. To go this, get rid of base64Stream, and tell the encryptStream to write straight to outputStream.

`crypt.IV` should most likely be `aes.IV` – Rand Random May 11 '21 at 17:27 — Rand Random, May 11 '21 at 17:27

C# Out of Memory while handling and encrypting large files

1 Answers1