Decrease memory consumption when encrypting files c#

Question

My encryption app is using to much memory and it simply can't handle large files, how can I optimize my code for handling large files? I am using the code below to convert the file to base64 (increasing the file size dramatically)

Console.Write("Enter File Path: ");
docPath = Console.ReadLine();
extension = docPath.Substring(docPath.IndexOf(".")).Trim();
byte[] binarydata = File.ReadAllBytes(docPath);
text = System.Convert.ToBase64String(binarydata, 0, binarydata.Length);
var Encrypted = AESCryptography.Encrypt(text, m.ToString(), extension);
using (FileStream fs = File.Create(docPath.Substring(0,docPath.IndexOf(".")) + ".aent"))
{
    Byte[] info = new UTF8Encoding(true).GetBytes(Encrypted);
    // Add some information to the file.
    fs.Write(info, 0, info.Length);
}

How can I do this in blocks? Heres my encryption class:

public static class AESCryptography
{
    private const int keysize = 256;
    public static string Encrypt(string plainText, string passPhrase, string extention)
    {
        byte[] plainTextBytes = Encoding.UTF8.GetBytes(plainText);
        using (PasswordDeriveBytes password = new PasswordDeriveBytes(passPhrase, null))
        {
            byte[] keyBytes = password.GetBytes(keysize / 8);
            using (RijndaelManaged symmetricKey = new RijndaelManaged())
            {
                symmetricKey.GenerateIV();
                symmetricKey.Mode = CipherMode.CBC;
                using (ICryptoTransform encryptor = symmetricKey.CreateEncryptor(keyBytes, symmetricKey.IV))
                {
                    using (MemoryStream memoryStream = new MemoryStream())
                    {
                        using (CryptoStream cryptoStream = new CryptoStream(memoryStream, encryptor, CryptoStreamMode.Write))
                        {
                            cryptoStream.Write(plainTextBytes, 0, plainTextBytes.Length);
                            cryptoStream.FlushFinalBlock();
                            byte[] cipherTextBytes = memoryStream.ToArray();
                            return Convert.ToBase64String(cipherTextBytes) + "\n" + Convert.ToBase64String(symmetricKey.IV) + "\n" + extention;
                        }
                    }
                }
            }
        }
    }

    public static string Decrypt(string cipherText, string passPhrase, string initVector)
    {
        byte[] initVectorBytes = Convert.FromBase64String(initVector);
        byte[] cipherTextBytes = Convert.FromBase64String(cipherText);
        using (PasswordDeriveBytes password = new PasswordDeriveBytes(passPhrase, null))
        {
            byte[] keyBytes = password.GetBytes(keysize / 8);
            using (RijndaelManaged symmetricKey = new RijndaelManaged())
            {
                symmetricKey.Mode = CipherMode.CBC;
                using (ICryptoTransform decryptor = symmetricKey.CreateDecryptor(keyBytes, initVectorBytes))
                {
                    using (MemoryStream memoryStream = new MemoryStream(cipherTextBytes))
                    {
                        using (CryptoStream cryptoStream = new CryptoStream(memoryStream, decryptor, CryptoStreamMode.Read))
                        {
                            byte[] plainTextBytes = new byte[cipherTextBytes.Length];
                            int decryptedByteCount = cryptoStream.Read(plainTextBytes, 0, plainTextBytes.Length);
                            return Encoding.UTF8.GetString(plainTextBytes, 0, decryptedByteCount);
                        }
                    }
                }
            }
        }
    }
}

Unless there's a really good reason for it, you should move away from strings and base64 encoding and go binary. — spender, Aug 20 '14 at 12:24

score 3 · Answer 1 · edited May 23 '17 at 10:26

3

First you're reading bytes from your file: byte[] binarydata = File.ReadAllBytes(docPath);
Then you're converting bytes to a string: text = System.Convert.ToBase64String(binarydata, 0, binarydata.Length);
Then you're getting your bytes back from the string: byte[] plainTextBytes = Encoding.UTF8.GetBytes(plainText);

You're currently storing in memory all of your large file twice (actually way more than twice because base64 string uses more memory than just a byte array).

Solution:

Drop the part that converts to/from string so you don't use strings at all but only bytes.
Then drop the whole ReadAllBytes thing that loads all your file in memory and process by small chunks instead. See Using Rijndael encryption for large files for example

edited May 23 '17 at 10:26

Community

1
1

answered Aug 20 '14 at 12:04

ken2k

48,145
10
116
176

can't I just modify my current code to do it in blocks? If I change my encryption class I will have to re-write my whole application! – javaseaayameradost Aug 20 '14 at 12:18
Sure you can, just adapt your code a bit. Have a look at the linked answer. You read your file into small chunk of bytes, then you write them to the CryptoStream, then you call FlushFinalBlock at the end. – ken2k Aug 20 '14 at 12:20
if I "Read [the] file into small chunk of bytes, then [I] write them to the CryptoStream, then call FlushFinalBlock at the end" will I have to do the same with my decryption class? – javaseaayameradost Aug 20 '14 at 12:49

score 2 · Answer 2 · answered Aug 20 '14 at 12:06

Don't store the IV at the end of the file. You are forcing a reader to read the entire encrypted file first just to discover the IV needed top read the file. Store it at the beginning of the file.
Don't encode the file base64. Base64 is a transmission format, not a storage one.
Use Stream.CopyTo to read the file and write into the CryptoStream w/o reading all in memory at once. Back the CryptoStream by the destination file stream, not by a memory stream.
Ditto for decryption, read the CryptoStream (backed by input stream, after skipping the IV) and CopyTo the destination file stream

Decrease memory consumption when encrypting files c#

2 Answers2

Linked