Convert.ToBase64String throws 'System.OutOfMemoryException' for byte [] (file: large size)

Question

I am trying to convert byte[] to base64 string format so that i can send that information to third party. My code as below:

byte[] ByteArray = System.IO.File.ReadAllBytes(path);
string base64Encoded = System.Convert.ToBase64String(ByteArray);

I am getting below error:

Exception of type 'System.OutOfMemoryException' was thrown. Can you help me please ?

*Don't* load the entire file in memory perhaps? How big is that file? Where is the exception thrown? Post the *full* exception, including the call stack. You can get it easily with `Exception.ToString()` — Panagiotis Kanavos, Oct 05 '18 at 11:20
@PanagiotisKanavos Unfortunately, the Base64 implementation in .NET doesn't really help you do this in chunks unless you know exactly how you need to do it. Might be a nuget package that adds something that works better with large datasets. — Lasse V. Karlsen, Oct 05 '18 at 11:21
@LasseVågsætherKarlsen somehow I doubt the file is 16GB large. I suspect this code runs frequently resulting in a lot of garbage buffers and memory fragmentation. — Panagiotis Kanavos, Oct 05 '18 at 11:23
@LasseVågsætherKarlsen plus, CryptoStream can be used to perform Base64 transformations [as shown here](https://stackoverflow.com/questions/2525533/is-there-a-base64stream-for-net-where). This could actually be a duplicate. — Panagiotis Kanavos, Oct 05 '18 at 11:26
Possible duplicate of [Is there a Base64Stream for .NET? where?](https://stackoverflow.com/questions/2525533/is-there-a-base64stream-for-net-where) — Panagiotis Kanavos, Oct 05 '18 at 11:28
How is the Base64 string used? Is it saved in a file? Written to an HTTP response? In both cases [Stream.CopyTo](https://learn.microsoft.com/en-us/dotnet/api/system.io.stream.copyto?view=netframework-4.7.2#System_IO_Stream_CopyTo_System_IO_Stream_) could be used to copy encoded data directly to the target instead of creating one big string — Panagiotis Kanavos, Oct 05 '18 at 11:30
Voting to close, there has been no clarification or response from the author — TheGeneral, Oct 05 '18 at 11:36
Why base64? Is this a requirement from the other side of the conversation? If you're having problems serializing, then the receiving end is likely to encounter similar troubles deserializing your super-enormous string. Do you really ***need*** to base64 this? With a payload this large, sending the data directly as binary would be hugely preferable and avoid considerable memory churn. — spender, Oct 05 '18 at 11:54
@PanagiotisKanavos Nice about CryptoStream, didn't know that! — Lasse V. Karlsen, Oct 05 '18 at 12:12
@LasseVågsætherKarlsen me neither but there *had* to be something that can work with streams. It's too common a scenario — Panagiotis Kanavos, Oct 05 '18 at 12:29
I voted to close because by not providing clarification to the questions asked in the comments, it's not clear what you're actually asking. — spender, Oct 05 '18 at 14:29
I use (asp.net) to play video files outside the website directory. So I need to convert the byte[] to base64 string to display (code behind: video.src = base64string). If I open the file (size over 300MB), I am getting error: System.OutOfMemoryException was throws. Thank all! — HoBin, Oct 06 '18 at 04:09

JohnLBevan · Answer 1 · 2018-10-06T11:52:15.387

Update

I just spotted @PanagiotisKanavos' comment pointing to Is there a Base64Stream for .NET?. This does essentially the same thing as my code below attempts to achieve (i.e. allows you to process the file without having to hold the whole thing in memory in one go), but without the overhead/risk of self-rolled code / rather using a standard .Net library method for the job.

Original

The below code will create a new temporary file containing the Base64 encoded version of your input file.

This should have a lower memory footprint, since rather than doing all data at once, we handle it several bytes at a time.

To avoid holding the output in memory, I've pushed that back to a temp file, which is returned. When you later need to use that data for some other process, you'd need to stream it (i.e. so that again you're not consuming all of this data at once).

You'll also notice that I've used WriteLine instead of Write; which will introduce non base64 encoded characters (i.e. the line breaks). That's deliberate, so that if you consume the temp file with a text reader you can easily process it line by line.
However, you can amend per your needs.

void Main()
{
    var inputFilePath = @"c:\temp\bigfile.zip";
    var convertedDataPath = ConvertToBase64TempFile(inputFilePath);
    Console.WriteLine($"Take a look in {convertedDataPath} for your converted data");
}

//inputFilePath = where your source file can be found.  This is not impacted by the below code
//bufferSizeInBytesDiv3  = how many bytes to read at a time (divided by 3); the larger this value the more memory is required, but the better you'll find performance.  The Div3 part is because we later multiple this by 3 / this ensures we never have to deal with remainders (i.e. since 3 bytes = 4 base64 chars)
public string ConvertToBase64TempFile(string inputFilePath, int bufferSizeInBytesDiv3 = 1024)
{
    var tempFilePath = System.IO.Path.GetTempFileName();
    using (var fileStream = File.Open(inputFilePath,FileMode.Open))
    {
        using (var reader = new BinaryReader(fileStream))
        {
            using (var writer = new StreamWriter(tempFilePath))
            {
                byte[] data;
                while ((data = reader.ReadBytes(bufferSizeInBytesDiv3 * 3)).Length > 0)
                {
                    writer.WriteLine(System.Convert.ToBase64String(data)); //NB: using WriteLine rather than Write; so when consuming this content consider removing line breaks (I've used this instead of write so you can easily stream the data in chunks later)
                }
            }
        }
    }
    return tempFilePath;
}

Hi JohnLBevan, I use your code but the return result is an incomplete base64 string compared to using: string base64Encoded = System.Convert.ToBase64String (ByteArray); Example: ConvertToBase64TempFile(inputFilePath) -> output: aaabbb | System.Convert.ToBase64String (ByteArray) -> output: aaabbbcccddd. Thanks! — HoBin, Oct 06 '18 at 02:36

Convert.ToBase64String throws 'System.OutOfMemoryException' for byte [] (file: large size)

1 Answers1

Update

Original