C# HttpWebRequest POST data in chunk

Question

What is the max length of byte array? I'm trying to create an array of bytes whose length is 551858003. I have created on zip file of near about 526 MB. But it gives me error Out of memory exception I am uploading the file on google drive.

Here I tried some code. I am reading the bytes of zip file by using following code.

 byte[] FileByteArray = null;

                using (IsolatedStorageFile storage = IsolatedStorageFile.GetUserStoreForApplication())
                {
                    if (storage.FileExists(fileName))
                    { 
                        using (IsolatedStorageFileStream fileStream = storage.OpenFile(fileName,FileMode.Open,FileAccess.Read))
                        {
                            fileStream.Flush();
                            fileStream.Position = 0;
                            long len = fileStream.Length;
                            FileByteArray = new byte[len];
                            fileStream.Read(FileByteArray, 0, FileByteArray.Length);

                            //using (BinaryReader binReader = new BinaryReader(fileStream))
                            //{
                            //    Int32 Filelength = Convert.ToInt32(fileStream.Length);
                            //    FileByteArray = binReader.ReadBytes(Filelength);
                            //}
                            fileStream.Flush();
                            fileStream.Dispose();
                            fileStream.Close();
                        }
                    }
                }

How can I resolve this issue ? I am getting OutOfMemoryException while uploading large file. I can upload near about 100MB.

Here is my Method for sending images in chunk

 public void Images_ChunkRequst(string uploadURL, byte[] FileByteArray, int startIndex)
    {
        try
        {
            int chunkSize = 256 * 1024 * 2;
            int totalChunks = (int)Math.Ceiling((double)FileByteArray.Length / chunkSize);
            int endIndex = (int)(startIndex + chunkSize > FileByteArray.Length ? FileByteArray.Length : startIndex + chunkSize);
            int length = endIndex - startIndex;

            if (i < totalChunks)
            {
                 CollectionIP = CheckInternet.Find();
                if (CollectionIP.Count == 2 && DeviceNetworkInformation.IsWiFiEnabled)
                    NetworkIPaddress = IPAddress.Parse(CollectionIP[1]).ToString();
                else if (CollectionIP.Count > 0 && DeviceNetworkInformation.IsWiFiEnabled)
                    NetworkIPaddress = IPAddress.Parse(CollectionIP[0]).ToString();

                if (!string.IsNullOrEmpty(NetworkIPaddress))
                {
                    i = i + 1;
                    var request = WebRequest.Create(uploadURL) as HttpWebRequest;
                    request.Method = "PUT";
                    request.Headers["Authorization"] = string.Format("Bearer {0} ", AccessToken);
                    request.ContentType = "application/zip";
                    request.ContentLength = length;
                    request.Headers["Content-Range"] = "bytes " + startIndex + "-" + (endIndex - 1) + "/" + FileByteArray.Length;                        
                    request.AllowWriteStreamBuffering = false;

                    request.BeginGetRequestStream(arPut =>
                    {
                        var request1 = (HttpWebRequest)arPut.AsyncState;
                        using (var dataStream = request1.EndGetRequestStream(arPut))
                        {
                            //getting exception here
                            dataStream.Write(FileByteArray.Skip(startIndex).Take(endIndex).ToArray(), 0, FileByteArray.Take(length).Count());                                
                            dataStream.Flush();
                            dataStream.Dispose();
                            dataStream.Close();
                        }
                        request1.BeginGetResponse(aPut =>
                        {
                            var request2 = (HttpWebRequest)aPut.AsyncState;
                            WebRequest.RegisterPrefix("http://", WebRequestCreator.ClientHttp);
                            WebRequest.RegisterPrefix("https://", WebRequestCreator.ClientHttp);
                            var response = (HttpWebResponse)request2.EndGetResponse(aPut);

                            if (response.StatusCode.ToString() == "308") // Resume Incomplete
                            {
                                response.Dispose();
                                response.Close();

                                string packet = response.Headers["Range"];
                                string[] rng = packet.Remove(0, 6).Split('-');
                                Images_ChunkRequst(uploadURL, FileByteArray, Convert.ToInt32(rng[1]) + 1);
                            }
                            else if (response.StatusCode.ToString() == "500") //Internal Server Error
                            {
                                i = i - 1;
                                response.Dispose();
                                response.Close();
                                Images_ChunkRequst(uploadURL, FileByteArray, startIndex);
                            }
                            else if (response.StatusCode.ToString() == "502") //Bad Gateway
                            {
                                i = i - 1;
                                response.Dispose();
                                response.Close();
                                Images_ChunkRequst(uploadURL, FileByteArray, startIndex);
                            }
                            else if (response.StatusCode.ToString() == "503") //Service Unavailable
                            {
                                i = i - 1;
                                response.Dispose();
                                response.Close();
                                Images_ChunkRequst(uploadURL, FileByteArray, startIndex);
                            }
                            else if (response.StatusCode.ToString() == "504") //Gateway Timeout
                            {
                                i = i - 1;

                                response.Dispose();
                                response.Close();
                                Images_ChunkRequst(uploadURL, FileByteArray, startIndex);
                            }
                            else if (response.StatusCode == HttpStatusCode.NotFound) //Not Found
                            {
                                i = i - 1;

                                response.Dispose();
                                response.Close();
                                Images_ChunkRequst(uploadURL, FileByteArray, startIndex);
                            }
                            else if (response.StatusCode == HttpStatusCode.OK) // upload complete.
                            {
                                Dispatcher.BeginInvoke(() =>
                                {
                                    i = 0;// i must be 0 after each upload success
                                    imgPass_Images.Visibility = Visibility.Visible;
                                    this.LayoutRoot.IsHitTestVisible = true;
                                    SystemTray.ProgressIndicator.IsIndeterminate = false;
                                    SystemTray.IsVisible = false;
                                    GetNextAutoOrder();
                                });
                            }
                        }, request1);
                    }, request);
                }
                else
                {
                    this.LayoutRoot.IsHitTestVisible = true;
                    SystemTray.ProgressIndicator.IsIndeterminate = false;
                    SystemTray.IsVisible = false;
                    MessageBox.Show("Please check your internet connection.");
                }
            }
        }
        catch (Exception ex)
        {
            MessageBox.Show("Error occured. Trying to send failed package.");
            Images_ChunkRequst(uploadURL, FileByteArray, startIndex);
        }
    }

I am getting exception here dataStream.Write(FileByteArray.Skip(startIndex).Take(endIndex).ToArray(), 0, FileByteArray.Take(length).Count());. I have tried lot to solve this issue. Can some one please help me? I am getting this exception Insufficient memory to continue the execution of the program

score 1 · Answer 1 · answered Mar 05 '14 at 09:37

1

The current implementation of System.Array uses Int32 for all its internal counters etc, so the theoretical maximum number of elements is Int32.MaxValue. So the problem is somewhere else.

answered Mar 05 '14 at 09:37

Sadique

22,572
7
65
91

But if the file size is small then all works fine. The problem with large lenght. – Ajay Mar 05 '14 at 09:40
The problem seems to be with the Stream. Create small buffers. – Sadique Mar 05 '14 at 09:42
Small buffers works perfect. Same code works for small buffer. – Ajay Mar 05 '14 at 09:43
That is what i am trying to say. – Sadique Mar 05 '14 at 09:44
Use buffered stream -http://msdn.microsoft.com/en-us/library/system.io.bufferedstream%28v=vs.110%29.aspx – Sadique Mar 05 '14 at 09:46
`BufferedStream` not support for windows phone 8. – Ajay Mar 05 '14 at 10:07
Check this link - http://stackoverflow.com/questions/14499570/how-to-get-a-stream-from-a-byte-array-in-a-windows8-store-app – Sadique Mar 05 '14 at 10:08

score 0 · Answer 2 · answered Mar 05 '14 at 09:39

0

This has nothing to do with Array.Length. You're running out of process memory. You should do allocations in chunks.

answered Mar 05 '14 at 09:39

Sriram Sakthivel

72,067
7
111
189

How can I allocate it in chunk? – Ajay Mar 06 '14 at 06:36
If you can give more context about what you're going to do with that `FileByteArray` in further code I can help you. because you can't allocate that much memory for sure. Even if you do it in chunks you have to release the previously allocated memory else you'll get the same exception again – Sriram Sakthivel Mar 06 '14 at 06:59
Actually I am collecting all images from device and creating a `zip` file of all images. Now I want to upload this file on google drive, so i need byte of that file. – Ajay Mar 06 '14 at 07:16
Point us the "google drive API" you use, we can see any alternate available. but am sure you can't allocate such a huge memory at a time – Sriram Sakthivel Mar 06 '14 at 07:20
I am using Google.Apis.Drive.v2. – Ajay Mar 06 '14 at 07:40
That is not helpful, please update your question with what API class and which method you need to call, with the link to documentation – Sriram Sakthivel Mar 06 '14 at 07:43

score 0 · Answer 3 · answered Mar 14 '14 at 18:43

This might not be a solution, but I have some issues with the code you posted:

                    using (IsolatedStorageFileStream fileStream = storage.OpenFile(fileName,FileMode.Open,FileAccess.Read))
                    {
                        fileStream.Flush();
                        fileStream.Position = 0;
                        long len = fileStream.Length;
                        FileByteArray = new byte[len];
                        fileStream.Read(FileByteArray, 0, FileByteArray.Length);

                        //using (BinaryReader binReader = new BinaryReader(fileStream))
                        //{
                        //    Int32 Filelength = Convert.ToInt32(fileStream.Length);
                        //    FileByteArray = binReader.ReadBytes(Filelength);
                        //}
                        fileStream.Flush();
                        fileStream.Dispose();
                        fileStream.Close();
                    }

fileStream.Flush() if forcing to write the buffer to disk. Not required for Read
fileStream.Close() would fail because Dispose() is called before it
fileStream.Dispose() and fileStream.Close() aren't required since it should automatic be handled by the using() statement
fileStream.Read(array, start, lenght) returns an integer of how many bytes that was actual read, so it doesn't always match the actual length. You should keep reading until zero is returned. Which is why I suggest following:

Instead of create FileByteArray to the size of the stream, do it in chucks as well and maybe write the result to maybe a memory stream (or even better send it directly to the API if it support sending in chucks as well.) So basically:

Create a byte array of like 1024 (could be bigger if you want bigger reads from the buffer - but avoid too big)

Do the read in a loop:

byte[] buffer = new byte[1024];
int read = 0;
while((read = fileStream.Read(buffer, 0, buffer.Length)) > 0)
     // write buffer[0:read] to memory location or temp file on disk (avoid Flushing for each write);

C# HttpWebRequest POST data in chunk

3 Answers3