Posting file on Background Agent / HttpWebRequest stream buffer keeps growing?

Question

I need to POST a 5MB file from within a ResourceIntensiveTask, where the OS sets a max memory usage of 5MB. So trying to stream the file directly from storage, but the Stream associated to the HttpWebRequest keeps growing in size. This is the code:

        public void writeStream(Stream writer, string filesource, string filename)
        {
            var store = System.IO.IsolatedStorage.IsolatedStorageFile.GetUserStoreForApplication();
            var f = store.OpenFile(filesource, FileMode.Open, FileAccess.Read);
            store.Dispose();

            byte[] buffer = Encoding.UTF8.GetBytes(String.Format(@"Content-Disposition: form-data; name=""file""; filename=""{0}""\n", filename));
            writer.Write(buffer, 0, buffer.Length);

            buffer = Encoding.UTF8.GetBytes("Content-Type: application/octet-stream\n");
            writer.Write(buffer, 0, buffer.Length);

            long initialMemory = Microsoft.Phone.Info.DeviceStatus.ApplicationCurrentMemoryUsage;

            buffer = new byte[2048]; 
            int DataRead = 0;
            do
            {
                DataRead = f.Read(buffer, 0, 2048);
                if (DataRead > 0)
                {
                    writer.Write(buffer, 0, DataRead);
                    Array.Clear(buffer, 0, 2048);
                }
            } while (DataRead > 0);

            double increasedMemory = ((double)Microsoft.Phone.Info.DeviceStatus.ApplicationCurrentMemoryUsage - initialMemory) / 1000000;                

            buffer = Encoding.UTF8.GetBytes("\n--" + boundary + "\n--");
            writer.Write(buffer, 0, buffer.Length);
            writer.Flush();
        }

increasedMemory debug variable is used to get the differential memory before and after the file is read and streamed to the HttpWebRequest, and it gives almost the exact size of the file (5MB) which means the process memory is increasing 5MB.

I am also setting AllowReadStreamBuffering=false to the HttpWebRequest.

How to keep memory low? How to upload large files when memory usage limit is 5MB?

Answer 1

The problem is that without being able to turn off write buffering, the connection to the server is not even made until BeginGetResponse() is called after closing the request stream (verified with WireShark).

The only way I can think of to get around this would be to use sockets directly (although that will be way more complicated if using an SSL connection).

This code works for me and doesn't increase memory usage while sending data to the server. I haven't tested it in a background task but don't see any reason it wouldn't work.

Socket _socket;
const int BUFFERSIZE = 4096;
byte[] writebuffer = new byte[BUFFERSIZE];
string hostName = "www.testdomain.com";
string hostPath = "/test/testupload.aspx";
IsolatedStorageFileStream isoFile;


public void SocketPOST(string hostName, string filesource)
{
    using (IsolatedStorageFile store = IsolatedStorageFile.GetUserStoreForApplication())
    {
        if (store.FileExists(filesource))
        {
            isoFile = store.OpenFile(filesource, FileMode.Open, FileAccess.Read);
        }
    }

    _socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
    _socket.SetNetworkRequirement(NetworkSelectionCharacteristics.NonCellular);

    SocketAsyncEventArgs socketEventArg = new SocketAsyncEventArgs();
    socketEventArg.RemoteEndPoint = new DnsEndPoint(hostName, 80);
    socketEventArg.Completed += new EventHandler<SocketAsyncEventArgs>(Socket_Completed);

    _socket.ConnectAsync(socketEventArg);
}


private void Socket_Completed(object sender, SocketAsyncEventArgs e)
{
    if (e.SocketError == SocketError.Success)
    {
        switch (e.LastOperation)
        {
            case SocketAsyncOperation.Connect:   // Connected so started sending data, headers first

                if (e.ConnectSocket.Connected)
                {
                    StringBuilder sbHeaders = new StringBuilder("POST " + hostPath + " HTTP/1.1\r\n");
                    sbHeaders.Append("HOST: " + hostName + "\r\n");
                    sbHeaders.Append("USER-AGENT: MyWP7App/1.0\r\n");
                    sbHeaders.Append("Content-Type: text/plain; charset=\"utf-8\"\r\n");
                    sbHeaders.Append("Content-Length: " + isoFile.Length.ToString() + "\r\n\r\n");

                    byte[] headerBuffer = Encoding.UTF8.GetBytes(sbHeaders.ToString());
                    e.SetBuffer(headerBuffer, 0, headerBuffer.Length);

                    if (!e.ConnectSocket.SendAsync(e)) Socket_Completed(e.ConnectSocket, e);
                }

                break;

            case SocketAsyncOperation.Send:
            case SocketAsyncOperation.SendTo:   // Previous buffer sent so send next one if stream not finished

                Array.Clear(writebuffer, 0, BUFFERSIZE);

                int DataRead = 0;

                DataRead = isoFile.Read(writebuffer, 0, BUFFERSIZE);
                if (DataRead > 0)
                {
                    e.SetBuffer(writebuffer, 0, DataRead);
                    if (!_socket.SendAsync(e)) Socket_Completed(e.ConnectSocket, e);
                }
                else
                {
                    isoFile.Dispose();
                    if (!_socket.ReceiveAsync(e)) Socket_Completed(e.ConnectSocket, e);
                }

                break;

            case SocketAsyncOperation.Receive:
            case SocketAsyncOperation.ReceiveFrom:

                if (e.BytesTransferred > 0)
                {
                    string response = Encoding.UTF8.GetString(e.Buffer, e.Offset, e.BytesTransferred).Trim('\0');

                    // Check response if necessary 

                    e.ConnectSocket.Shutdown(SocketShutdown.Both);
                    e.ConnectSocket.Dispose();
                }

                break;

            default:
                break;
        }

    }
}

Note: I've left a lot of the error handling out to keep the example short.

SSL Note: Because SSL works at the TCP level and WP7 doesn't currently support SSL sockets (SslStream) you would need to handle the certificate handshake, cipher exchange, etc yourself to set up the SSL connection on the socket and then encrypt everything being sent (and decrypt everything received) with the agreed algorithms. There has been some success using the Bouncy Castle API so that could be possible (see this blog post).

Answer 2

One thing I noticed: you forgot to dispose f!

I personally would use the code like this:

public void writeStream(Stream writer, string filesource, string filename)
{
    using (var store = System.IO.IsolatedStorage.IsolatedStorageFile.GetUserStoreForApplication())
    {
        long initialMemory = Microsoft.Phone.Info.DeviceStatus.ApplicationCurrentMemoryUsage;

        using (var f = store.OpenFile(filesource, FileMode.Open, FileAccess.Read))
        {
            byte[] buffer = Encoding.UTF8.GetBytes(string.Format(@"Content-Disposition: form-data; name=""file""; filename=""{0}""\n", filename));
            writer.Write(buffer, 0, buffer.Length);

            buffer = Encoding.UTF8.GetBytes("Content-Type: application/octet-stream\n");
            writer.Write(buffer, 0, buffer.Length);

            buffer = new byte[2048];
            int DataRead = 0;

            do
            {
                DataRead = f.Read(buffer, 0, 2048);
                if (DataRead > 0)
                {
                    writer.Write(buffer, 0, DataRead);
                }
            } while (DataRead > 0);

            buffer = Encoding.UTF8.GetBytes("\n--" + boundary + "\n--");
            writer.Write(buffer, 0, buffer.Length);
            writer.Flush();
        }

        double increasedMemory = ((double)Microsoft.Phone.Info.DeviceStatus.ApplicationCurrentMemoryUsage - initialMemory) / 1000000;
    }
}

The boundary var seems to be missing, so a coding error still remains here!