Android : Read blocks while reading HTTP response

Question

I am trying to read a fairly large response body (video data > 5 MB) using Android HttpClient. I have used the AsyncTask method to execute the network task in background. The code gets stuck at the read call and times out with OutofMemoryError.

I have also tried the defauly EntityUtils method which also returns the OutofMemoryError which is justifiable as this is a fairly large amount of data to be read in memory.

There is a minor crude check for checking if it is a binary data just in case.

Any ideas as to why the read is blocking ??

protected HttpResponse doInBackground(String... params) {
        String link = params[0];
        HttpResponse response = null;
        HttpGet request = new HttpGet(link);

        //Just in case we need proxy
        if (isCancelled())
        {
            Log.d(LOGTAG, "Background task cancelled");
        }

        //Start android client for sending first request
        Log.d(LOGTAG, "Proxy is disabled");
        client = AndroidHttpClient.newInstance("Android");
        try 
        {

            response = client.execute(request);
            if(response != null)
            {
                try 
                {
                    //HttpEntity entity = response.getEntity();
                    Log.d(LOGTAG,"Parsing Response");
                    String line = "";

                    // Wrap a BufferedReader around the InputStream

                    InputStream responseStream = response.getEntity().getContent();

                     //This causes OutofMemoryError
                    //responseBody = EntityUtils.toString(getResponseEntity); 

                    int read=0;
                    byte[] bytes = new byte[12288];
                    OutputStream output = null;

                    String filename = searchContentUrl.replace('/', '_');
                    File outfile = new File(AppMainActivity.dataDir, filename);

                    output = new BufferedOutputStream(new FileOutputStream(outfile));
                    int firstChunkCheck = 0;
                    while((read = responseStream.read(bytes))!= -1)
                    {
                        CharsetEncoder encoder =  Charset.forName("ISO-8859-1").newEncoder();  

                        if(binaryData == 0)
                        {
                            if (firstChunkCheck == 0) //Crude check for binary data
                            {  
                                if(encoder.canEncode(new String(bytes)))
                                Log.d(LOGTAG,"Got normal Data");
                                binaryData = 0;
                                firstChunkCheck = 1;
                            }
                            responseBody += new String(bytes);
                        }


                        else 
                        {  
                            //Toast.makeText(getApplicationContext(), "Downloading file ", Toast.LENGTH_SHORT).show();
                            binaryData = 1;
                            Log.d(LOGTAG, "Writing binary file...");
                            try 
                            {
                                output.write(bytes);
                            }
                            catch(FileNotFoundException ex)
                            {
                                Log.d(LOGTAG, "FileNotFoundException");
                                ex.printStackTrace();
                            }
                            catch(IOException ex)
                            {
                                ex.printStackTrace();
                            }
                        }  
                    }
                    if(output != null)
                        output.close();


                    Log.d(LOGTAG,"Read Response");
                    return response;
                }
                catch (ParseException e)
                {
                    Log.d(LOGTAG,"IOException while getting Response");
                    e.printStackTrace();
                    return null;
                }
                catch (UnsupportedEncodingException e) 
                {
                    Log.d(LOGTAG,"UnsupportedEncodingException while decoding uri");
                    e.printStackTrace();
                    return null;
                }
                catch (IOException e) 
                {
                    Log.d(LOGTAG,"IOException while decoding uri");
                    e.printStackTrace();
                    return null;
                }
            }
        } 
        catch (IOException e) 
        {
            e.printStackTrace();
            return null;
        } 
        finally
        {
            //client.close();
        }
        return response;
    }

Answer 1

You are trying to encode the data as you write it. Instead, write the raw data to a file, and then encode it as you read it. This allows you to use an InputStreamReader for your specific character set, which takes care of all the details for you. This can only be done when all the data is available, as you have probably realised.

In addition, the line responseBody += new String(bytes); looks a bit odd.