Java/Android: java.lang.OutOfMemoryError while building a JSON object

Question

I am importing JSON data from a public database URI http://data.seattle.gov/api/views/3k2p-39jp/rows.json and the rows go as far as 445454. Using the following code I am constructing the JSON object of the entire data.

   HttpGet get = new HttpGet(uri);
   HttpClient client = new DefaultHttpClient();
   HttpResponse response = client.execute(get);
   BufferedReader reader = new BufferedReader(new InputStreamReader(response.getEntity().getContent(), "UTF-8"));
    StringBuilder builder=new StringBuilder();
for(String line=null;(line = reader.readLine()) != null;){
      builder.append(line).append("\n");
    }
  JSONTokener jsonTokener=new JSONTokener(builder.toString());
  JSONObject finalJson=new JSONObject(jsonTokener);
  JSONArray data=finalJson.getJSONArray("data");

Because the data is too large, i am getting 03-21 03:41:49.714: E/AndroidRuntime(666): Caused by: java.lang.OutOfMemoryError pointing the source of error at buildr.append(line).append("\n"). Is there anyway I can handle large datasets without getting memory allocation issues?

read parts of it instead of the entire thing.. ? – JoxTraex Mar 21 '12 at 06:24 — JoxTraex, Mar 21 '12 at 06:24

score 5 · Accepted Answer · edited Jun 20 '20 at 09:12

5

That JSON is huge!

You definitely need to use a streaming JSON parser. There are two out there for Android: GSON and Jackson.

GSON Streaming is explained at: https://sites.google.com/site/gson/streaming

I like how GSON explains the problem you're having:

Most applications should use only the object model API. JSON streaming is useful in just a few situations:

When it is impossible or undesirable to load the entire object model into memory. This is most relevant on mobile platforms where memory is limited.

Jackson Streaming is documented at: http://wiki.fasterxml.com/JacksonInFiveMinutes#Streaming_API_Example

edited Jun 20 '20 at 09:12

Community

1
1

answered Mar 21 '12 at 06:49

louielouie

14,881
3
26
31

I like the GSON. So you are saying instead of using `BufferedReader` and `StringBuilder` I should do `JSONReader` and `List` to have an advantage over memory issues? – jmishra Mar 21 '12 at 07:05
Correct. JSONReader will take in your inputStream, so that will replace both BufferedReader and StringBuilder in your code. Note that Message is an example class. Looking at that JSON, you'll probably want to replace that with your own PoliceResponse class instead. Since the JSON isn't just one big array like in the examples, you'll have to navigate to the array that contains the police responses first, then you can start iterating through that array. – louielouie Mar 21 '12 at 07:16

score 1 · Answer 2 · answered Mar 21 '12 at 06:26

If possible only request parts of the data - this also reduces time for network io and thus saves battery.

Otherwise you could try to not keep the incoming data in memory, but to 'stream' it onto the sd-card. When it is stored there you can then iterate over it. Most likely this will mean to use your own JSON tokenizer that does not build a full tree, but which is able to (like a SAX parser) only look at a part of the object tree at a time.

You may have a look at Jackson, which has a streaming mode, which may be applicable.

score 1 · Answer 3 · answered Mar 21 '12 at 06:58

Streaming pull parser is the way. I recommend GSON, as this has small memory footpring (just pull parsing is about 16K , jackson is way bigger)

Your code is problematic because you allocate:

buffer to hold all the string data coming from service
all the JSON DOM objects

and this is slow, and gives you memory meltdown.

In case you need java objects out of your JSON data , you may try my small databinding library building on GSON (shameles self advertising off):

https://github.com/ko5tik/jsonserializer

score 0 · Answer 4 · answered Aug 06 '14 at 05:17

I did it a bit differently, My JSON code was waiting for status, which comes towards the end. So I modified the code to return earlier.

// try to get formattedAddress without reading the entire JSON
        String formattedAddress;
        while ((read = in.read(buff)) != -1) {
            jsonResults.append(buff, 0, read);
            formattedAddress = ((String) ((JSONObject) new JSONObject(
                    jsonResults.toString()).getJSONArray("results").get(0))
                    .get("formatted_address"));
            if (formattedAddress != null) {
                Log.i("Taxeeta", "Saved memory, returned early from json") ;
                return formattedAddress;
            }               
        }
JSONObject statusObj = new JSONObject(jsonResults.toString());
        String status = (String) (statusObj.optString("status"));
        if (status.toLowerCase().equals("ok")) {
            formattedAddress = ((String) ((JSONObject) new JSONObject(
                    jsonResults.toString()).getJSONArray("results").get(0))
                    .get("formatted_address"));
            if (formattedAddress != null) {
                Log.w("Taxeeta", "Did not saved memory, returned late from json") ;
                return formattedAddress;
            }           
        }

Java/Android: java.lang.OutOfMemoryError while building a JSON object

4 Answers4

Linked