Working with extra large json files. Return always out of memory

Question

How to use the stream API json with jackson? See my code below:

    ObjectMapper mapper = new ObjectMapper();

    Map<String, Object> map = new HashMap<String, Object>();

    List<Object> list = new ArrayList<Object>();

    // Get images in database
    try {
            Class.forName(DRIVER);
            connection = DriverManager.getConnection(URL, USER, PASSWORD);

            Statement s = connection.createStatement();
            ResultSet r = s.executeQuery("select * from images");

            while (r.next()) {

                byte[] imageBytes = r.getBytes("image");
                String imageBase64 = DatatypeConverter.printBase64Binary(imageBytes);
                list.add(imageBase64);
            }

    } catch (SQLException e) {

    }

    map.put("images", list);

    // Stream Json API
    try {
            mapper.writeValue(new File("c:\\images.json"), map);
    } catch (JsonGenerationException e) {
            e.printStackTrace();
    } catch (JsonMappingException e) {
            e.printStackTrace();
    } catch (IOException e) {
            e.printStackTrace();
    }

Always return out of memory. I dont know to use stream with jackson. I working with extra large json, average 2000 images, to each image a imageBase64. what am I doing wrong?

[Here's an example](http://www.mkyong.com/java/jackson-streaming-api-to-read-and-write-json/) — Sotirios Delimanolis, Feb 26 '13 at 19:58
@SotiriosDelimanolis excellent, thank you so much. Solve the out of memory problem. Now, how to show generate json? — Schlacter James, Feb 26 '13 at 20:10
You can just read the file through some InputStream, shove it inside a String, and print it. — Sotirios Delimanolis, Feb 26 '13 at 20:12
@SotiriosDelimanolis The same problem when I read the json file. Oh god, I'm crazy.. — Schlacter James, Feb 26 '13 at 20:30
How big is the file? You could increase the maximum heap size for java. Search on SO for that. — Sotirios Delimanolis, Feb 26 '13 at 20:31
@SotiriosDelimanolis I dont know but are 2000 images into json file. I try increase the heap size but crash my computer. — Schlacter James, Feb 26 '13 at 20:34
@SotiriosDelimanolis Do you have idea for store the images without image bytes and image base 64. I need store the images because the application will be offline. — Schlacter James, Feb 26 '13 at 20:38
Why don't you just store them as individual images, why does it have to be in json? — Sotirios Delimanolis, Feb 26 '13 at 20:39
@SotiriosDelimanolis Well, This will be happening in my web service, right? How to store the images in my web service for after my mobile app get the images? I use the image with base64 in my json because no need save the image in mobile disk. Any idea? — Schlacter James, Feb 26 '13 at 20:45
What's your use-case here? Are you trying to retrieve 2000 images in your mobile application, from your web service as one object (a map containing a list of base 64 strings)? — Steinar, Feb 26 '13 at 22:48

StaxMan · Accepted Answer · 2013-02-27T18:04:45.160

1

Instead of keeping all images in memory, just read and write them incrementally. An example of Jackson Streaming API can be found here ("Reading and Writing Event Streams").

EDIT: This should be too hard to figure out folks... but here's a skeletal example:

// typed from memory, some methods may be off a bit
JsonFactory f = objectMapper.getFactory();
JsonGenerator gen = f.createGenerator(new File("c:\\images.json"));
gen.writeStartArray(); // to get array of objects
// get the DB connection etc
while (r.next()) {
  gen.writeFieldName("image");
  InputStream in = r.getBinaryStream("image");
  gen.writeBinary(in, -1); // length optional for JSON
  in.close();
}

gen.writeEndArray(); // to get array of objects gen.close();

and that should about do the trick.

edited Feb 27 '13 at 18:04

answered Feb 27 '13 at 05:34

StaxMan

113,358
34
211
239

The same problem. Cannot use the base64 to generate json because the file is huge. No Way. How can I generate a url for each image from database? Any idea? – Schlacter James Feb 27 '13 at 13:52
Same what? When you read and write things incrementally, they are NOT stored in memory -- you obviously write to a File or other streamable storage. And instead of encoding Base64 in memory, that too can be written incrementally, as long as binary data is accessible using `InputStream`. As to URL, well, that's another question and outside scope of JSON data-binding. But really, instead of asking questions you should try to understand the example I pointed, and see _why_ it makes a difference. – StaxMan Feb 27 '13 at 17:58
thank you, I can generate images.json, no problem, but to read images.json is impossible, huge files. No way for mobile application with limited resources. Now, generate a URL to each image, solve my problem. – Schlacter James Feb 27 '13 at 21:35
Why would reading be a problem? Use streaming API as well, to read data reverse of writing, using `JsonParser`. I'll let you figure that one out. – StaxMan Feb 27 '13 at 22:27

Working with extra large json files. Return always out of memory

1 Answers1

Linked