9

I get the feeling that the answer might be a duplicate of this: Jackson - Json to POJO With Multiple Entries but I think that potentially the question is different enough. Also I'm using raw data binding rather than full data binding.

So like the asker of that question, I have multiple objects in a file and I'm trying to turn them into POJOs and stuff them into a database of my design so I can access the data quickly rather than slowly.

The files here are in the order of tens of GB, with up to millions of objects in each file. Anyway here is what I have so far:

ObjectMapper mapper = new ObjectMapper();
Map<String,Object> data = mapper.readValue(new File("foo.json"), Map.class);
System.out.println(data.get("bar"));

And this works great for printing the bar element of the first object in foo, but I need a way to iterate through every element in a way that won't eat up all my memory.

Thanks.

Community
  • 1
  • 1
Tom Carrick
  • 6,349
  • 13
  • 54
  • 78

3 Answers3

12

You don't have to choose between Streaming (JsonParser) and ObjectMapper, do both! Traverse a bit with parser, but then call JsonParser.readValueAs(MyType.class) to bind individual JSON Object.

Or, call ObjectMapper's readValue() method passing JsonParser at appropriate points. Or use ObjectMapper.reader(Type.class).readValues() and iterate that way.

Mifeet
  • 12,949
  • 5
  • 60
  • 108
StaxMan
  • 113,358
  • 34
  • 211
  • 239
  • +1 for using both! I've been using streaming + data binding and it is great in terms of both simplicity and memory usage. – andr Dec 05 '12 at 10:29
8

Use this code sample to see the basic idea.

final InputStream in = new FileInputStream("json.json");
try {
  for (Iterator it = new ObjectMapper().readValues(
      new JsonFactory().createJsonParser(in), Map.class); it.hasNext();)
    System.out.println(it.next());
}
finally { in.close();} }
Marko Topolnik
  • 195,646
  • 29
  • 319
  • 436
  • And your file represents a single list of JSON objects (which makes it just one object overall), or is it just the concatenation of JSON objects (which would allow you to read them one by one)? – Marko Topolnik May 02 '12 at 10:12
  • Not knowing too much about json, I think the latter. The file structure is something like: {blah blah blah} {blah blah blah} {etc.} – Tom Carrick May 02 '12 at 11:07
  • This "works" but seems to cause more problems than it solves since I never know what types I have (there are arrays with more objects in the objects) which I'll probably have to ask a new question about. – Tom Carrick May 03 '12 at 09:32
  • @Marko glad you figured out a goo way to do this, assuming types are the same. Yes, I know it can be made to work, having lots of experience with Jackson; but devil is in details, i.e. many small things can go wrong wrt closing of the stream. One important thing is that one MUST use same `JsonParser`, as otherwise first one will read and buffer block full of input; and this buffered content will be missing if separate parsers are constructed. – StaxMan May 04 '12 at 05:47
  • @Knyght you can also read things as `Object` or `JsonNode` (tree model) instead of `Map`; and also use `ObjectMapper.convertValue(map, OtherType.class` to do conversions to actual type dynamically. – StaxMan May 04 '12 at 05:49
  • This is getting a bit long so I'll start a new question but for now this answer works at least. – Tom Carrick May 04 '12 at 14:24
1

Assuming you have an array wrapping your objects, create a JsonParser and then call readValuesAs with the appropriate type. It gives you back an Iterator with all your objects that reads through the file as you consume the objects.

Steven Schlansker
  • 37,580
  • 14
  • 81
  • 100