2

Right now, I'm using SuperObject to parse my JSON data (when receiving it from the server) and then dump it into a client dataset. There can be very large amounts of data coming from the server. Currently, I have to first parse that JSON data into an ISuperObject, and then iterate through that to populate the client dataset.

What I'm wondering is whether there's a way to bypass the double-loading, triggering events as the raw data is being parsed. For example, as soon as the parser detects the beginning of a new object, it would trigger an event that I would then prepare that corresponding object on the spot. Or when it parses an array, I can prepare a new dataset record.

The reason is because with very large datasets, it takes a few seconds some times, and I have to wait for it to finish being parsed before I can make use of it. If I get the data while it's being parsed, I can immediately make use of that data on the spot.

I'm already implementing pagination at between 200-500 records per page (on millions of records) with many columns. Pagination still doesn't solve the full needs when it comes to responsiveness.

How can I go about this, either using SuperObject, or any other known mechanism, without having to write my own complete parser?

Jerry Dodge
  • 26,858
  • 31
  • 155
  • 327
  • PS - I will award accepted answer with a bounty of 100 – Jerry Dodge Dec 22 '13 at 23:48
  • 2
    Did you try to use `AsMethod`? or `.M[]`? This example shows how you can define a callback when a certain attribute is found: https://code.google.com/p/superobject/source/browse/demos/googlesearch/main.pas I just didn't look if it call those on the fly, or that it first parses everything. It's probably faster to just try it.. – Wouter van Nifterick Dec 23 '13 at 03:20
  • @Wouter Thanks I've never understood the .M until now ;-) – Jan Doggen Dec 23 '13 at 07:17
  • 2
    http://stackoverflow.com/questions/413139/requirements-for-json-parser You are looking for pull based or SAX style parsers. – David Heffernan Dec 23 '13 at 09:17
  • @DavidHeffernan i think your link might just be a duplicate for this question. At least i think mORMot parser - http://stackoverflow.com/a/6241372/976391 - would be fastest if maybe not so easily used as SO – Arioch 'The Dec 23 '13 at 13:16

1 Answers1

0

Writing your own parser is not that hard. I've written one for the mongodb connector I've written: https://github.com/stijnsanders/TMongoWire/blob/master/bsonUtils.pas You could easily adapt that one to fire the events you describe.

Stijn Sanders
  • 35,982
  • 11
  • 45
  • 67