How to parse ndjson in Pharo with NeoJSON

Question

I want to parse ndjson (newline delimited json) data with NeoJSON on Pharo Smalltalk.

ndjson data looks like this:

{"smalltalk": "cool"}
{"pharo": "cooler"}

At the moment I convert my file stream to a string, split it on newline and then parse the single parts using NeoJSON. This seems to use an unnecessary (and extremely huge) amount of memory and time, probably because of converting streams to strings and vice-versa all the time. What would be an efficient way to do this task?

If you look for sample data: NYPL-publicdomain: pd_items_1.ndjson

score 3 · Accepted Answer · answered Jan 21 '16 at 12:52

This the answer of Sven (the author of NeoJSON) at pharo-users mailing list (he is not on SO):

Reading the 'format' is easy, just keep on doing #next for each JSON expression (whitespace is ignored).

| data reader |
data := '{"smalltalk": "cool"}
{"pharo": "cooler"}'.
reader := NeoJSONReader on: data readStream.
Array streamContents: [ :out |
  [ reader atEnd ] whileFalse: [ out nextPut: reader next ] ].

Preventing intermediary data structures is easy too, use streaming.

| client reader data networkStream |
(client := ZnClient new)
  streaming: true;
  url: 'https://github.com/NYPL-publicdomain/data-and-utilities/blob/master/items/pd_items_1.ndjson?raw=true';
  get.
networkStream := ZnCharacterReadStream on: client contents.
reader := NeoJSONReader on: networkStream.
data := Array streamContents: [ :out |
  [ reader atEnd ] whileFalse: [ out nextPut: reader next ] ].
client close.
data.

It took a couple of seconds, it is 80MB+ over the network for 50K items after all.

score 2 · Answer 2 · answered Jan 21 '16 at 11:18

2

Would it work if you opened a new ReadWriteStream, first wrote ${ onto it, then stream all contents of your original stream separated by commas onto it and then write a trailing $}. The resulting stream should be good for NeoJSON... ? This is probably a STTCPW attack to the problem, but the W is impprtant ;-) And it should be faster and less memory consuming, because NeoJSON would just do one pass..

Just an idea, haven't tried it.

answered Jan 21 '16 at 11:18

Joachim Tuchel

447
2
10

I guess, you meant $[ and $] to make an array? This works well. – MartinW Jan 23 '16 at 11:52

score 1 · Answer 3 · answered Jan 21 '16 at 12:50

You could try something like this:

| input reader |
input := FileStream readOnlyFileNamed: 'resources/pd_items_1.ndjson.txt'.
[ 
Array
    streamContents: [ :strm | 
        | ln |
        [ (ln := input nextLine) isNil ] 
          whileFalse: [ strm nextPut: (NeoJSONReader fromString: ln) ] ] ] timeToRun.

Unless this is what you tried already...

How to parse ndjson in Pharo with NeoJSON

3 Answers3