Get properties of a JSON object streamed as NDJSON using fetch

Question

I am trying to get NDJSON data from an API using fetch. Since I only need one JSON object, I would like to do this using fetch. The data provided by the API is in the form (formatting by me, actual response is a single line):

{
  "a": "value",
  "b": "value",
  "c": "value",
  "d": "value",
  "e": "value"
}

When I simply log the data, everything works fine and I get the above response as an object:

const obj = await fetch(url, {
  method: "GET"
}).then(res => res.json());
console.log(obj);

But when I try to log one of the properties of that object, nothing gets logged (no errors):

const obj = await fetch(url, {
  method: "GET"
}).then(res => res.json());
console.log(obj.a);

Even logging JSON.stringify(obj) does not work. How can I get around this issue?

Hey Wais - when you say `ndjson` you mean the endpoint responds with a number of these objects one per line, right? — Michał Karpacki, May 28 '21 at 11:20
Yes. The API has an option for the number of items to request. In my case I am only requesting one item. — Wais Kamal, May 28 '21 at 11:22
Can you try this: `scramjet.StringStream.from(() => fetch(url).then(res => res.body)).JSONParse().each(console.log).catch(console.error)` - you'd need to install `scramjet` and require it. — Michał Karpacki, May 28 '21 at 11:25
I understand that, I'll propose a way without additional deps - I just need to know what's the issue there to offer an answer. :) — Michał Karpacki, May 28 '21 at 11:38
The issue is that whenever I log a property or write it to a file, nothing is logged/written. — Wais Kamal, May 28 '21 at 11:57
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/233008/discussion-between-michal-karpacki-and-wais-kamal). — Michał Karpacki, May 28 '21 at 13:53

score 3 · Accepted Answer · answered May 28 '21 at 15:16

I'd like to drill down into what happened and point out couple problems you encountered - which probably made the whole thing much harder to solve than it should be.

I suspect that the API kept sending more than one line of data regardless of what you told,
ndjson is not a standard JSON string and fails,
Promises in node tend to fail silently if you don't add proper handling.

The three issues caused the result to be <nothing> while it should be an error explaining that the file cannot be parsed.

The solution I offered was to use fetch with scramjet like this:

const {StringSteram} = require("scramjet");

const stream = StringStream
  .from(async () => (await fetch(url)).body)
  .JSONParse()
;

StringStream.from accepts a stream or a method that returns one, and then the magic happens in JSONParse

the method takes every line apart
then parses that line as json

So now stream is a flowing list of objects. In node >= 12 you can simply iterate over it in a loop:

for await (const item of stream) {
   console.log(item);
}

And since the resulting stream class has some creature comfort functions if you just want the data as an Array:

console.log(await stream.toArray());

You don't have to use scramjet and you can work it out with just existing modules like this:

const { createInterface } = require("readline");
const { PassThrough } = require("stream");

const input = (await fetch(url)).body;
const output = new PassThrough();
createInterface({ input, output });

for await (const line of output) {
   console.log(JSON.parse(line));
}

Both solutions will take you there - with scramjet you can add more processing to the stream like: stream.filter(item => checkValid(item)) or whatever you may need, but in the end the goal can be reached either way.

Well laid out! But I am absolutely sure I am only getting one JSON object, because I can see the response when I type in the URL in the browser. Thank you, this helped. — Wais Kamal, May 28 '21 at 20:34
Yes there is actually, but does that make a difference? It should still be valid JSON. — Wais Kamal, May 28 '21 at 22:40
Not if that's just whitespace. If there are any non-whitespace characters then that would solve it, otherwise it should work with Scramjet as well as with just normal Fetch API `body.json()`. — Michał Karpacki, Jun 01 '21 at 20:13
I mean it's quite easy to say: found a solution, dunno why it works but it does. I'd take a look at the exact response, maybe even by dumping the response with `curl` and checking exactly what the API returns character by character. I mean it should just work as in your question, so this is a bit of a mistery - all scramjet does is splitting that response line by line. — Michał Karpacki, Jun 01 '21 at 20:15
There is a carriage return character followed by a newline character at the end of the NDJSON file, but nevertheless `JSON.parse("[]\r\n")` works correctly. — Wais Kamal, Jun 01 '21 at 21:50
That's true... I don't think then that the issue is in the server indeed. I'm not sure how this makes a difference. I'm happy it did though. :) — Michał Karpacki, Jun 02 '21 at 15:28

Get properties of a JSON object streamed as NDJSON using fetch

1 Answers1