How do I parse ndjson files?

Question

I performed a GET request from lichess.org, API reference: https://lichess.org/api#operation/apiGamesUser

import requests
import json
import ndjson
response = requests.get('https://lichess.org/api/games/user/ardito_bryan', params={'max':10})

I am unable to decode ndjson, python library here: https://pypi.org/project/ndjson/ According to the library reference I should be using:

items = response.json(cls=ndjson.Decoder)

It does not work, runs error:

JSONDecodeError: Expecting value: line 1 column 3 (char 2)

Ty.

score 3 · Answer 1 · answered Oct 18 '20 at 18:40

Fixed it by adding in the following header to the request.

headers={
'Authorization': f'Bearer {token}', # Need this or you will get a 401: Not Authorized response
"Accept": "application/x-ndjson"
}

After this I convert the ndjson by:

resp_json = []
ndjson = response.content.decode().split('\n')

for json_obj in ndjson:
    if json_obj:
        resp_json.append(json.loads(json_obj))

Now you will have your parsable object.

score 0 · Answer 2 · answered Aug 15 '23 at 14:40

here is how I implemented it. In this case the api is returning a ndjson as a stream (which is what lichess also does on some endpoints). I am reading the stream in chunks with a reader. In ndjson format, data is split by new lines, so each line by itself is a basic json which I parsed and added to fetchedData variable.

var fetchedData = [];

fetch('LinkGoesHere', {
    method: 'get',
    headers: {
        'Authorization': 'Bearer TokenGoesHere' // this part is irrelevant and you may not need it for your application
    }
})
.then(response => {
    if (!response.ok) {
        throw new Error(`HTTP error! Status: ${response.status}`);
    }
    return response.body.getReader();
})
.then(reader => {
    let partialData = '';

    // Read and process the NDJSON response
    return reader.read().then(function processResult(result) {
        if (result.done) {
            return;
        }

        partialData += new TextDecoder().decode(result.value, { stream: true });
        const lines = partialData.split('\n');

        for (let i = 0; i < lines.length - 1; i++) {
            const json = JSON.parse(lines[i]);
            fetchedData.push(json); // Store the parsed JSON object in the array
        }

        partialData = lines[lines.length - 1];

        return reader.read().then(processResult);
    });
})
.then(() => {
    // At this point, fetchedData contains all the parsed JSON objects
    console.log(fetchedData);
})
.catch(error => {
    console.error('Fetch error:', error);
});

How do I parse ndjson files?

2 Answers2