4

I have a JSON file which I am reading with node, modifying and saving it as a json file.

I'm looking to save the new json as newline delimited vs being in an array.

I came across https://github.com/CrowdProcess/newline-json but don't fully understand streams. If i have the following stream setup, how can I pipe it though the parser and stringifier?

fileStream = fs.createReadStream('source.json')
writeStream = fs.createWriteStream('output.txt');

var Parser = require('newline-json').Parser;
var Stringifier = require('newline-json').Stringifier;

var parser = new Parser();
var stringifier = new Stringifier();

But running the following only outputs a blank file.

fileStream.pipe(parser).pipe(stringifier).pipe(writeStream)

what am I missing about streams?

peak
  • 105,803
  • 17
  • 152
  • 177
kisonay
  • 351
  • 2
  • 7
  • 15

3 Answers3

18

One way to convert a JSON array to a stream of newline-delimited JSON entities is to use jq with the -c option, e.g.

$ jq -c ".[]"

Input:

[[1,2], 3, {"4":5}]

Output:

[1,2]
3
{"4":5}

See https://stedolan.github.io/jq

peak
  • 105,803
  • 17
  • 152
  • 177
  • jq looks like a great tool. I guess at this point my problem is understanding exactly how streaming works. How would I be able to read a file, run it through jq and write the output? All of the documentation regarding streaming (and I don't need to use streaming) is very confusing to me. – kisonay Sep 24 '15 at 02:10
  • 1
    @kisonay - You don't need to worry about the "streaming parser" at all. If your JSON is in a file, say input.json, you could (for example) just give the filename as a command-line argument: jq -c ".[]" input.json – peak Sep 24 '15 at 02:27
  • your a lifesaver! just sending the output to a file `> output.txt` did exactly what I needed. Leave it to me to overcomplicate things! Thank you. – kisonay Sep 24 '15 at 11:56
16

For anybody who is looking to a solution on how to convert json array of objects to nd-json. Here is the solution:

Input:

const arrObj = [{
  id: 1,
  name: 'joe'
}, {
  id: 2,
  name: 'ben'
}, {
  id: 3,
  name: 'jake'
}, {
  id: 4,
  name: 'marsh'
}];

// stringify the object and join with \n
const ndJson = arrObj.map(JSON.stringify).join('\n');

console.log(ndJson);

Output:

{"id":1,"name":"joe"}
{"id":2,"name":"ben"}
{"id":3,"name":"jake"}
{"id":4,"name":"marsh"}

Example Usecase: When importing bulk request from a json file to elasticsearch.

Happy Coding :)

Vishnu
  • 704
  • 12
  • 34
  • 1
    The Elasticsearch Bulk API was my exact use case for finding this thread and this comment is a really great start. However, for the Bulk API, you also want to remember to add your headers and append a newline character at the end of the file. Here's a more complete example: ` ` ` const header = { "index": {} }; const ndJson = data.flatMap(d => [header, d]).map(d => JSON.stringify(d)).join('\n').concat('\n'); ` ` ` – annmarie-switzer Oct 15 '21 at 01:04
1

In node.js you can use the node-jq package to do what @peak has shown above.

var stream = require('stream');
var fs = require('fs');
const jq = require('node-jq');

var fileName = 'YOUR_FILE_NAME'; //e.g abc.json
var bucketName = 'YOUR_BUCKET NAME'; // e.g gs://def
var dataStream = new stream.PassThrough();


async function formatJson() {

    jq.run('.[]', fileName, {output: 'compact'})
    .then((output) => {
        dataStream.push(output)
        dataStream.push(null)
        console.log(dataStream)    
    })
    .catch((err) => {
        console.log(err)
    })
}

formatJson()

I am not an experienced node person, so apologies if the code is clumsy but it works.

Urvah Shabbir
  • 945
  • 2
  • 15
  • 46