Transforming large array of objects to csv using json2csv

Question

I need to transform a large array of JSON (that can have over 100k positions) into a CSV. This array is created directly in the application, it's not the result of an uploaded file.
Looking at the documentation, I've thought on using parser but it says that:
For that reason is rarely a good reason to use it until your data is very small or your application doesn't do anything else.
Because the data is not small and my app will do other things than creating the csv, I don't think it'll be the best approach but I may be misunderstanding the documentation.
Is it possible to use the others options (async parser or transform) with an already created data (and not a stream of data)?

FYI: It's a nest application but I'm using this node.js lib.
Update: I've tryied to insert with an array with over 300k positions, and it went smoothly.

not so sure that I understand your question? what is current input, I mean what is already provided (JSON file or transformed CSV file)? Where do you want to return? from controller response as a CSV file or raw/plain data? — Pete Houston, May 12 '22 at 21:29
My input is provided through datebase's results. Then I created my variable that will store the array of JSON and finally I need to create this csv (with the variable). Later on, I'll send this csv to AWS s3. — PedroSG, May 13 '22 at 14:39
then it is very straight-forward, load from database, transform into CSV file, upload file to S3. what's the problem you face here? — Pete Houston, May 14 '22 at 03:21

score 1 · Answer 1 · answered May 12 '22 at 16:19

Why do you need any external modules? Converting JSON into a javascript array of javascript objects is a piece of cake with the native JSON.parse() function.

    let jsontxt=await fs.readFile('mythings.json','uft8');
    let mythings = JSON.parse(jsontxt);
    if (!Array.isArray(mythings)) throw "Oooops, stranger things happen!"

And, then, converting a javascript array into a CSV is very straightforward. The most obvious and absurd case is just mapping every element of the array into a string that is the JSON representation of the object element. You end up with a useless CSV with a single column containing every element of your original array. And then joining the resulting strings array into a single string, separated by newlines \n. It's good for nothing but, heck, it's a CSV!

 let csvtxt = mythings.map(JSON.stringify).join("\n");
 await fs.writeFile("mythings.csv",csvtxt,"utf8");

Now, you can feel that you are almost there. Replace the useless mapping function into your own

 let csvtxt = mythings.map(mapElementToColumns).join("\n");

and choose a good mapping between the fields of the objects of your array, and the columns of your csv.

 function mapElementToColumns(element) {
   return `${JSON.stringify(element.id)},${JSON.stringify(element.name)},${JSON.stringify(element.value)}`;
 }

or, in a more thorough way

  function mapElementToColumns(fieldNames) {
    return function (element) {
      let fields = fieldnames.map(n => element[n] ? JSON.stringify(element[n]) : '""');
      return fields.join(',');
    }
  }

that you may invoke in your map

  mythings.map(mapElementToColumns(["id","name","element"])).join("\n");

Finally, you might decide to use an automated for "all fields in all objects" approach; which requires that all the objects in the original array maintain a similar fields schema.

You extract all the fields of the first object of the array, and use them as the header row of the csv and as the template for extracting the rest of the elements.

    let fieldnames = Object.keys(mythings[0]);

and then use this field names array as parameter of your map function

  let csvtxt= mythings.map(mapElementToColumns(fieldnames)).join("\n");

and, also, prepending them as the CSV header

 csvtxt.unshift(fieldnames.join(','))

Putting all the pieces together...

function mapElementToColumns(fieldNames) {
  return function (element) {
    let fields = fieldnames.map(n => element[n] ? JSON.stringify(element[n]) : '""');
    return fields.join(',');
  }
}

let jsontxt=await fs.readFile('mythings.json','uft8');
let mythings = JSON.parse(jsontxt);
if (!Array.isArray(mythings)) throw "Oooops, stranger things happen!";
let fieldnames = Object.keys(mythings[0]);
let csvtxt= mythings.map(mapElementToColumns(fieldnames)).join("\n");
csvtxt.unshift(fieldnames.join(','));
await fs.writeFile("mythings.csv",csvtxt,"utf8");

And that's it. Pretty neat, uh?

Transforming large array of objects to csv using json2csv

1 Answers1