0

I'm trying to create a XLSX file from json, but when I'm trying this code:

const convertJsonToExcel = () => {
    

    //var objeto=JSON.parse(JSON.stringify(jsonData))
    console.log(typeof(bigJson))

    const workSheet = XLSX.utils.json_to_sheet(bigJson['titulo']);
    const workBook = XLSX.utils.book_new();

    XLSX.utils.book_append_sheet(workBook, workSheet, "students")
    // Generate buffer
    XLSX.write(workBook, { bookType: 'xlsx', type: "buffer" })

    // Binary string
    XLSX.write(workBook, { bookType: "xlsx", type: "binary" })

    XLSX.writeFile(workBook, "studentsData.xlsx")
}

I get: FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

This json file is 132MBs, but I'll have files until 250MBs. I looked for ExcelJS library, but it has some issues too : https://github.com/exceljs/exceljs/issues/709

What would be a good strategy?

EDIT:

const convertJsonToExcel = () => {

var arr=bigJson['titulo']

// arr2=arr.slice(arr.length/2, arr.length-1)
// arr=arr.slice(0, arr.length/2)
// arr3=arr.slice(0, 1)

console.log(typeof(bigJson))

const workSheet = XLSX.utils.json_to_sheet(arr);

//XLSX.utils.sheet_add_json(workSheet,arr2,{origin: -1});

const workBook = XLSX.utils.book_new();
XLSX.utils.book_append_sheet(workBook, workSheet, "students")
XLSX.writeFile(workBook, "studentsData.xlsx")
}

I was trying to create a xlsx file from a json of 500MBs, but it only keeps working without response. This works for 250MBs file.

I tried with the commented part dividing in two blocks the json file, and not getting response again

Will
  • 49
  • 8
  • Other than not to use JavaScript for this? Why do you have multiple files with up to 250Mb of "students data" as JSON in the first place? – Andreas Feb 24 '22 at 16:34
  • I need it to upload it to Google Storage and people could download it – Will Feb 24 '22 at 16:38

1 Answers1

0

Did you try to run your script with params to increase your max heap size? You might reach the heap limit of node.

node --max_old_space_size=4096 app.js

reference: https://nodejs.org/api/cli.html#--max-old-space-sizesize-in-megabytes

buithienquyet
  • 351
  • 1
  • 6
  • Still getting FATAL ERROR: invalid array length Allocation failed - JavaScript heap out of memory – Will Feb 24 '22 at 16:58
  • try removing "Generate buffer " and "Binary string" ,your file is too large so it can be very very large to buffer (check more https://stackoverflow.com/questions/63937742/nodejs-invalid-array-length-allocation-failed-javascript-heap-out-of-memory) – buithienquyet Feb 24 '22 at 17:05
  • It works for 250 MBs ty, I tried with 430 MBs and it only keeps running but not response – Will Feb 24 '22 at 17:46
  • you mentioned max is "250MBs" :), 430MB seem be too large => after parsing much more space and time needed, you may find out some other solutions, I can't help more :) – buithienquyet Feb 24 '22 at 18:10