CSV convert to JSON Returning as an empty array

Question

I have an API route in my next.js project, the idea is that it process the stats from an API the company I work for is using. It makes it's way through the whole function without issue until it comes time to convert the CSV to a JSON array, then it just returns as []

I figured out if I remove the CSVtoJSON conversion code and run to download the file, then alter the code so it has the CSVtoJSON code but removing the download of the file it will parse the CSV into the JSON I want without issue.

I am using asyn/await functions so I am not sure why it is reading the file as empty when I try to run it as one whole function but not if I run each chunk individually.

// logs.ts
import {NextApiRequest, NextApiResponse} from 'next'
import {processStats, fetchCsv, downloadFile} from './functions'

export default async (req: NextApiRequest, res: NextApiResponse) => {
 const request = await processStats(1, 1, 'UTC', 'records', 'calls')
 const csv_url = await fetchCsv(request.request_id)
 const parsedf = await downloadFile(csv_url.download_url)
 res.status(200).json(parsedf)
}

// downloadFile function
export async function downloadFile(url) {
  const fs = require('fs')
  const https = require('https')
  const csv = require('csvtojson')

  const fp = await fs.createWriteStream('pages/api/stats/data.csv')
  const req = await https.get(url, function (res) {
    res.pipe(fp)
  })

  const jsonArray = await csv().fromFile('pages/api/stats/data.csv')
  return jsonArray
}

No errors are thrown it just sends back the empty [] array.

Any help would be amazingly appreciated.

Mac

Does this answer your question? [Read remote file with node.js (http.get)](https://stackoverflow.com/questions/14552638/read-remote-file-with-node-js-http-get) — traktor, Jun 16 '21 at 12:23

traktor · Answer 1 · 2021-06-16T12:15:03.747

 const fp = await fs.createWriteStream('pages/api/stats/data.csv')
 const req = await https.get(url, function (res) {
    res.pipe(fp)
 })

only waits for a microtask queue job to run because https.get doesn't return a promise. It needs to be rewritten to handle getting and writing all the data from the get request to the pipe, and possibly closing the pipe before initiating a call to

 const jsonArray = await csv().fromFile('pages/api/stats/data.csv')

The best resources to start with are the documentation for https.get and http.get in combination. As I understand it, monitoring the data and end events fired on the response object will be needed.

Search this site with "[node.js] http.get data chunk]" in the (site) search box for more information.

score 0 · Answer 2 · answered Jun 16 '21 at 13:04

Just as I was about to give up, I figured out how to fix this.

All I needed to do was read the contents of the file into a string using fs.readFileSync('pages/api/stats/data.csv) and then return csv().fromString(data.toString())

this worked. Final code below.

export async function downloadFile(url) {
  const https = require('https')
  const fs = require('fs')
  const fp = fs.createWriteStream('pages/api/stats/data.csv')
  const csv = require('csvtojson')

  https.get(url, function (res) {
    res.pipe(fp)
  })
  var data = fs.readFileSync('pages/api/stats/data.csv')
  return csv().fromString(data.toString())
}

CSV convert to JSON Returning as an empty array

2 Answers2