CSV parse multiple files

Question

My goal: I need to grab N number of files from an FTP, parse them, and send the parsed data somewhere. My current implementation is working fine but when I get past the first file it begins to hang.

The package I am using for the csv stream / parse is fast-csv. Below is an example of what I was trying to accomplish. It only ever runs 1 time. I have a feeling it has something to do with the stream not ending or finishing properly. I was hoping someone could give me some pointers on a direction I could take. Let me know if you need anymore clarification.

async handleStreams (files) {
  for await (const file of files) {
    await this.func(file)
  }
}

async func (file) {
    return new Promise((resolve, reject) => {
    const stream = readableStream(file)
    stream
      .on('error', error => { 

      })
      .on('data', (row: string[]) => {
        // do stuff
                stream.end()
      })
      .on('end', async (rowCount: Number) => { 
        await sendOff()
        // resolve()????
      })
      .on('close', async () => {
                // resolve()????
      })
  })  
}

*My current implementation is working fine but when I get past the first file it begins to hang* so, it isn't working fine, really - since your code doesn't resolve the promise, the await will wait forever - also, do you really need to `for await`? is `files` an *async iterable object* — Jaromanda X, Apr 11 '20 at 01:08
I have tried resolving where I have it commented at but neither seemed to work. I could probably omit the await in the loop. — ColorCutClarity, Apr 11 '20 at 01:13
perhaps some console.log's in the `func` function to trace what is happening will help — Jaromanda X, Apr 11 '20 at 01:16
I have tried logging things. It logs, it just never resolves. within the .on listeners — ColorCutClarity, Apr 11 '20 at 01:22
It does, I can console log within them but the resolve never triggers — ColorCutClarity, Apr 11 '20 at 01:32
which resolve, the one after await sendOff or the one in on close? perhaps it's a typescript issue — Jaromanda X, Apr 11 '20 at 01:34
Neither one works. How could typescript be the culprit? some type of rule? — ColorCutClarity, Apr 11 '20 at 01:37
no idea, I don't use typescript ... if you put console.log either side of the resolve, do either of the console.log's output? perhaps there's no `end` or `close` events happening at all — Jaromanda X, Apr 11 '20 at 03:22

CSV parse multiple files

0 Answers0