Papa parser to await for step

Question

I was able to configure all in papa parser and it works well even with 200k items parsing. So I can process all the data and them await for each row in the result array. the problem is if I am using a step how I can make the step await the previous step? This is causing the parser to fires thousands of calls and then blocking me. I tried to wrapped the step in another function but that did not help either.

here is a sample of what I am doing in the step call

async function stepFn(results, parser) {
stepped++;
      (await (async function () {
        if (results) {
          if (results.data) {
            rowCount += results.data.length;
            console.log('checking if item exists');
            var itemExistsP = await restcallToRetrieve();
            if (itemExistsP.length > 0) {
              console.log('Item exists, skipping');
              //return;
            } else {
              console.log('adding item to list');
              await restcalltoAdd();
              // return item;
            }
          }
          if (results.errors) {
            errorCount += results.errors.length;
            firstError = firstError || results.errors[0];
          }
        }
      }()));
}

Here is the Parser call:

Papa.parse($('#parse-file')[0].files[0], {
            delimiter: '',
            header: true,
            dynamicTyping: false,
            skipEmptyLines: true,
            preview: 0,
            encoding: '',
            step: stepFn,
            worker: false,
            comments: '',
            download: false,
            complete: completeFn,
            error: errorFn,
            });

How are you actually using this `stepFn` in your parser? That will need to change, please show us that code. — Bergi, Sep 29 '20 at 12:55
That `await (async function() { … }());` is pointless, drop the wrapper and just put the code right in the `stepFn` body. — Bergi, Sep 29 '20 at 12:56
@Bergi, I am using with the config object. edited the question with the code. — Ricardo Silva, Sep 29 '20 at 14:12
Thanks. From what I see in the docs, it doesn't look like papaparse does support asynchronous stream processing. I think there are some libraries (maybe node.js backend only) that do support asynchronous iterators which would greatly simplify this. — Bergi, Sep 29 '20 at 15:08

Laurent Dhont · Accepted Answer · 2020-09-30T11:38:19.560

5

Just create an async stepFunction:

async function stepFn(results, parser) {
    parser.pause(); // pause the parser

    if (results.data) {
        rowCount += results.data.length;

        let itemExists = await restCallToRetrieve(/*probably the data goes here*/);
        if (itemExists.length > 0) {
            console.log('Item exists, skipping');
        }
        else {
            console.log('adding item to list');
            await restCallToAdd(/*probably the data goes here*/);
        }
    }
    else if (results.errors.length > 0) { //Note that I am checking if there are errors
        errorCount += results.errors.length;
        firstError = results.errors[0];
    }
    parser.resume(); // resume the parser
}

Then in the parse config object:

Papa.parse(file, {
    step: stepFn
});

It would also be good to put everything you await in a try catch block but that's of course beside the point.

edited Sep 30 '20 at 11:38

answered Sep 29 '20 at 13:00

Laurent Dhont

1,012
1
9
22

I tried that but still the step will be called faster than the api can handle. I think it falls in the same situation as mapping, since it is callback based await alone won't work. or maximum chunck each step into a 50-100 calls await all and then move on. But again the issue is that step is called internally I am just providing the config. So I don't know how to handle this case. – Ricardo Silva Sep 29 '20 at 14:11
@RicardoSilva Ahn, okay I understand the problem now. Let me check. – Laurent Dhont Sep 29 '20 at 14:23
1

@RicardoSilva I edited the answer, please note the parser.pause() and parser.resume() – Laurent Dhont Sep 29 '20 at 14:27
Thanks I will give it a try and accept the answer if this works. – Ricardo Silva Sep 29 '20 at 15:36
Thanks for the answer, with `parser.pause()` and `parser.resume()` it works reliably. – György Balássy Jun 26 '23 at 06:25

Papa parser to await for step

1 Answers1