Loading large CSV file with Papaparse not working (only first chunk loaded)

Question

I would like to load a local file (client-side) with papaparse into my React application. Unfortunately, it only loads the first chunk but never the whole file. My file contains about 500 rows and there are never more than 300 rows loaded. It seems like the complete functions is already called after the first chunk.

Since I need to navigate to another page when the file is loaded completely, this is bothering me since I need the complete file for further functions.

The code I use at the moment:

    async getData() {
        const self = this;
        let dataList = [];
        Papa.parse(await this.fetchCsv(),
            {
                delimiter: ',',
                header: true,
                chunk: function (result, parser) {
                    parser.pause();
                    dataList = dataList.concat(result.data)
                    parser.resume();
                },
                complete: function () {
                    self.updateData(dataList);
                }
            });
    }

    async fetchCsv() {
        const response = await fetch(this.props.location.state.filename);
        const reader = response.body.getReader();
        const result = await reader.read();
        const decoder = new TextDecoder('utf-8');
        return decoder.decode(result.value);
    }

What I've also tried is using step instead of chunk but this did not change anything. Can anyone tell me what I'm doing wrong here and why papaparse does not load the whole file?

score 0 · Answer 1 · answered Sep 09 '21 at 16:03

You may be able to let papaparse do more. It can read local File or stream data from a remote server.

If you only have about 500 records, you may not need to add the complexity associated with streaming. This is especially true if you're just accumulating the data (which it appears you are). Use streaming primarily to process the data 1 record at a time.

If you want to stream, I'd recommend using the "step" callback instead of the "chunk" callback so you can process each row of data.

If you use the step or chunk callbacks, then you don't need the complete callback. If it's called, it won't have the data.

Loading large CSV file with Papaparse not working (only first chunk loaded)

1 Answers1