0

I'm trying to parse the data in my csv with Papa Parse, but the returned object is only 50,000 of the million rows. I attempted to use the chunk callback function to break up the file into smaller components, but haven't had any luck because I don't understand where I pass in the break up size and where I call back the function. my code is here...

function loadData(evt){
var rowCount = 0;
var file = evt.target.files[0];
Papa.parse(file, {
    header:true,
    delimeter: ',',
    worker:true,

    // chunk: function(results){
    //    rowCount += results.data.length;
    // },
    complete: function(results) {
        csvData = results
        console.log(csvData);
    }
}); 
}

I know that my rowCount variable is not being used but I am unsure where I pass that in. i've looked a the documentation online but still don't really understand the chunk function, so any additional documentation or advice would be appreciated. thanks.

user3014093
  • 389
  • 2
  • 5
  • 20
  • 1
    Does the 50,000 rows happen to be the last 50,000 rows of the file? – Matt Aug 27 '15 at 16:29
  • Yes, the last 50,000 rows. – user3014093 Aug 27 '15 at 17:02
  • and its 53,293 rows, not 50,000 if that matters – user3014093 Aug 27 '15 at 17:10
  • Thanks for checking. That's expected. The complete callback, in conjunction with chunk, only has the last chunk of rows available to it (and the exact count varies based on chunk size). Basically, inside your chunk callback, the results of each chunk are available, but by the time complete is called, only the last chunk's rows are available. – Matt Aug 27 '15 at 18:38
  • Thanks for responding, when I look at the Papa variable the local chunk size is 10485760, and the remote size if half of that, so how do I move the chunk then 53293 in the callback, which seems to me like a random value. How are you saying I should write the chunk callback? Thanks! – user3014093 Aug 27 '15 at 18:55
  • You shouldn't need to mess with chunk size. Just deal with the results as they come into the chunk callback, chunk-by-chunk. Each time that callback is executed, a new chunk of data is passed into it. – Matt Aug 27 '15 at 18:57
  • Awesome, all works now. thanks. – user3014093 Aug 27 '15 at 22:11
  • How did you break the CSV file into chunks before the callback is executed ? – Anubhav Gupta Mar 03 '17 at 06:10

0 Answers0