How to force server-side insert on Meteor's collection.insert

Question

I've got a simple CSV file with 40,000 rows which I'm processing browser-side with papa-parse.

I'm trying to insert them one-by-one into a collection using the techniques in Discover Meteor and other 101 posts I find when Googling.

40000 insert browser-side pretty quickly - but when I check mongo server side it's only got 387 records.

Eventually (usually after 20 seconds or so) it starts to insert server-side. But if I close or interrupt the browser, the already-inserted records disappear obviously.

How do I force inserts to go server-side, or at least monitor so I know when to notify the user of success?

I tried Tracker.flush() no difference.

I'd go server-side inserts in a Meteor.method, but all the server-side CSV libraries are more complex to operate than client-side (I'm a beginner to pretty much everything programming :)

Thanks!

This is the main part of my code (inside client folder):

Template.hello.events({

    "submit form": function (event) {

        event.preventDefault();

        var reader = new FileReader();

        reader.onload = function (event) {

            var csv = Papa.parse(this.result, {header: true});

            var count = 0;

            _.forEach(csv.data, function (csvPerson) {
                count++;
                Person.insert(csvPerson);
                console.log('Inserting: ' + count + ' -> ' + csvPerson.FirstName);
            });

        };

        reader.readAsText(event.target[0].files[0]);
    }
});

The last few lines of console output:

Inserting: 39997 -> Joan 
Inserting: 39998 -> Sydnee 
Inserting: 39999 -> Yael 
Inserting: 40000 -> Kirk

The last few lines of CSV (random generated data):

Jescie,Ayala,27/10/82,"P.O. Box 289, 5336 Tristique Road",Dandenong,7903,VI,mus.Proin@gravida.co.uk
Joan,Petersen,01/09/61,299-1763 Aliquam Rd.,Sydney,1637,NS,sollicitudin@Donectempor.ca
Sydnee,Oliver,30/07/13,Ap #648-5619 Aliquam Av.,Albury,1084,NS,Nam@rutrumlorem.ca
Yael,Barton,30/12/66,521 Auctor. Rd.,South Perth,2343,WA,non.cursus.non@etcommodo.co.uk
Kirk,Camacho,25/09/08,"Ap #454-7701 A, Road",Stirling,3121,WA,dictum.eu@morbitristiquesenectus.com

The hello template is a simple form obviously, just file select and submit. Client code is under client directory. Person defined in a file in application root. CSV parsed as strings for now, to avoid complexity. The records inserted look fine, retrieve by name, whatever.

Person.find().count() browser-side in console results in 40000.

Happy to send the file, which is only 1.5MB and it's random data - not sensitive.

No, autopublish and insecure are still there, does this affect the answer? — almostawake, Nov 19 '14 at 06:57
No, only in the case were you don't have autopublish, if you need to publish and subscribe to the collection. Create a Collection on the client side that are not published by the server means to create local in-browser collection. This can be why on the client-side all your rows are inserted, on the other side, on the server you can have only partial data, maybe from a past run. Try to drop the Person database inside mongo and retry the import. If you see 0 records on mongo after your insert, this means that you are not using a published collection on the client. — markov00, Nov 20 '14 at 09:49

score 0 · Answer 1 · answered Nov 18 '14 at 10:15

0

I think call() should work as follows:

On client side

Meteor.call("insertMethod",csvPerson);

And method on server side

insertMethod: function(csvPerson){
Person.insert(csvPerson);
}

answered Nov 18 '14 at 10:15

sdooo

1,851
13
20

I will eventually do this server side as you have shown, but it limits my options with parsing libraries. For now I'm trying to get client-side working reliably. I figure there's a simple way to force insert, just haven't found it yet :) – almostawake Nov 19 '14 at 07:01
WEll that's weird your method doesn't work, does console log returns all records or just those inserted? – sdooo Nov 19 '14 at 12:29

score 0 · Answer 2 · answered Nov 19 '14 at 01:24

In Meteor, on some scenarios, if you don't pass a callback the operation will sync.

If you run the code Person.insert(csvPerson); on the server, the operation will be sync not async. Depending on what you want to do, you might have serious problems in the future. On the client, it won't be sync but async.

Since node.js is an event-based server, a single sync operation can halt the entire system. You've to be really about your sync operations.

For importing data, the best option is to do at server-side inside Meteor.startup(function(){ //import code goes here}).

The solution propose by Sindis works but it slow and if the browser closes (for some reason), you're not keeping a track of the already inserted records. If you use Meteor.call("insertMethod",csvPerson);, this operation will be sync on the client.

The best option on your beginner scenario (not optimal) is to:

1- While (You have record to insert)

2- Call Meteor.call without a callback

3- Count all the inserted fields in the Collection

4- Save this value to localStorage

5- Go back to step 1

This works assuming that the order of insertion is the same on every insert attempt. If you browser fails, you can always get the value from localStorage and skip that number of records.

Thanks Mário I will definitely be doing this server side once I "get it". Your answer helps but -> for this current solution, is there any built-in way that I can check whether the client or server has outstanding (async) operations queued up to be processed? Ideally something that allows me differentiate between normal DOM/session stuff and this type of mongo operation? If not, I'll close on your answer anyway but I'd love to understand my options for educational purposes :) — almostawake, Nov 19 '14 at 07:09

How to force server-side insert on Meteor's collection.insert

2 Answers2