Procedural 2d-array mongoDB insert on node.js runs out of memory

Question

I am trying to insert a 2000 x 2000 2d array into my mongoDB instance via my node.js server.

Right now the way i'm doing it is:

for (i=0; i<2000; i++){
            //i = height of the tile map
            for(x=0; x<2000; x++){
                //x is the width of the tile map
                driver.mongo.insert('worldTiles', {loc: [x,i], elevation: obj.server.theArray[i][x]}, function (err,results){
                    if(!err){

                    }else{
                        console.log('Ran into an error inserting into worldTiles on Mongo');
                    }
                })
            }
            console.log(i);
        }

Now, the problem I am facing is that at around line 576 my node.js instance slows down dramatically (one line processed per 40 seconds or so) and then runs out of memory. I don't understand how/why this is happening.

Am I approaching this problem correctly? Any ideas on how I can get this 2d array into my database without floundering?

Thanks for your help all.

What mongodb driver is this? I'm not familiar with the `driver.mongo.insert` syntax. — JohnnyHK, Aug 23 '12 at 05:20
It's a private driver that is pretty basic, just wrapping the nativeParser. I could try it with another driver... any that you recommend? — Mavorus, Aug 23 '12 at 07:47
The standard 10gen supported one for JavaScript: https://github.com/mongodb/node-mongodb-native — JohnnyHK, Aug 23 '12 at 12:59

score 1 · Accepted Answer · answered Aug 24 '12 at 00:08

It looks like the problem is with the provider of my mongoDB instance. I just found out that when spooling a mongo instance on jitsu there is a ~64MB limit on the size of the memory usage. Thus, at exactly that point, I got the "out of memory" error.

Wish they would somehow make the error message a bit more descriptive including what device was running out of memory... rolls eyes.

Case closed.

Procedural 2d-array mongoDB insert on node.js runs out of memory

1 Answers1