High insertion times in mongoDB

Question

We have a collection which feeds from several (3) UDP servers which get a entry and puts it into mongodb.

I've started to check profile on mongodb and witnessed that some time (most of the time it's fine) I get high (~5000) for insertion.

There are some MapReduce operations working in a Cronjob from another component on nodejs.

I can notice there is alot (50~) of insertion operation at the same time.

There is 2 indices on that collection, and every couple of seconds most of it's documents are removed so it can't have more than about 50. The collection isn't capped.

MongoDB version is 2.4.6

What can cause this? Can a MapReduce which runs at the same time on a different collection be the cause ?

   {
        "allUsers": [],
        "client": "127.0.0.1",
        "keyUpdates": 0,
        "lockStats": {
            "timeAcquiringMicros": {
                "r": 0,
                "w": 10
            },
            "timeLockedMicros": {
                "r": 0,
                "w": 45
            }
        },
        "millis": 9527,
        "ninserted": 1,
        "ns": "dbname.hits",
        "numYield": 0,
        "op": "insert",
        "ts": {
            "$date": 1418186296785
        },
        "user": ""
    },
    {
        "allUsers": [],
        "client": "127.0.0.1",
        "keyUpdates": 0,
        "lockStats": {
            "timeAcquiringMicros": {
                "r": 0,
                "w": 8
            },
            "timeLockedMicros": {
                "r": 0,
                "w": 35
            }
        },
        "millis": 9396,
        "ninserted": 1,
        "ns": "dbname.hits",
        "numYield": 0,
        "op": "insert",
        "ts": {
            "$date": 1418186296785
        },
        "user": ""
    },
    {
        "allUsers": [],
        "client": "127.0.0.1",
        "keyUpdates": 0,
        "lockStats": {
            "timeAcquiringMicros": {
                "r": 0,
                "w": 6
            },
            "timeLockedMicros": {
                "r": 0,
                "w": 29
            }
        },
        "millis": 9257,
        "ninserted": 1,
        "ns": "dbname.hits",
        "numYield": 0,
        "op": "insert",
        "ts": {
            "$date": 1418186296785
        },
        "user": ""
    },
    {
        "allUsers": [],
        "client": "127.0.0.1",
        "keyUpdates": 0,
        "lockStats": {
            "timeAcquiringMicros": {
                "r": 0,
                "w": 7
            },
            "timeLockedMicros": {
                "r": 0,
                "w": 65
            }
        },
        "millis": 8768,
        "ninserted": 1,
        "ns": "dbname.hits",
        "numYield": 0,
        "op": "insert",
        "ts": {
            "$date": 1418186296785
        },
        "user": ""
    },
    {
        "allUsers": [],
        "client": "127.0.0.1",
        "keyUpdates": 0,
        "lockStats": {
            "timeAcquiringMicros": {
                "r": 0,
                "w": 22
            },
            "timeLockedMicros": {
                "r": 0,
                "w": 62
            }
        },
        "millis": 8566,
        "ninserted": 1,
        "ns": "dbname.hits",
        "numYield": 0,
        "op": "insert",
        "ts": {
            "$date": 1418186296786
        },
        "user": ""
    },
    ...

Update 1:

The problem I'm trying to solve is:

We get hits from some servers for existence of a match on several streams of data, I need to aggregate them to a [begin, end] tuples if they have a difference of MAX_DIFF between them. For example:

{name:A, time:0, stream:A_1}
{name:A, time:7, stream:A_1}
{name:A, time:15, stream:A_1}
{name:A, time:26, stream:A_1}

MAX_DIFF= 10

I'll aggregate to another collection:

{name:A, stream:A_1, time_begin:0, time_end:15}
{name:A, stream:A_1, time_begin:26, time_end:26}

Update 2: After some checking in the profile, it seems like Map Reduce is the culprit, using it's Global write lock at final stage. map reduce concurrency There are some operation I give to MR to do on the entire site which takes a lot of time.

score 2 · Accepted Answer · answered Dec 10 '14 at 14:03

There are lots of possible issues here - these are the main ones as I see them:

MongoDB locks the entire database during a write. With so many writes the DB will be locked for most of the time. Which write concern are you using for your writes?
Constant deletes. It is generally a bad idea to constantly delete data. Could you perhaps re-model your data to avoid constant deletes? Also with only 50 documents an index is a waste of time.
Storage. You don't mention what kind of storage you are using, but MongoDB really likes SSd's
2.6 was a big improvement over 2.4 so you should really look into upgrading.

You don't mention what type of problem you are trying to solve, but with data being deleted with the same rate inserts then a database might not be the best fit for your problem.

1. I think the write concern is defaulted to Acknowledged, 2. I need to set that I've looked at an entry, I think it's a good idea to consider another model. 3. I think it's EBS and not SSD. 4. I agree. My main question if it is reasonable to get such high time for these parameters — shevski, Dec 10 '14 at 15:02

High insertion times in mongoDB

1 Answers1