0

I have a document that logs a user journey within my site.

Mouse movements are stored as a subdocument of this document. I log mouse movements then periodically use $push to append to the subdocument array. From my initial readings on Mongo this seemed to be a sensible/efficient approach. Now I am not so sure.

I have recently been having some issues with my replica set -secondaries fall out of sync with the primary when I have spikes in usage. Looking at my oplog I noticed that there were lots of $set commands each containing the whole history of the user's mouse movements up to that point and it seems that Mongo is doing this expansion so that oplog records can be indepotant.

Believe it is related to this issue on JIRA https://jira.mongodb.org/browse/SERVER-9784

Have I interpreted this issue correctly? If so, seems like a very inefficient way to be propagating changes over the wire. Is there any way to get around the problem? Would I be better off separating out the mouse movements into individual tiny documents? Or could I get around the problem by using dynamic keys on the subdocument array (therefore avoiding the $push command).

John Greenall
  • 1,670
  • 11
  • 17
  • So, the replica sets are in fact, not replicas? Is your code reading from secondaries and without a write concern that would assure consistency? – WiredPrairie Nov 18 '13 at 15:35
  • Not sure I understand your comment. I am using NEAREST write concern for reads. I realize that this means I might get slightly out of date information but that is not an issue for me. – John Greenall Nov 18 '13 at 15:48
  • 1
    @JohnGreenall You are using `$pushAll` not `$push`, right? – zero323 Nov 18 '13 at 15:51
  • I'm not using `$pushAll` since I want to log mouse movements as they come in. I do this because I don't want to lose record of users that suddenly navigate away from the page / close tab etc. Now wondering if I need to think of another way to do this? – John Greenall Nov 18 '13 at 15:59
  • I don't think so, but this behavior looks more like something you would expect from `$pushAll`. – zero323 Nov 18 '13 at 16:08
  • have been back to my code to check and can confirm that I am definitely doing a `$push` which I can then see show up in the oplog as `$set`. Running mongo version 2.4.3. – John Greenall Nov 18 '13 at 18:16
  • found related issue on JIRA and updated the question. I'm just baffled if this is the case. Seems like it makes updates to subdocuments chronically slow with replication! – John Greenall Nov 18 '13 at 18:31
  • So, you're basically concerned that the replicas are slow because of the way that `$push` works? So, yes, you'll need to find a different way to store them. Maybe stored in batches of smaller documents, without using `$push`. – WiredPrairie Nov 18 '13 at 18:35
  • Answering questions about the storage design would require more knowledge of how you plan on using all of the data later. – WiredPrairie Nov 18 '13 at 18:36

1 Answers1

0

Thanks to all for suggestions. Think I have got to the bottom of it now.

Looks like my problem is a bug.

Using $each in conjunction with $push in mongoDB is currently(v2.4.3) horrifically inefficient in replication terms. Have updated the related Jira issue to narrow the criteria.

https://jira.mongodb.org/browse/SERVER-9784

John Greenall
  • 1,670
  • 11
  • 17