1

I have a node.js app with mongodb backend going to production in a week and i have few doubts on how to handle app crashes and restart .

Say i have a simple route /followUser in which i have 2 database operations

/followUser
----->Update User1 Document.followers = User2
----->Update User2 Document.followers = User1
----->Some other mongodb(via mongoose)operation

What happens if there is a server crash(due to power failure or maybe the remote mongodb server is down ) like this scenario :

----->Update User1 Document.followers = User2
 SERVER CRASHED , FOREVER RESTARTS NODE

What happens to these operations below ? The system is now in inconsistent state and i may have error everytime i ask for User2 followers

----->Update User2 Document.followers = User1
----->Some other mongodb(via mongoose)operation

Also please recommend good logging and restart/monitor modules for apps running in linux.

Right now im using domains for to catch exceptions , doing server.close , but before process.exit() i want to make sure all database transactions are done , can i check this by testing if the event loop is empty or not (how?) and then process.exit(1) ?

Rana Deep
  • 617
  • 7
  • 19

3 Answers3

3

You need transactions for this, and since MongoDB doesn't have them here is a workaround http://docs.mongodb.org/manual/tutorial/perform-two-phase-commits/

vkurchatkin
  • 13,364
  • 2
  • 47
  • 55
  • 1
    What do you say about this TokuMX , its promising this multistatement transactions support . – Rana Deep Oct 20 '13 at 10:25
  • I'm not into MongoDB internals, but as far as I know you can't just implement ACID without any trade-offs (see http://en.wikipedia.org/wiki/CAP_theorem). So if you really need transactions you're better to use DBMS with native transaction support. – vkurchatkin Oct 20 '13 at 10:30
  • ^ This. MongoDB doesn't support full atomic operation on multiple documents yet. Or you try out with some more or less cheap workaround or you go RDBMS. – durum Oct 23 '13 at 15:15
2

One way to address this problem is to add cleanup code to your application that runs whenever the application starts. You write the cleanup code to perform sanity checks on any portions of your data that can be updated in multiple steps like your example and then repairs that data in whatever way make sense for your application.

Depending on the complexity of your application/data, this may also require that you keep a log of actions the app was trying to perform, but that gets complicated real fast. Ideally it's more a matter of refreshing denormalized data and deleting partial data.

You want to do this during startup rather than shutdown as there's no guarantee your shutdown code will fully run and if you're shutting down because of an exception you don't know what the state of your system is at that point.

JohnnyHK
  • 305,182
  • 66
  • 621
  • 471
  • Now i implemented domains and im calling mongoose.connection.close() after server.close(). But i heard the mongoose close() is not called if there are any pending db i/o rather its ignored . And i have put the process.exit(1) in the close callback of mongoose . is that ok ? – Rana Deep Oct 23 '13 at 10:32
  • @RanaDeep Calling `mongoose.connection.close()` or `mongoose.disconnect()` will abort any pending I/O and should prevent the need to even call `process.exit`, but calling that in the `close` callback is also ok. – JohnnyHK Oct 23 '13 at 13:18
  • http://stackoverflow.com/questions/8813838/properly-close-mongooses-connection-once-youre-done In the edit of the question, its said that mongoose.connection.close() is ignored if there is any pending I/O , i dont know if its fixed in the latest version ? – Rana Deep Oct 24 '13 at 12:14
  • @RanaDeep Maybe there was a time it worked that way, but it doesn't any more. I just tested it. – JohnnyHK Oct 24 '13 at 15:23
  • Oh great thanks. But i have problem ,In case of a error i want to call close() only if all the I/O is complete , how do i make sure all mongo related I/O is complete so that i can be almost sure all the transactions are complete and then exit safely . I know nothing is completely 'safe' here – Rana Deep Oct 24 '13 at 16:41
  • @RanaDeep Don't worry about that. You need to handle the case where you're not able to complete any of that anyway. Keep it as simple as possible. – JohnnyHK Oct 24 '13 at 17:07
1

the solution given by vkurchatkin in this link is a workaround in case your appserver crashes, because you will be able of knowing which transactions were pending at that moment. If you implement this in your code you can create cleanup code when your system restart as suggested by JohnnyHK. The code you mention (catching exceptions, test when closing, etc) will not work because...well.. your server crashed! ;-)

That said, this is done using the database, so you will have to warrantee to a certain point that your database does not crash. I would suggest your use replication for that. It is basically a cluster of servers that recovers itself if one node fails, and also you can make some check to make sure that the data reached the servers and is safe.

Hope this helps.

Moreno
  • 526
  • 2
  • 14
  • replication can't guarantee that your database won't crash! in fact, replica sets exist because it's guaranteed that sooner or later your server will crash! :) – Asya Kamsky Oct 27 '13 at 03:07
  • @Asya Kamsky - I get your point here, the server will crash sooner or later. In fact, as you point out, that is why I suggest the replica set. The database server will crash but if you have replication active, the database still works, the recovery will be automatically done (a new primary will be selected if it is primary the one that crashes, and if it is a secondary, well, nothing bad happens). Of course I am talking of replica sets in different servers (otherwise it makes no sense!). I was suggesting to follow vkurchatkin approach for appserver crashes **plus** replica sets. – Moreno Oct 27 '13 at 18:03