0

I've got an application. It writes its object graphs to Mongo. For certain collections, we'd like to normalise the data and mirror it onto a relational database (SQL Server or SQLite).

I was thinking the best way to do this would be by a MapReduce job on the collections. Would this be the right way? Is this possible? We're running on Windows if that makes a difference.

Update

What I'm looking for is just general guidance about whether a MapReduce job in JavaScript is the best way to approach this. Let's say I have an object with two fields, FirstName and LastName. When I write this to a collection, I need a row in a relational database (which has two columns FirstName and LastName).

Of course, I could write this to the SQL table in my application at the point where I write to the collection. But I thought that perhaps a better place to do it would be IN the database. This would allow me to shape the data, if needed, before writing it to the relational database.

Now that I've elaborated a bit, I can see a few other questions which I didn't explicitly ask but which are pertinent. Can the JavaScript that runs the MapReduce job even access anything outside of Mongo? Is it sand-boxed? And would doing this as a MapReduce job impact write performance (I'm guessing not, but IANAE, hence the question(s)).

Community
  • 1
  • 1
Steve Dunn
  • 21,044
  • 11
  • 62
  • 87

2 Answers2

2

To reinforce the previous answer, no.

The Map reduce environment cannot directly attach to drivers for other DBs etc, also I am unsure as to whether it should even if it could. This would cause an integration hairball to say the least.

Map reduces will always effect the performance of a system while running so be sure to make sure that the MR runs in a way that is compatible with your hardware, many individuals run large datasets incrementally process a sort of delta update to the result collection.

Also, no, MySQL does not support pinging other external Datasources (last I checked) from triggers.

This would fit much better within a Active Record handler in your App. It would be much more flexible and much easier to integrate across your network.

Sammaye
  • 43,242
  • 7
  • 104
  • 146
1

No, MapReduce job can't write to an SQL database. It can't even write to a mongo collection (other than the one specified in out option).

Similarly, would you also attempt to write to mongodb from a mysql trigger? The idea seems bizarre to me :)

So, I suggest that you do it in the application.

Sergio Tulentsev
  • 226,338
  • 43
  • 373
  • 367
  • Thanks Sergio. You've given me an idea with the `out` hint: perhaps I could just flatten the documents and have a job (outside of Mongo) that matches these flattened documents to associated tables in the relational database. – Steve Dunn Aug 10 '12 at 13:39