We are moving to two servers from one server due to performance issues. We are moving our MySQL DB to its own dedicated server and will keep the original machine as the front end machine running nginx/apache (Django backend.)
How do we monitor the value of this change? It is possible our whole site could actually get slower since the MySQL queries will be going out over a secure remote connection instead of locally?
Moving the database to it's own machine is very common, and if done right should help your performance considerably.
However, I'm worried about your "secure remote connection". Normally, the database should be in the same data center (and even the same rack, on the same switch) as your web server. I believe the appropriate phrase here is "well-connected". When you say, "secure remote connection", I have images of vpn to somewhere who knows how far away. That's likely to be trouble.
If, as it sounds, these two servers aren't so well connected after all, some things you should do to make sure your site remains responsive is check the average latency between them. A quick and dirty way to do this is just ping from one to other so that it keeps running, and watch the times. Mine are almost all 1ms or <1ms, with the occasional 2ms or 3ms. If you're seeing much larger numbers, you might want to rethink this.
I would run some benchmarks using something like ab (ApacheBench) or siege before and after moving the MySQL server to see what the net effect is.
It is possible that moving MySQL to an external server will actually have worse performance but it depends what sort of queries you're running and exactly how far away the new database server is. If it is within the same datacenter on a 100Mbps/1Gbps line I wouldn't expect a decrease in performance. Chances are that moving MySQL to its own server will help just due to "doubling" the total RAM/CPU you have.
Typically having the DB server away from what is using it is a bad idea. Of course if your queries are complex and returning small amounts of data, it might be okay, but since you are running Django, expect a lot of queries, and therefore a lot of transfer and latency.
A simple way to measure general site performance is with Fiddler. It is a tool that inserts itself as a proxy between your web browser, and the sites you're hitting. I have found it very useful for measuring overall site performance. While this doesn't directly measure the added latency from this DB move, you should be able to get a decent comparison, and it is painless to do.
