4

This is my first time here. I recently joined as a system admin at a company and my recent assignment is to make friendlier dev environments for our devs. Until now our devs connect to our remote box, copy the production code, do a restore of the production database, and correct apache vhost settings and then begin development. Most of the development happens via putty and its extremely tedious.

Quite recently I learnt about Vagrant and was amazed by it. So I quickly set up a simple LAMP stack that our devs can use. However, my biggest complication at this point is how to set up a prod db like mysql environment. Our database is around 7GB in size and it doesnt make sense to download it and then run it in your vagrant VM.

I'm sure that this is common problem lots of sysadmins have dealt with in the past. How do I set up a prod-like dev DB since Vagrant without transferring over that massive data dump.

Prakhar
  • 145
  • 1
  • 7
  • 2
    Have you considered separating the database from the Vagrant LAMP stack? Have a separate database server not included in your Vagrant machine. Change the connection strings in the application to point to the external test database. PArt of your rollout procedure would then be to change connection strings to point to the production data sets. – Wesley Dec 14 '13 at 19:47

4 Answers4

2

Basically have a dev environment. Last time I worked with LARGE databases (and seriously, 7GB is TINY) the dev kit was around 10000gb. We used one of the three servers we had - the one in reserve for real disasters - as a development box, ready to be wiped if ops needed it.

Now I work on smaller stuff (only around 300GB per database) and seriously, we have a central pool of development SQL servers that the developers use.

You need a proper dev and testing environment - and even with a small database like yours that is a little problematic. Wait until you have at least SMALL Databases. 7GB still is tiny.

Wesley
  • 32,690
  • 9
  • 82
  • 117
TomTom
  • 51,649
  • 7
  • 54
  • 136
0

Our developers use a smaller data set than one that's in production. All of the tables are the same, but the data is not a copy of the live data set. This will vary depending on your needs, but for us this is a great way to work.

toppledwagon
  • 4,245
  • 25
  • 15
  • 3
    This assumes you have non-demanding database work. The moment you get into really larger databaes you must test SQL statements against realistic database sizes - or pay the price (bad coding that goes horribly slow in production later). – TomTom Dec 15 '13 at 10:26
0

We have solved this problem. We started out using jetpants, an open source MySQL sharding tool from tumblr. From there, we realized we didn't need the immediately point in time syncing that jetpants provided so we further streamlined with a nightly backup of production stored as a file. We compress that file with lzop then ship it to dev machines over netcat. Time from start to finish for a 20 GB DB? 4 minutes. SSD's help.

dmourati
  • 25,540
  • 2
  • 42
  • 72
0

While your prod DB is 7GB - how big is it without activity data? (Activity data is data that gets added by users or programs - whereas reference data is data that you need to key off of for other things. An example would be an Address record - the street name and address is activity data, since it was added by somebody. But the address type is reference data - since they had to select from either 'Home', 'Work', or 'Other'.)

With just the schema and reference data it shouldn't too much to create a new instance with every environment. For some reason if it is, what is the issue with having developers use a "Dev" database?

tdk2fe
  • 600
  • 2
  • 13
  • Nice and dandy. The result will be crappy application runtimes. You HAVE at least to test against realistic data sizes otherwise little beginner errors (missing index thus table scan) will go to production and make you look stupid. Yes, dev can use smaller sizes, but testing must use real sizes - and even in development a good size is not always unnecessary. – TomTom Dec 15 '13 at 10:25