Best development setup for multiple server roles?

Question

At the company I work for we have a pretty complex development environment. Our system runs on several servers, each server fills a different role and has its own code base. There might be more than one actual server for each role (for example we might have 2 messaging servers, 3 storage servers, 1 web server, 1 DB, etc.)

Up until now, the server side developers have been working on SVN, while other developers have been keeping track of their own code by themselves. Everyone has been uploading manually to the correct server via FTP. That means that if someone changed the code for a messaging server for example, he would need to manually upload the changes to 2 or more servers.

Important note: not all servers are on the same LAN.

I've been thinking about changing everyone's work flow and I wish to consult with as to what the best setup would be.

I was thinking to have a single SVN server (basicly open the server side svn to everyone), and have different repositories for each "server role". And have a post-commit for each repository to the server of that type, that would run a "svn update" on a working copy that will be on each server. (I would rather have all the server role's codes on single repository - but how will I handle the post-commit script?)

My question is this: a) Is this the best setup? b) How would you implement the post-commit scripts? How will it know to which server to send updates? c) What is the best way to secure this (with multiple developers)?

Thanks in advance for any replies. Any help and suggestions are greatly appreciated!

Ken.

Answer 1

After reading your question, I think that what you're looking for is more along the lines of a deployment toolset rather than a development setup. Regardless of the direction you choose to go to, I think the first thing you need to do is get all the code under source control. You cannot reasonably have any reliable system for deployment without being able to version every sub-service. Putting all the projects under one SVN server is fine; you'd need a massively large project to outgrow a single-server approach, and it gives you the ability to easily branch/tag a release across all the sub-projects easily.

I'd recommend looking at purpose-built tools for deployment, like Capistrano, Fabric, or ControlTier. Each project has their strengths and weaknesses (Capistrano works well with Rails, ControlTier with Java, etc.) and you'll probably find the best tool depends on the language your sub-projects are written in. Some reasons to prefer a deployment system over a simple post-commit hook:

• It's not hard to write some scripts that run an "svn up" on a given directory on a handful of servers, but deployment systems will handle things like staging vs. production environments, rollbacks of failed deployments, etc.
• Deployment should be a separate step from checking in code. Oftentimes I'll prefer to deploy at a certain time of day (say, mid-morning when engineering support is most likely available, or in the middle of the night when website traffic is near the lowest point), and developers should be free to check in code that doesn't necessarily result in a deployment. If you have continuous integration systems in place, you want your code to be tested/verified before being deployed. Much of this could be accomplished by having the post-commit hook inspect the path of the files committed and only deploy if changes are made to a release branch.
• The deployment process itself should be documented, repeatable, and version-controlled. There are legitimate reasons to want to re-deploy the same release (for example, adding more servers to your setup) that shouldn't require a bogus commit to source control just to kick off another build. Additionally, I like to have the steps necessary to deploy available and visible to the developers. Capistrano is great for this, since you're literally checking in a file to source control that describes how to run a deployment. If you're just using a post-commit hook, make sure it's backed up along with your SVN repository data itself!

Security is largely going to be dictated by business requirements. In smaller shops, I see nothing wrong with giving the developers "keys to the castle" and even rotating the deployment duties among the team. If you work in an environment where developers can't access production data, you'll need to devise a system where they can trigger a deployment (either through a checkin or some other process) but can't access the deployment system's authentication itself. If you're running on a UNIX platform, this can be as simple as generating SSH keys for deployment, and only giving the keys to those responsible for the deployment.