What is required for developers to run their own VM server in an enterprise environment

Question

This scenario was also posted on SO, with different questions for different audiences - and I'm very glad I did as I've received some very good responses.

We are attempting to implement a development environment using virtualization for a small team of 4 developers within an enterprise organization. This would allow us to set up separate development, testing, and staging environments - as well as allowing access to new operating systems that are requirements for systems or tools we are evaluating.

We re-purposed an existing workstation-class machine, threw in 24GB RAM and RAID-10, and were doing fine until we attempted to get the machine added to the domain.

Now we are beginning the war that all enterprise developers since the beginning of time have had to fight - the fight for local control of a development and testing environment. The network and IT admins have raised a number of concerns ranging from "ESX Server is the enterprise standard" to "servers are not allowed on client VLANs" to "[fill-in-the-blank] is not a skill set currently possessed in the local or enterprise IT organization".

We could probably justify production-level hardware and formal IT support (read: we could justify the need if we had to, but it would take time and involve a whole lot of headache) - but it would likely take months to formally get IT resources assigned by treating this as a production system - and even if we did, we would likely lose the local control we want.

I imagine that many of you have had similar struggles with developers within your enterprise for developer control of non-production environments, so my questions are as follows:

1. What arguments have your developers made that won you over to allow these types of silos to exist within enterprises which have standard network and security policies in place that would generally (and understandably) preclude this type of non-(centrally)-managed infrastructure?
2. Is this just a matter of the developers making a technical or business justification and ensuring that patch management and AV is going to happen - or more of a political struggle for control and ownership?
3. Given the choice, would you prefer to take ownership and support of the hardware/OS while giving devs local admin rights, or let them manage it entirely, while ensuring that they institute patch management/AV and charging them with responsibility should they cause problems?
4. If you successfully blocked developers from having local control of "rogue servers" on your infrastructure, did the developers just make due or did they (or you) move the development environment to a disconnected VLAN/entirely separate network?

A couple assumptions to limit the scope of this question:

1. To re-iterate, this is for a development environment - no production loads or supportability is required. Nothing externally accessible.
2. This is not a Hyper-V vs. ESX holy war (we would be fine with either - but Hyper-V was selected since it is "free" with MSDN for these purposes [yes, VMWare has free tools too - but the good management tools generally aren't], and would be easier to manage by the local developers in a "Microsoft Shop") - so arguments for or against either are outside the scope of this question.
3. The dev team has already made assurances to either manage patch management and antivirus, or integrate with the existing enterprise systems if IT will support it - but it is certainly within scope whether or not you are willing to accept that.

Answer 1

First of all, I do see a few reasons why your admins are right to push back:

• IT is also responsible for reporting on things like patch management, antivirus software, pci compliance, annual (or more frequent) security audits, etc. It's not just a matter of having your assurance that this is taken care of, but of also being able to prove it to outsiders.

  As an example, I run the network at a small college, and we have a physics lab with a few data collections machines for student experiments. The only thing they do is collect data from a scientific instrument and print out the results (to a directly attached printer) for the student to analyze and hand in to the instructor. They are never on the internet — even AV and Windows updates are applied via the local network. The only reason they are connected to the network and run AV software at all is the explicit purposes of reporting to my monitoring software that they still exist and are current. It's silly, as they would in reality be more secure to remove the network connection, but they were first paid for with an education grant and so those are my reporting requirements.

• Like it or not, your dev server is a production system from the standpoint of the developers. Give it maybe a month, and developers will have a hard time getting work done if it goes down because of processes you will set up that assume the server is available. Avoiding/limiting situations where workers are set idle because of technology failures is a big reason businesses still use centralized IT departments.
• If "ESX Server is the Enterprise Standard", you need to follow that. At this time, there are significant differences between how hyper-v, vmware, xen, and others do things, and a machine built for one can't just be assumed to run well on the other. If you are going to do this, IT is going to need to help manage it at some point, and they don't want to have to convert it over to vmware after there's a bunch of cruft on the machine that might cause a problem.
• Someday this machine will get old, and either need more regular maintenance or setting up a standard replacement cycle. Even new servers sometimes have bum parts. This situation almost always lands in IT's lap only after something has broken that prevents people from doing their jobs. By taking responsibility for the server early, IT can do a much better job making sure you avoid unplanned outages down the road.
• This one's personal, but I can tell you that the very last thing I want around my network is another desktop masquerading as a server. I've dealt with enough of those the last couple years to last me a lifetime.

That said, IT needs to be able to support this initiative. It's not enough for them to just say, "No". Challenge them to come up with an alternative that meets your (very real) needs. The only political situation here should be that their alternative is likely to have a bigger sticker price (because they're planning for costs that you can't see yet), and so the question will be who has to pay for it. IT won't want to because they didn't budget for it, but you'll balk because it's 6 times what you spent for a solution that you were happy with (for the moment).

Also, it sounds like you might be trying to run before you can walk. You want to revamp your development process. As a former developer, I think that's great. But don't just throw out a bunch of VMs and "environments" (ie: dev, stage, qa, etc). Plan out what the new processes will look like — how developers will get work done. Will you use continuous integration? Automated builds? Using what software to support them? Will developers be allowed to move code to production or staging, or will only QA have that ability? Do you need a separate staging? What about two dev branches (one for vNext, one for bugs with vCurrent)?

You might need a server just so the development lead or manager can work all that out, but if so then that needs to be the first step, and the setup and initial process design needs to be done before developers get their hands on it for actual use.

Answer 2

1) What arguments have your developers made that won you over to allow these types of silos to exist within enterprises which have standard network and security policies in place that would generally (and understandably) preclude this type of non-(centrally)-managed infrastructure?

None - mostly because I don't play a management role in my organization so any of these "political things" don't really involve me. The only argument that would really persuade me is to allow something that is explicitly against the network policy and exempted from both the system operations team control and visibility is an air-gap and a CYA letter from my boss's boss.

It's not that I really want to be a the business of saying "No", it's just that there's no good way for this to end well from the operation team's standpoint.

1. We end up with servers managed by people who's primary skill set isn't managing servers on the network who don't have visibility of the entire network and its associated problem space. This is not just a political concern of "protecting" turf; as a trite example imagine what happens when a developer turns on DHCP for some reason.
2. We end up managing the development team's servers for them. This is messy for the opposite reason. The developers are constantly annoyed that we're patching this (breaking something that they know about but we don't), or their fighting for us to enable features that for a variety of good reasons we don't want enabled. This quickly turns into a deadlock where the operations team feels burdened and harassed and the development team feels frustrated and ignored.
3. There are political consequences - because then you have to explain to another department why the developers are "special" and why they get exempted from network policy.

2) Is this just a matter of the developers making a technical or business justification and ensuring that patch management and AV is going to happen - or more of a political struggle for control and ownership?

I don't think the developers need to make a business case - it's pretty clear that developers got to develop and in order to do that that they need a development environment of some kind. As for patch management and AV - that's the operations team job and we will ensure that it is done. It's not that we think developers can't do it. It's just we don't trust you do it right - System Administrators stay System Administrators because they don't trust anything to do anything right so it's nothing personal. Of course there is an obvious political issue of feeling like someone else is "doing your job" but that's not really a technical problem and hence is outside of SF's scope.

3) Given the choice, would you prefer to take ownership and support of the hardware/OS while giving devs local admin rights, or let them manage it entirely, while ensuring that they institute patch management/AV and charging them with responsibility should they cause problems?

Neither for the reasons outlined above.

4) If you successfully blocked developers from having local control of "rogue servers" on your infrastructure, did the developers just make due or did they (or you) move the development environment to a disconnected VLAN/entirely separate network?

Air-gap. The best way to handle this situation is to give the developers their environment (and control of it) and then air-gap or use some other robust method to separate it from the network. This is essentially how we handle public wifi. You want wifi services? Sure. You pay for the network connection, we'll manage the WAPs, but it'll never touch our network. Sorry. Your needs are but one of hundreds. There are other concerns we have to consider.

You don't want to be in the business of saying no, because developers (who are particularly technically clever) will find ways to get what they want anyway. So provide them an environment that suits their needs, make them happy and then find some way to prevent anything they do in development environment from touching the rest of your business network.

TL;DR: I would give you a server with whatever virtualization platform you wanted on either a separate physical network or VLAN. Access to your development environment would be through a single bastion host that the operations team would control and monitor. What you do with it your business - It won't be supported but we'll provide advice and help as time permits for the server administration side of things.

Answer 3

If you came to me with a workstation class machine loaded to the hilt with consumer grade RAM, consumer-grade HDD's, consumer-grade PSU and consumer-grade RAID, I would refuse to put it on the server network too.

There's a lot you need to understand about putting something like that on server VLAN.

1. The server VLAN may very well be a DMZ. In a DMZ you do not put anything that is not hardened and secured. This is just a machine you handed them, they've no idea what you did with it. It also means regular patching and updates, which means being centrally managed. I'm sure as hell not going to log onto each un-managed server and apply patches by hand.

2. The components in that machine are going to fail. I promise. Within 6 or 12, 24 months, it's going to go belly up. Then, where are the backups? Oh, you didn't set them up? But, I thought it was a server? Oh, its a server that someone else provisioned?... and the blame game starts again

3. Who is going to take responsibility when it crashes and shit hits the fan? In most organisations "I gave it to the dev's to look after" is not going to fly.

4. Where are they going to put it? Servers are all rack mounted these days and putting a tower in a rack wastes space and their racks may not be designed for it.

So, the IT department are very justified in not putting this random computer on their server network.

However it is the IT departments job to ensure that YOU can do YOUR job properly. They need to make sure that you have what you need, when you need it. If you have a piece of software that the business needs to keep running, they have to provide a platform for it to run on. That's their job description. But you need to make sure that they have the information they need to do their jobs.

If you had come to me, in my organisation, told me you're starting a new project, I would have given you three VM's: Dev, Live and Staging. You would have full admin rights to Dev, and we would discuss what you needed to do your job for the other two. If you needed full admin rights to them, and could justify it, then you would get it. We have our VM deployment down-pat. VMWare makes this incredibly simple - it only takes about 5 minutes per VM to get it deployed.

It sounds like your IT department suffers from what pretty much every IT department in a large company suffers from. Building little castles and defending them with your life, not letting others in, being bossy, etc. As someone who deals with other people's IT departments every day, I see it all the time. And it's frustrating.

The basic fact is though that the change needs to happen from within the IT department, and it has to be initated from above them. And if you can get IT to realise that they are not a force unto themselves (as that most of them do not generate income for their businesses this can be quite a slap in the face), and that they are there to support the existing staff and enhance the business, then you'll find that your questions become irrelevant, because everyone will be playing happy families.

Answer 4

Why do you want to get it added to the domain? To put it another way, which answers the question better : You can set up a lab to do whatever the hell you want, as long as it's not connected to the corporate LAN. (If you needed internet access, maybe you can get a DMZ-ed VLAN; that shouldn't be a problem, especially if you're only using it to go out, like for downloads.)

That is one, of many many many, different answers to the question.

Answer 5

You're going to get A LOT of answers here, for and against developers having admin access to any part of the environment (probably mostly against), but here's the bottom line:

The sysadmin group is tasked with keeping production systems running, stable, and secure and are responsible for making sure those systems provide the services that the company is paying for (because they are paying for them) at the levels that they expect.

Likewise, the dev team has been tasked to provide services to the company (web, apps, etc.), albeit in a different arena. Fighting for control over the development environment is counterproductive and serves no useful purpose to either side.

I work at a small ISV/ASP. We've got one developer and one sysadmin (me). We have a relationship based on mutual respect and trust. We need to work as a team to help achieve the overarching goals of the company. I give the developer complete, unfettered access to his dev environment, which includes workstations and servers. I manage the dev systems for security, updates, AV, and hardware and the dev does the rest. When his code is ready for production, he hands it off to me, assists me with any configuration needed, and steps back. We provide mutual assistance to each other.

Developers should be the masters of the development environment and Sysadmins should be the masters of the production environment, within reasonable boundaries and with reasonable checks, balances, and controls. When either side needs to "cross over" it should be in cooperation and concert with the "reigning" party under their purview and guidance.

Answer 6

First of all, my experience is strictly in a smaller organization, but this issue does come up in companies of all sizes, so...

1.  What arguments have your developers made that won you over to
allow these types of silos to exist within enterprises which have
standard network and security policies in

From my point of view, the only argument the developers need to make is "we need this." If they came to me first, I'd try to understand their needs and see what we could work out. But, ultimately, if they say "we need this," I'd give them the benefit of the doubt and trust that they know what they're doing.

But that's just the beginning - that's the "Pro" side of the equation. The "Con" is where we get into the wrangling...

2. Is this just a matter of the developers making a technical or
business justification

Except that "just" is an incredible understatement, yeah, if the developers can make a technical and business justification, there wouldn't be a problem. Others here and on programmers.SE (where your SO question got migrated) have pointed out a ton of pitfalls to your setup, so I won't repeat them. If you come up with a plan to deal with all those problems and any others your IT department thinks of, and justify ALL the costs, then it makes sense to go ahead.

3.  Given the choice, would you prefer to take ownership and support
of the hardware/OS while giving devs local admin rights,

This one's a non-starter, you can't have two groups with different goals and responsibilities trying to manage the same systems. It won't just end badly, it'll start badly and end in bloodshed.

(more of 3.) ... or let them manage it entirely, while ensuring that
they institute patch management/AV and charging them with
responsibility should they cause problems?

I think this is covered by my answer to 2.: these are technical details that they'd have to come up with solutions for.

4.  If you successfully blocked developers from having local control
of "rogue servers" on your infrastructure, did the developers just
make due or did they (or you) move the development environment to a
disconnected VLAN/entirely separate network?

I agree with kce: "Air-gap"

If they can justify the additional overhead they're taking on (by becoming admins of their environment) developers can have their own mini-network where they have free rein, BUT it's completely quarantined: nothing touches the rest of the network. So they have to come up with more technical and business justifications, e.g. for "how are we going to backup critical data?"

Again, I have to agree with kce: "System Administrators stay System Administrators because they don't trust anything to do anything right" It's our job to build the most reliable systems we can out of unreliable components, so any proposal that includes half a dozen things that an experienced sysadmin knows are incredibly flaky is gonna get a strong negative reaction.

From the answers and comments here and on programmers.se, I think it's clear there are aspects to this you haven't considered. Although it's going to take longer, I think you really need to go and talk to your IT guys and present things differently: "here's what we need to do, is there any way to integrate this into existing infrastructure and operations?"

Answer 7

The general problem, in your, and millions of similar cases, is :

1) Fuzzy responsibility - there's no direct connection between a corporate worker's actions and his profits. He is payed by month, and not by effects, which are the harder to measure, the bigger the organization is. It applies to security, managers, etc. If they paralize your work, they dont' care.

2) Policymakers and security usually have little or no knowledge about programming. They couldn't understand that they are paralyzing your work even if they would care (which usually doesn't apply).

3) Preferred psychological profile for work in security is paranoid personality or obsesive-compulsive disorder. These people see the conspiracy everywhere. If developers want something, such as new server, they surely want to use it to steal corporate data and publish them in WikiLeaks, or sell to North Korea.