Migrating existing infrastructure & scaling with Terraform

Question

We are planning to automate creation & deletion of VMs in our DCs which power our cloud service. The service is such that every new customer gets dedicated VMs (at least 3) - so the number of VMs keep growing. We already have about 2000 VMs running on ESXi. So we now have two problems to solve before adopting terraform -

1. How do we migrate existing VMs to be managed by Terraform (or should we, at all)? Generating resource specification could be scripted but verifying the plan to ensure nothing is affected will be a challenge - given the volume of VMs & the fact that they are all LIVE puts extra pressure on the engineers.

2. As the number of VMs increases, the number of .tf files will keep increasing on the disk. We could club multiple VMs into a single file but that would make deletion of individual VMs, programmatically, a bit tricky. Splitting files into multiple directories is simple workaround I can think of but... Is there a better way to handle scale with terraform?

I couldn't find any blogs which discuss these problems, hence looking for some advice from practical experience here.

Answer 1

Good to see community starting to ask Terraform related questions on Stack Overflow more and more.

Regarding your questions:

1. Migrate existing VMs to be managed by Terraform means updating tfstate file. As of now there is no way to create resource definitions for already created resources and put it into state file automatically (though there are tools like Terraforming, which does it partially and only for AWS resources). So, you will have to describe resources in *.tf files, update tfstate file manually and then verify that tfstate == tf by running terraform plan which should say that there are no changes, which should be applied. Regarding what exactly to put into tfstate file - I would recommend to create resource definition in tf first, then create dummy VM (terraform apply) based on that, find relevant objects in updated tfstate file and update those dummy values with real values of your VMs in that tfstate file (you will also need to update serial to prevent local/remote state inconsistency error).

2. I don't know other smarter way of handling large amount of related resources other than grouping them by directories. In that way you can execute plan/apply just for specific logically separated directories, but you will have to have separated state files. It may easily be overkill (kind-of-warning-so-do-not-try-at-home).

Mostly there are these suggestions I keep in mind when working with Terraform (especially with large amount of resources as you have):

1. Organize your code, so that you have modules in one place and you pass parameters into them in another place. Reusability of code, or how it is called now :)
2. Use -target flag on commands like terraform plan and terraform apply to limit resources you want to touch.

Hope it helps! And more people will enjoy Terraform.