1

I am new to terraform and have a few questions. Is there a way in terraform, to use the outputs of a module, in another tf file, WITHOUT having a "main" terraform file, or defining both modules in the same file?

For example, this is a project structure:

project/
├── networking
│   ├── init.tf
│   ├── terraform.tfvars
│   ├── variables.tf
│   └── vpc.tf
├── prd-project-1
│   ├── init.tf
│   ├── instances.tf
│   ├── terraform.tfvars
│   └── variables.tf
├── prd-project-2
│   ├── init.tf
│   ├── instances.tf
│   ├── terraform.tfvars
│   └── variables.tf
├── prd-project-3
│   ├── init.tf
│   ├── instances.tf
│   ├── terraform.tfvars
│   └── variables.tf
├── test
│   ├── init.tf
│   ├── instances.tf
│   ├── terraform.tfvars
│   └── variables.tf

vpc.tf would create a VPC and subnets. instances.tf would create EC2 instances, based on the vpc-id and subnets created previously in vpc.tf. I want to treat/manage these as two seperate actions. ie. First, I would run terraform to create the vpc and subnets, then later I might run terraform to create instances in prod/test for various projects. In this example, I would have only 1 VPC for test and 1 VPC for prod. I would not want a VPC per project (which I am again assuming TF might create if the vpc.tf module is referenced in one of the sub-projects).

Is it best practise to manage terraform code in this way? My concern is if eventually there are many projects needing EC2 instances, I don't want all of these in a single huge main.tf. I am thinking it is safer/cleaner to maintain these all seperately, as long as the vpc outputs can be referenced. I also don't want my terraform plan/applies being slow and checking each and every resource from every project if I am standing up a new project.

The second part of my question is - consider the following code snippet:

module "main-vpc" {
  source     = "../modules/vpc"
  ENV        = "prod"
  AWS_REGION = var.AWS_REGION
}

module "instances" {
  source         = "../modules/instances"
  ENV            = "prod"
  VPC_ID         = module.main-vpc.vpc_id
  PUBLIC_SUBNETS = module.main-vpc.public_subnets
}

Is terraform smart enough to determine that a VPC was already created with an ENV = prod and would not try to recreate it if I called the same "main-vpc" module in another tf file/project?

I've read a few other posts here on stackoverflow, but they all allude that outputs from a module can only be referenced in another module within the same tf file.

Some other posts I've reviewed:

Steven Falzon
  • 67
  • 2
  • 10
  • 1
    Sorry its not clear what you want to do. What's wrong with having main.tf? – Marcin Nov 04 '21 at 06:14
  • If I have, for example, 100 projects. And I want to make a change for project 84, running terraform plan/apply is going to be a long/time consuming task as it needs to go through every project and check for changes. Instead, I would like to contain and deploy seperate projects, but reference outputs of other modules e.g. VPC ID and Subnet IDs. – Steven Falzon Nov 04 '21 at 06:21
  • 1
    I think you have to redesign your approach. You either have the project fully separated or you use terraform workspaces to sperate them. – Marcin Nov 04 '21 at 06:47

2 Answers2

2

If you want to keep your current folder structure, its best to add main.tf to each project:

project/
├── networking
│   ├── init.tf
│   ├── terraform.tfvars
│   ├── variables.tf
│   └── vpc.tf
├── prd-project-1
│   ├── main.tf
│   ├── init.tf
│   ├── instances.tf
│   ├── terraform.tfvars
│   └── variables.tf
├── prd-project-2
│   ├── main.tf
│   ├── init.tf
│   ├── instances.tf
│   ├── terraform.tfvars
│   └── variables.tf
├── prd-project-3
│   ├── main.tf
│   ├── init.tf
│   ├── instances.tf
│   ├── terraform.tfvars
│   └── variables.tf
├── test
│   ├── init.tf
│   ├── instances.tf
│   ├── terraform.tfvars
│   └── variables.tf

Then each main.tf would be something as follow:

module "main-vpc" {
  source     = "../networking"
  ENV        = "prod"
  AWS_REGION = var.AWS_REGION
}

module "instances" {
  source         = "../instances"
  ENV            = "prod"
  VPC_ID         = module.main-vpc.vpc_id
  PUBLIC_SUBNETS = module.main-vpc.public_subnets
}

This way you keep your project separate and each project will have its own TF state file.

Is terraform smart enough to determine that a VPC was already created with an ENV

No its not. You would have to program your own logic for that. A common way is through a variable

variable "vpc_id" {
  default = ""
}

then if you do not explicitly specify the vpc_id for a given project, it would create new VPC. Otherwise, it would use var.vpc_id that you provide as an input parameter. Using data sources you would then get other information for that VPC, such as its subnets.

Having this setup, you would cd to each project folder first, before deploying it, e.g.

cd ./prd-project-1
terraform apply
Marcin
  • 215,873
  • 14
  • 235
  • 294
1

If I understand your question, you want to have independent projects that can still access outputs from one another. So, for example, a project that spins up EC2 instances can refer to the subnets defined in the project that creates a VPC.

There are two ways to do this. First, you can use data sources to query the current deployment environment. For example, the following will look for the VPC with the ENV tag having a value of "prod" (assuming that I haven't made any typos):

data "aws_vpc" "prod" {
  tags = { "ENV": "prod" }
}

Second, you can use remote state (which is a good idea anyway, for security). Each project would write its outputs to a data store (S3 is common, but Terraform offers its own store), and they would be referenced by the others.

And while SO doesn't like links, I can't explain it better than the Terraform docs: https://www.terraform.io/docs/language/state/backends.html

Parsifal
  • 3,928
  • 5
  • 9