1

I'm using Terraform to create an AWS EMR cluster that uses spot instances as core instances.

I know I can use the bid_price variable within the core_instance_group block on a aws_emr_cluster resource, but I don't want to hardcode prices as I'd have to change them manually every time the instance type changes.

Using the AWS Web UI, I'm able to choose the "Use on-demand as max price" option. That's exactly what I'm trying to reproduce, but in Terraform.

on-demand

Right now I am trying to solve my problem using the aws_pricing_product data source. You can see what I have so far below:

data "aws_pricing_product" "m4_large_price" {
  service_code = "AmazonEC2"

  filters {
   field = "instanceType"
   value = "m4.large"
  }

  filters {
   field = "operatingSystem"
   value = "Linux"
  }

  filters {
   field = "tenancy"
   value = "Shared"
  }

  filters {
   field = "usagetype"
   value = "BoxUsage:m4.large"
  }

  filters {
   field = "preInstalledSw"
   value = "NA"
  }

  filters {
    field = "location"
    value = "US East (N. Virginia)"
  }
}

data.aws_pricing_product.m4_large_price.result returns a json containing the details of a single product (you can check the response of the example here). The actual on-demand price is buried somewhere inside this json, but I don't know how can I get it (image generated with http://jsonviewer.stack.hu/):

price-json

I know I might be able solve this by using an external data source and piping the output of an aws cli call to something like jq, e.g:

aws pricing get-products --filters "Type=TERM_MATCH,Field=sku,Value=8VCNEHQMSCQS4P39" --format-version aws_v1 --service-code AmazonEC2 | jq [........]

But I'd like to know if there is any way to accomplish what I'm trying to do with pure Terraform. Thanks in advance!

cfelipe
  • 327
  • 1
  • 9

2 Answers2

1

Unfortunately the aws_pricing_product data source docs don't expand on how it should be used effectively but the discussion in the pull request that added it adds some insight.

In Terraform 0.12 you should be able to use the jsondecode function to nicely get at what you want with the following given as an example in the linked pull request:

data "aws_pricing_product" "example" {
  service_code = "AmazonRedshift"

  filters = [
    {
      field = "instanceType"
      value = "ds1.xlarge"
    },
    {
      field = "location"
      value = "US East (N. Virginia)"
    },
  ]
}

# Potential Terraform 0.12 syntax - may change during implementation
# Also, not sure about the exact attribute reference architecture myself :)
output "example" {
  values = jsondecode(data.json_query.example.value).terms.OnDemand.*.priceDimensions.*.pricePerUnit.USD
}

If you are stuck on Terraform <0.12 you might struggle to do this natively in Terraform other than the external data source approach you've already suggested.

ydaetskcoR
  • 53,225
  • 8
  • 158
  • 177
  • I'm using Terraform 0.12, so that's OK. I tried creating a local variable with value `"${jsondecode(data.aws_pricing_product.m4_large_price.value).terms.OnDemand.*.priceDimensions.*.pricePerUnit.USD}"`, but I've got the following error: **A splat expression (*) cannot be used inside another attribute-only splat expression**. I tried a few variations of this expression, but no luck. – cfelipe May 30 '19 at 23:09
1

@cfelipe put that ${jsondecode(data.aws_pricing_product.m4_large_price.value).terms.OnDemand.*.priceDimensions.*.pricePerUnit.USD}" in a Locals

  • Hello! I ultimately gave up on managing EMR and went to Databricks. It's a great product for my use case. That said, thanks for your answer. People like you make this SO awesome! =D – cfelipe Oct 04 '19 at 12:27