1

Is there a way I can use a terraform data call for a bucket (perhaps created and stored in a different state file) and then in the event nothing is in data, create the resource by setting a count?

I've been doing some experiments and continually get the following:

Error: Failed getting S3 bucket (example_random_bucket_name): NotFound: Not Found
        status code: 404, request id: <ID here>, host id: <host ID here>

Sample code to test (this has been modified from the original code which generated this error):

variable "bucket_name" {
  default = "example_random_bucket_name"
}

data "aws_s3_bucket" "new" {
  bucket = var.bucket_name
}

resource "aws_s3_bucket" "s3_bucket" {
  count = try(1, data.aws_s3_bucket.new.id == "" ? 1 : 0 )
  bucket = var.bucket_name
}

I feel like rather than generating an error I should get an empty result, but that's not the case.

Marcin
  • 215,873
  • 14
  • 235
  • 294
keeer
  • 783
  • 1
  • 5
  • 11
  • I've rewritten the module which in turn caused problems. This was just an example of what I was trying to achieve. In the end I made it so once namespace would create the bucket and all other namespaces would apply the policy to it. I had problems where it would check if the bucket existed and if it did the count would change, which in turn would destroy the bucket (I essentially had 2 modules at one point; on for apply policy and one for create bucket... very messy). – keeer Dec 16 '21 at 16:13

3 Answers3

2

Sadly you can't do this. data sources must exist, otherwise they error out. There is no build in way in TF to check if a resource exists or not. There is nothing in between, in a sense that a resource may, or may not exist.

If you require such functionality, you have to program it yourself using External Data Source. Or maybe simpler, provide an input variable bucket_exist, so that you explicitly set it during apply.

Marcin
  • 215,873
  • 14
  • 235
  • 294
  • This is a variation of what I ended up doing... For the curious: `data "external" "check_bucket_exists" { program = ["bash", "-c", "aws s3 ls ${local.example_bucket} > /dev/null 2>&1 && echo -n '{\"result\":\"yes\"}' || echo -n '{\"result\":\"no\"}' "] }` – keeer Dec 16 '21 at 16:14
2

Terraform is a desired-state system, so you can only describe what result you want, not the steps/conditions to get there.

If Terraform did allow you to decide whether to declare a bucket based on whether there is already a bucket of that name, you would create a configuration that could never converge: on the first run, it would not exist and so your configuration would declare it. But on the second run, the bucket would then exist and therefore your configuration would not declare it anymore, and so Terraform would plan to destroy it. On the third run, it would propose to create it again, and so on.

Instead, you must decide as part of your system design which Terraform configuration (or other system) is responsible for managing each object:

  • If you decide that a particular Terraform configuration is responsible for managing this S3 bucket then you can declare it with an unconditional aws_s3_bucket resource.
  • If you decide that some other system ought to manage the bucket then you'll write your configuration to somehow learn about the bucket name from elsewhere, such as by an input variable or using the aws_s3_bucket data source.
Martin Atkins
  • 62,420
  • 8
  • 120
  • 138
  • The only problem that arises are for resources that AWS reserves, that you then have to edit later such as the elastic beanstalk bucket or the glue bucket, which by default is a reserved name, and created upon first creation of a resource. They're made unencrypted and unversioned. So I want terraform to encrypt and version them. But I don't want it to fail if AWS has not made it yet. – DFTR Aug 24 '23 at 21:41
  • In that situation I think you must just accept that it will fail if AWS hasn't made it yet, and ensure that AWS has made it before you run Terraform. AWS making something itself is, from Terraform's perspective, exactly the same as something being made by a separate Terraform configuration: it's an external dependency for your current configuration that must be dealt with before applying that configuration. This belongs to the "if you decide that some other system ought to manage..." cateogry, except that in this case the decision is made for you by the designers of AWS. – Martin Atkins Aug 25 '23 at 00:21
1

Data sources are designed to fail this way.

However, if you use a state file from external configuration, it's possible to declare an output in the external state, based on whether the s3 bucket is managed by that state and use it in s3_bucket resource as condition.

For example, the output in external state will be empty string (not managed) or value for whatever property is useful for you. Boolean is another choice. Delete data source from this configuration and add condition to the resource based on the output.

It's your call if any such workarounds complicate or simplify your configuration.

Denis V
  • 78
  • 5