Terraform: store the contents of a set of files into a single variable

Question

This is in TF V0.12 if that helps In a directory, I have a variable number of single-line files.
I need to somehow read the contents of them all concatenating them with a newline between and store the result into a single variable. (not an array of strings) I know I can get the list of files to read using:

locals{
  my files=tolist(fileset("${var.file_path}", "**")) }

And if I only had a single file I know I can read in the contents by using

locals {
  file_value=file("${var.file_path}\\${local.my_files}") # if there was only a single file
}

But my brains just turned to cotton wool how I can do the read for multiple, I feel like I should be using a count and a concatenate within a null resource but can't seem to work the logic out. Is that the right path or how should I be doing it?

Why not use null resource local_exec and concatenate files locally first? — mon, Feb 18 '20 at 01:31

score 7 · Answer 1 · answered Feb 18 '20 at 19:10

We can break this problem down into three steps:

Enumerate the filenames matching your desired pattern.
Read the content of each file into memory.
Concatenate all of the file contents together.

We can reformulate the above into three local value expressions, with each one building on the previous one:

locals {
  filenames            = fileset(".", "${var.file_path}/**")
  file_contents        = { for fn in local.filenames : fn => file(fn) }
  file_contents_concat = join("\n", local.file_contents)
}

Alternatively, we can do all of those steps together in a single expression:

locals {
  file_contents_concat = join("\n", [
    for fn in fileset(".", "${var.file_path}/**") : file(fn)
  ])
}

Note that in the fileset call I included var.file_path in the pattern argument rather than in the path argument because that way the resulting file paths will all be relative to the current working directory, and so we can just pass them directly to file without having to re-prepend var.file_path there.

(The separation between path and pattern for fileset is there to help with situations like mirroring a bunch of files into S3, where it's helpful to have the resulting paths be relative to the bucket root rather than the current working directory, but that sort of mapping isn't important here because the filenames don't appear in the result at all.)

trying a test this way I get Error: Error in function call on main.tf line 7, in locals: 7: filenames = fileset(".", "${var.file_path}/*")
|---------------- | var.file_path is "C:/path/test/" Call to function "fileset" failed: failed to trim path of match (): Rel: can't make C:\path\test\test1.test.txt relative to .. — Marcus Adams, Feb 18 '20 at 20:41
apologies for poor markup i can't seem to get it formatting for me today — Marcus Adams, Feb 18 '20 at 20:42
Hmm... does your `var.file_path` include `..`? It looks like that's causing it to fail because the contract for `fileset` is to produce paths under the path given in the first argument, but `..` is not within `.`. If you do need to work in a parent dir then you will probably need to write it the way you originally had it, with `var.file_path` in the `path` argument and then re-prepend it in the `file` call. (Note that Terraform uses forward slashes for paths even on Windows, so that configurations can be portable between platforms.) — Martin Atkins, Feb 19 '20 at 00:20
I get an error: file_contents_concat = join("\n", local.file_contents) |---------------- | local.file_contents is object with 2 attributes — red888, Mar 01 '20 at 00:35

score 1 · Answer 2 · answered Feb 18 '20 at 02:58

1

Concatenate locally with null_resource.

resource "null_resource" "concatenate_my_files" {
  provisioner "local-exec" {
    working_dir = "${var.file_path}"
    command = <<EOF
      cat "${local.my_files}" > "${var.concatenated_file_name}"
EOF
  }
}

answered Feb 18 '20 at 02:58

mon

18,789
22
112
205

score 1 · Answer 3 · answered Feb 18 '20 at 10:48

Thank you @mon that reminded me I should always KISS

Removing the complexity I ended up

resource "null_resource" "concatenate_my_files" {
  provisioner "local-exec" {
    working_dir = var.file_path
    command = <<EOF
cat * > my_temp_file
EOF
  }
}

I can then read this in as previous.

Terraform: store the contents of a set of files into a single variable

3 Answers3