How do I apply a CRD from github to a cluster with terraform?

Question

I want to install a CRD with terraform, I was hoping it would be easy as doing this:

data "http" "crd" {
  url = "https://raw.githubusercontent.com/kubernetes-sigs/application/master/deploy/kube-app-manager-aio.yaml"
  request_headers = {
    Accept = "text/plain"
  }
}

resource "kubernetes_manifest" "install-crd" {
  manifest = data.http.crd.body
}

But I get this error:

can't unmarshal tftypes.String into *map[string]tftypes.Value, expected
map[string]tftypes.Value

Trying to convert it to yaml with yamldecode also doesn't work because yamldecode doesn't support multi-doc yaml files.

I could use exec, but I was already doing that while waiting for the kubernetes_manifest resource to be released. Does kubernetes_manifest only support a single resource or can it be used to create several from a raw text manifest file?

Answer 1

kubernetes_manifest (emphasis mine)

Represents one Kubernetes resource by supplying a manifest attribute

That sounds to me like it does not support multiple resources / a multi doc yaml file.

However you can manually split the incoming document and yamldecode the parts of it:

locals {
  yamls = [for data in split("---", data.http.crd.body): yamldecode(data)]
}

resource "kubernetes_manifest" "install-crd" {
  count = length(local.yamls)
  manifest = local.yamls[count.index]
}

Unfortunately on my machine this then complains about

'status' attribute key is not allowed in manifest configuration

for exactly one of the 11 manifests.

And since I have no clue of kubernetes I have no idea what that means or wether or not it needs fixing.

Alternatively you can always use a null_resource with a script that fetches the yaml document and uses bash tools or python or whatever is installed to convert and split and filter the incoming yaml.

Answer 2

I got this to work using kubectl provider. Eventually kubernetes_manifest should work as well, but it is currently (v2.5.0) still beta and has some bugs. This example only uses kind+name, but for full uniqueness, it should also include the API and the namespace params.

resource "kubectl_manifest" "cdr" {
  # Create a map { "kind--name" => yaml_doc } from the multi-document yaml text.
  # Each element is a separate kubernetes resource.
  # Must use \n---\n to avoid splitting on strings and comments containing "---".
  # YAML allows "---" to be the first and last line of a file, so make sure
  # raw yaml begins and ends with a newline.
  # The "---" can be followed by spaces, so need to remove those too.
  # Skip blocks that are empty or comments-only in case yaml began with a comment before "---".
  for_each = {
    for pair in [
      for yaml in split(
        "\n---\n",
        "\n${replace(data.http.crd.body, "/(?m)^---[[:blank:]]*(#.*)?$/", "---")}\n"
      ) :
      [yamldecode(yaml), yaml]
      if trimspace(replace(yaml, "/(?m)(^[[:blank:]]*(#.*)?$)+/", "")) != ""
    ] : "${pair.0["kind"]}--${pair.0["metadata"]["name"]}" => pair.1
  }
  yaml_body = each.value
}

Once Hashicorp fixes kubernetes_manifest, I would recommend using the same approach. Do not use count+element() because if the ordering of the elements change, Terraform will delete/recreate many resources without needed it.

resource "kubernetes_manifest" "crd" {
  for_each = {
    for value in [
      for yaml in split(
        "\n---\n",
        "\n${replace(data.http.crd.body, "/(?m)^---[[:blank:]]*(#.*)?$/", "---")}\n"
      ) :
      yamldecode(yaml)
      if trimspace(replace(yaml, "/(?m)(^[[:blank:]]*(#.*)?$)+/", "")) != ""
    ] : "${value["kind"]}--${value["metadata"]["name"]}" => value
  }
  manifest = each.value
}

P.S. Please support Terraform feature request for multi-document yamldecode. Will make things far easier than the above regex.

Answer 3

Terraform can split a multi-resource yaml (---) for you (docs):

# fetch a raw multi-resource yaml
data "http" "knative_serving_crds" {
  url = "https://github.com/knative/serving/releases/download/knative-v1.7.1/serving-crds.yaml"
}

# split raw yaml into individual resources
data "kubectl_file_documents" "knative_serving_crds" {
  content = data.http.knative_serving_crds.body
}

# apply each resource from the yaml one by one
resource "kubectl_manifest" "knative_serving_crds" {
  for_each   = data.kubectl_file_documents.knative_serving_crds.manifests
  yaml_body  = each.value
}