Create Cloud Composer environment with a private repository using Terraform

Question

I'm trying to create a Cloud Composer environment with a PyPI package from a private repository using Terraform. Cloud Composer supports private PyPI repositories. However, configuring a private repository requires an existing composer bucket.

When using Terraform to create an environment, the bucket and environment is created in one go. As far as I can see, the environment creation will fail before there is a chance to write the configuration file to the bucket. Is there any way to create a Cloud Composer environment with a private repository using Terraform?

This is roughly what I'm trying to do:

resource "google_composer_environment" "test" {
  provider = google-beta
  project = var.project_id
  region = var.region
  config {
    software_config {
      image_version = "composer-2.0.0-airflow-2.1.4"
      pypi_packages = {
        mypackage = "*"  # from a private PyPI repo
      }
    ...

I'm kind of confuse on what are you trying to achieve to create the composer environment with a custom repository. As described on this [link](https://cloud.google.com/composer/docs/composer-2/quickstart#create_an_environment) , bucket is created when the environment is created. You want to use a custom bucket?? or you want to use a custom image?? (you can only use the ones in the [official list](https://cloud.google.com/composer/docs/concepts/versioning/composer-versions#images)) — Betjens, Jan 05 '22 at 17:22
Thanks @MattSchuchard , updated the question. However, I'm getting convinced that this is a design issue in Cloud Composer. Even omitting Terraform and just using gcloud API directly, it seems it is not possible to create a Cloud Composer Environment with a private PyPI repository in one go. As far as I can see, it requires first creating the environment and then modifying it after it has been created. — ollik1, Jan 06 '22 at 02:56
Ok so this is multiple actions within a single API invocation and not two consecutive. That is where I was confused. You may be able to raise a feature request on the provider's GH issue tracker to wrap a workaround into the resource if this is not directly possible within the API endpoint parameters. — Matthew Schuchard, Jan 06 '22 at 11:50
@ollik1 ,You can raise feature request by going directly to the [project hub](https://github.com/hashicorp/terraform-provider-google/issues). — Betjens, Jan 12 '22 at 17:13
Thanks all, I have opened an issue here https://github.com/hashicorp/terraform-provider-google/issues/10850 — ollik1, Jan 13 '22 at 02:27

score 2 · Accepted Answer · answered Jan 19 '22 at 12:20

I'm leaving this as community wiki response for community visibility for this kind of questions. For features, I suggest is to go directly to the project site and request the feature there:

Terraform Provider Issues

This will alert the developers of the missing/requested features and it will be taking into consideration. For this case you can track the progress of this case on the issue link you created. Also, you can see the list of upconming features/bugs inside the project goals dashboard but as mention in the description the ETA is up to the team to provide.

Create Cloud Composer environment with a private repository using Terraform

1 Answers1