Gitlab as Terraform state management with HTTP backend not using project credentials?

Question

I've been working to get a monorepo for Terraform up and running. Terraform is executed inside a container. However, when I try to init the backend for http pointing towards my self-hosted Gitlab, it comes back as requiring auth:

Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.
2021-04-15T13:02:00.815Z [TRACE] Meta.Backend: instantiated backend of type *http.Backend
2021-04-15T13:02:00.815Z [DEBUG] checking for provisioner in "."
2021-04-15T13:02:00.815Z [DEBUG] checking for provisioner in "/bin"
2021-04-15T13:02:00.815Z [INFO]  Failed to read plugin lock file .terraform/plugins/linux_amd64/lock.json: open .terraform/plugins/linux_amd64/lock.json: no such file or directory
2021-04-15T13:02:00.815Z [TRACE] Meta.Backend: backend *http.Backend does not support operations, so wrapping it in a local backend
2021/04/15 13:02:00 [DEBUG] GET https://gitlab.my_domain.tld/api/v4/projects/2/terraform/state/infrastructure
Error refreshing state: HTTP remote state endpoint requires auth

I've confirmed that my project token does work by using curl against the project with the token that has API access:

bash-5.1# curl -k -I "https://gitlab.my_domain.tld/api/v4/projects?<my_token_id>"
HTTP/2 200 
server: nginx
date: Wed, 14 Apr 2021 23:59:43 GMT
content-type: application/json
vary: Accept-Encoding
cache-control: no-cache
link: <https://gitlab.my_domain.tld/api/v4/projects?<my_token_id>=&membership=false&order_by=created_at&owned=false&page=1&per_page=20&repository_checksum_failed=false&simple=false&sort=desc&starred=false&statistics=false&wiki_checksum_failed=false&with_custom_attributes=false&with_issues_enabled=false&with_merge_requests_enabled=false>; rel="first", <https://gitlab.my_domain.tld/api/v4/projects?<my_project_token>=&membership=false&order_by=created_at&owned=false&page=1&per_page=20&repository_checksum_failed=false&simple=false&sort=desc&starred=false&statistics=false&wiki_checksum_failed=false&with_custom_attributes=false&with_issues_enabled=false&with_merge_requests_enabled=false>; rel="last"
vary: Origin
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-gitlab-feature-category: projects
x-next-page:
x-page: 1
x-per-page: 20
x-prev-page:
x-request-id: 01F39D73G8ZXPJ1E1ZAK0ZS860
x-runtime: 0.184197
x-total: 1
x-total-pages: 1
strict-transport-security: max-age=31536000
referrer-policy: strict-origin-when-cross-origin

bash-5.1#

The credentials aren't passed via -backend-config nor are they put directly in the backend.tfvars file I have with the backend:

backend.tf:

terraform {
    backend "http" { }
}

backend.tfvars:

lock_method = "POST" 
unlock_method = "DELETE" 
retry_max = "3" 
skip_cert_verification = true

And I've confirmed with an export that the values for TF_HTTP_USERNAME and TF_HTTP_PASSWORD are being set in the environment variables. Hell, I even set them as global variables:

declare -x TERRAFORM_VERSION="0.15.0"
declare -x TF_HTTP_PASSWORD="[MASKED]"
declare -x TF_HTTP_USERNAME="project_2_bot"
declare -x TF_LOG="trace"
declare -x TF_PASSWORD="[MASKED]"
declare -x TF_USERNAME="project_2_bot"
declare -x bot_2_token="[MASKED]"

I checked the Gitlab logs for api_json.json, and I see requests coming in, but with INFO they don't tell me much, just that an attempt came in:

{
  "time": "2021-04-15T13:02:00.924Z",
  "severity": "INFO",
  "duration_s": 0.00915,
  "db_duration_s": 0.00207,
  "view_duration_s": 0.00708,
  "status": 401,
  "method": "GET",
  "path": "/api/v4/projects/2/terraform/state/infrastructure",
  "params": [],
  "host": "gitlab.my_domain.tld",
  "remote_ip": "10.4.6.95, 127.0.0.1",
  "ua": "Go-http-client/1.1",
  "route": "/api/:version/projects/:id/terraform/state/:name",
  "queue_duration_s": 0.011228,
  "db_count": 2,
  "db_write_count": 0,
  "db_cached_count": 1,
  "cpu_s": 0.017418,
  "mem_objects": 10346,
  "mem_bytes": 423664,
  "mem_mallocs": 1771,
  "correlation_id": "01F3ASZH67FMVS0449NC289C47",
  "meta.caller_id": "/api/:version/projects/:id/terraform/state/:name",
  "meta.remote_ip": "10.4.6.95",
  "meta.feature_category": "infrastructure_as_code",
  "meta.client_id": "ip/10.4.6.95"
}

I'm trying to find out how to increase the log level to capture all events coming in for the API, but so far nothing.

Every single time I attempt to run a command against the backend, it comes back as:

Error refreshing state: HTTP remote state endpoint requires auth

I've tried with Terraform version 0.14.x (can't remember what the latest was) and 0.15

And I'm running out of avenues to look. Anyone experience this or give ideas on how to troubleshoot it further?

score 0 · Accepted Answer · answered Apr 16 '21 at 16:21

Okay, I think it's important for any viewers following or finding this in later searches.

So, there's been two problems:

I think there's an issue with Terraform and the self-signed certificate behavior (this might even be the go code underneath). Because even when I tell it to not check the cert, it still reported a very odd error that the cert was using common name, and not SAN values. But that was a symptom of another problem.

My ACTUAL problem, is that I believe the Gitlab documentation is wrong.

So according to the gitlab documentation here says, on step 4:

4. Create a Personal Access Token with the api scope.

Which, instead of a personal access token which takes a license seat count, I created a Project Token.

This token is local to the project, and has a username of

project_$(project_id)_bot#

Which, in my case, is project_2_bot (no number since it's the first one, dumb)

According to docs, it says the token needs API access to hit the API.

I believe the docs are wrong. The token I created had only API access, and it kept failing. Once I gave it more access, (api, read/write to container, registry, repository), it actually went through.

Gitlab as Terraform state management with HTTP backend not using project credentials?

1 Answers1