0

Following the instructions from here, I'm attempting to get to-be-continuous up and running.

I've created the empty to-be-continuous root group and the Maintainer non-individual GitLab account, and generated its appropriately scoped personal access token.

Upon executing the curl command to recursively copy the tbc group, I notice that the tools sub-group isn't cloned.

Seeing that the tracking repo from the tools group is required for the next step, I manually created the tools sub-group and individually manually cloned each of the repos under it, effectively mirroring the structure and content of the authoritative tbc repo.

Additionally I've configured my self-hosted GitLab's CA in the OpenShift GitLab runner so that I no longer get x509 errors.

With the above in place, including an available GitLab runner on my OpenShift cluster, I attempted to manually run the tracking repo's pipeline (as I understand this to be prerequisite to any other pipeline runs?).

The GitLab runner seemed to pick up the pipeline, as runner's log scrolled off the following:

Checking for jobs... received                     [0;m  job[0;m=6103 repo_url[0;m=https://git.corp.odfl.com/to-be-continuous/tools/tracking.git runner[0;m=b3CyGtqD
Checking for jobs... received                     [0;m  job[0;m=6104 repo_url[0;m=https://git.corp.odfl.com/to-be-continuous/tools/tracking.git runner[0;m=b3CyGtqD
[31;1mERROR: Could not create cache adapter             [0;m  [31;1merror[0;m=cache factory not found: factory for cache adapter "" was not registered
[31;1mERROR: Could not create cache adapter             [0;m  [31;1merror[0;m=cache factory not found: factory for cache adapter "" was not registered
[31;1mERROR: Could not create cache adapter             [0;m  [31;1merror[0;m=cache factory not found: factory for cache adapter "" was not registered
[31;1mERROR: Could not create cache adapter             [0;m  [31;1merror[0;m=cache factory not found: factory for cache adapter "" was not registered
Checking for jobs... received                     [0;m  job[0;m=6105 repo_url[0;m=https://git.corp.odfl.com/to-be-continuous/tools/tracking.git runner[0;m=b3CyGtqD
[31;1mERROR: Could not create cache adapter             [0;m  [31;1merror[0;m=cache factory not found: factory for cache adapter "" was not registered
[31;1mERROR: Could not create cache adapter             [0;m  [31;1merror[0;m=cache factory not found: factory for cache adapter "" was not registered
[31;1mERROR: Could not create cache adapter             [0;m  [31;1merror[0;m=cache factory not found: factory for cache adapter "" was not registered
[31;1mERROR: Could not create cache adapter             [0;m  [31;1merror[0;m=cache factory not found: factory for cache adapter "" was not registered
[31;1mERROR: Could not create cache adapter             [0;m  [31;1merror[0;m=cache factory not found: factory for cache adapter "" was not registered
[31;1mERROR: Could not create cache adapter             [0;m  [31;1merror[0;m=cache factory not found: factory for cache adapter "" was not registered
[0;33mWARNING: Job failed: command terminated with exit code 1[0;m  [0;33mduration_s[0;m=9.30956493 [0;33mjob[0;m=6103 [0;33mproject[0;m=876 [0;33mrunner[0;m=b3CyGtqD
[0;33mWARNING: Failed to process runner                 [0;m  [0;33mbuilds[0;m=2 [0;33merror[0;m=command terminated with exit code 1 [0;33mexecutor[0;m=kubernetes [0;33mrunner[0;m=b3CyGtqD
[0;33mWARNING: Job failed: command terminated with exit code 1[0;m  [0;33mduration_s[0;m=9.808499871 [0;33mjob[0;m=6105 [0;33mproject[0;m=876 [0;33mrunner[0;m=b3CyGtqD
[0;33mWARNING: Failed to process runner                 [0;m  [0;33mbuilds[0;m=1 [0;33merror[0;m=command terminated with exit code 1 [0;33mexecutor[0;m=kubernetes [0;33mrunner[0;m=b3CyGtqD
[31;1mERROR: Could not create cache adapter             [0;m  [31;1merror[0;m=cache factory not found: factory for cache adapter "" was not registered
[31;1mERROR: Could not create cache adapter             [0;m  [31;1merror[0;m=cache factory not found: factory for cache adapter "" was not registered
[31;1mERROR: Could not create cache adapter             [0;m  [31;1merror[0;m=cache factory not found: factory for cache adapter "" was not registered
Job succeeded                                     [0;m  duration_s[0;m=30.342517342 job[0;m=6104 project[0;m=876 runner[0;m=b3CyGtqD

At the same time, the pipeline log on GitLab shows the following:

Running with gitlab-runner 14.1.0 (8925d9a0)
  on gitlab-runner-runner-5bc5455cfb-pmrpl b3CyGtqD
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: dle-test
Using Kubernetes executor with image hadolint/hadolint:latest-alpine ...
Using attach strategy to execute scripts...
Preparing environment
00:07
Waiting for pod dle-test/runner-b3cygtqd-project-876-concurrent-0fvm2z to be running, status is Pending
Waiting for pod dle-test/runner-b3cygtqd-project-876-concurrent-0fvm2z to be running, status is Pending
    ContainersNotInitialized: "containers with incomplete status: [init-logs]"
    ContainersNotReady: "containers with unready status: [build helper]"
    ContainersNotReady: "containers with unready status: [build helper]"
Running on runner-b3cygtqd-project-876-concurrent-0fvm2z via gitlab-runner-runner-5bc5455cfb-pmrpl...
Getting source from Git repository
00:01
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/b3CyGtqD/0/to-be-continuous/tools/tracking/.git/
Created fresh repository.
Checking out e31d6d28 as master...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:01
$ # BEGSCRIPT # collapsed multi-line command
/scripts-876-6103/step_script: eval: line 162: can't create /etc/ssl/certs/ca-certificates.crt: Permission denied
Uploading artifacts for failed job
00:00
Uploading artifacts...
WARNING: reports/hadolint-*.json: no matching files 
ERROR: No files to upload                          
Uploading artifacts...
WARNING: reports/hadolint-*.json: no matching files 
ERROR: No files to upload                          
Cleaning up file based variables
00:01
ERROR: Job failed: command terminated with exit code 1

Having spent quite a few hours getting this far, I'm stumped. Any idea what I'm doing wrong?

Added kaniko log as requested:

Running with gitlab-runner 14.1.0 (8925d9a0)
  on gitlab-runner-runner-5bc5455cfb-4ggsp n8KiyZgX
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: dle-test
Using Kubernetes executor with image gcr.io/kaniko-project/executor:debug ...
Using attach strategy to execute scripts...
Preparing environment
00:13
Waiting for pod dle-test/runner-n8kiyzgx-project-876-concurrent-0knvl9 to be running, status is Pending
Waiting for pod dle-test/runner-n8kiyzgx-project-876-concurrent-0knvl9 to be running, status is Pending
    ContainersNotInitialized: "containers with incomplete status: [init-logs]"
    ContainersNotReady: "containers with unready status: [build helper]"
    ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod dle-test/runner-n8kiyzgx-project-876-concurrent-0knvl9 to be running, status is Pending
    ContainersNotReady: "containers with unready status: [build helper]"
    ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod dle-test/runner-n8kiyzgx-project-876-concurrent-0knvl9 to be running, status is Pending
    ContainersNotReady: "containers with unready status: [build helper]"
    ContainersNotReady: "containers with unready status: [build helper]"
Running on runner-n8kiyzgx-project-876-concurrent-0knvl9 via gitlab-runner-runner-5bc5455cfb-4ggsp...
Getting source from Git repository
00:02
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/n8KiyZgX/0/to-be-continuous/tools/tracking/.git/
Created fresh repository.
Checking out e31d6d28 as master...
Skipping Git submodules setup
Restoring cache
00:00
Checking cache for master-docker-2...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache
Downloading artifacts
00:01
Downloading artifacts for docker-hadolint (6121)...
Downloading artifacts from coordinator... ok        id=6121 responseStatus=200 OK token=LRUFpXw7
WARNING: reports/hadolint-dde65eefd6c9a71b70c22f15c806082e.json: lchown reports/hadolint-dde65eefd6c9a71b70c22f15c806082e.json: operation not permitted (suppressing repeats) 
Downloading artifacts for go-build-test (6122)...
Downloading artifacts from coordinator... ok        id=6122 responseStatus=200 OK token=nqXz2-2P
WARNING: bin/: lchown bin/: operation not permitted (suppressing repeats) 
Executing "step_script" stage of the job script
00:08
$ # BEGSCRIPT # collapsed multi-line command
[WARN] =======================================================================================================
[WARN] The template docker:1.2.0 you're using is not up-to-date: consider upgrading to version 2.1.1
[WARN] (set $TEMPLATE_CHECK_UPDATE_DISABLED to disable this message)
[WARN] =======================================================================================================
[INFO] Custom CA certificates configured in /kaniko/ssl/certs/ca-certificates.crt
[INFO] Docker authentication configured for 
$ run_build_kaniko "$DOCKER_SNAPSHOT_IMAGE" --build-arg http_proxy="$http_proxy" --build-arg https_proxy="$https_proxy" --build-arg no_proxy="$no_proxy"
[INFO] Build & deploy image /snapshot:master
[INFO] Kaniko command: /kaniko/executor --context . --dockerfile ./Dockerfile --destination /snapshot:master --cache --cache-dir=/builds/n8KiyZgX/0/to-be-continuous/tools/tracking/.cache --verbosity info  --build-arg CI_PROJECT_URL --build-arg TRACKING_CONFIGURATION --build-arg http_proxy= --build-arg https_proxy= --build-arg no_proxy=
E1013 18:05:11.931688      44 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "/snapshot:master": GET https://index.docker.io/v2/snapshot/blobs/uploads/: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:snapshot Type:repository]]
Uploading artifacts for failed job
00:01
Uploading artifacts...
WARNING: docker.env: no matching files             
ERROR: No files to upload                          
Cleaning up file based variables
00:00
ERROR: Job failed: command terminated with exit code 1
pismy
  • 733
  • 5
  • 12
LiveByTheCode
  • 31
  • 2
  • 5

1 Answers1

0

First of all thanks for your feedback. I thoroughly investigated and you're right: we've recently introduced a bug in our gitlab-sync.sh script that prevented it from recursing :(

A fix is on its way, you should be able to retry it once it's merged.

About your second issue, the logs clearly suggest the hadolint job failed importing your custom CA certificates, but that should not happen using the hadolint/hadolint:latest-alpine image.

See the same job logs on gitlab.com:

[INFO] Custom CA certificates imported in /etc/ssl/certs/ca-certificates.crt

I don't see clearly where the problem could come from.

A few questions to help me investigate:

  • which kind of GitLab runners did you configure ?
  • which technique did you use to configure your custom CA certificates ? did you configure a global DEFAULT_CA_CERTS as suggested in our doc ?
  • is docker-hadolint the only job to fail ? You should also have go-build-test and go-ci-lint on the same stage that also import the custom CA certificates in the same way...
pismy
  • 733
  • 5
  • 12
  • We could find another way to build a tracking image, but I'm afraid you'll have this error with importing your custom CA certificates everywhere, because this is a global pattern implemented in all our jobs & templates... – pismy Oct 09 '21 at 17:17
  • Using a sharable runner deployed in OCP via Operator Hub. Configured a global DEFAULT_CA_CERTS CI variable in GitLab for the cert chain. docker-hadolint and go-ci-lint both fail. go-build-test passes. – LiveByTheCode Oct 11 '21 at 18:21
  • I remember having debugged a very strange behavior on OpenShift: the platform seems to be always executing containers with a non-`root` user, whichever the docker image user. That is a problem with supporting custom CA certificates because the way it is implemented requires running as `root`. But anyway that might be an issue when running container programs with unexpected user... Do you have a way of turning this OCP behavior off ? – pismy Oct 12 '21 at 08:10
  • You could first generate a GitLab server certificate using a valid CA (letsencrypt for e.g.), that will remove the need of declaring `DEFAULT_CA_CERTS` But I'm afraid you will run into other issues later on due to running container programs with non-`root`users... – pismy Oct 12 '21 at 08:17
  • From OpenShift doc: [Enable Images to Run with USER in the Dockerfile](https://docs.openshift.com/enterprise/3.2/admin_guide/manage_scc.html#enable-images-to-run-with-user-in-the-dockerfile) – pismy Oct 12 '21 at 08:29
  • Troubleshooting from GitLab doc: [Root vs non-root](https://docs.gitlab.com/runner/configuration/configuring_runner_openshift.html#root-vs-non-root) – pismy Oct 12 '21 at 08:44
  • Thanks for your time and the suggestions. Unfortunately using a recognized CA like letsencrypt isn't an option and relaxing security as described in the links really isn't an option either. – LiveByTheCode Oct 13 '21 at 13:23
  • I went ahead and tried: "oc adm policy add-scc-to-user anyuid -z gitlab-runner-sa" as noted in the link you provided. This allowed the docker-hadolint, go-ci-lint and go-build-test all to complete successfully. Now it's failing on docker-kaniko-build and I'm getting the message "44 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated" I'm not running this in AWS. This is a local OCP instance. It further says "error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly..." – LiveByTheCode Oct 13 '21 at 18:11
  • Error message continued: error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "/snapshot:master": GET https://index.docker.io/v2/snapshot/blobs/uploads/: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:snapshot Type:repository]] – LiveByTheCode Oct 13 '21 at 18:28
  • Hmmm, strange. The `docker-kaniko-build` is supposed to build the Docker image then push it to your GitLab's registry (`$CI_REGISTRY_IMAGE`), using GitLab provided credentials (`$CI_REGISTRY_USER` / `$CI_REGISTRY_PASSWORD`). Could you please post the complete `docker-kaniko-build` job logs in another answer in the thread ? – pismy Oct 14 '21 at 07:00
  • I've edited the original question to include the addition of the entire kaniko log. – LiveByTheCode Oct 14 '21 at 10:47
  • In [our same logs on gitlab.com](https://gitlab.com/to-be-continuous/tools/tracking/-/jobs/1664675378) you can see the line 46 `[INFO] Docker authentication configured for registry.gitlab.com` while yours is blank. The Docker registry url is supposed to be retrieved with `$CI_REGISTRY_IMAGE` variable (provided by GitLab CI). In your configuration it is unset. Something not configured on your GitLab instance ? Or maybe [check-out that issue](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/31182) – pismy Oct 14 '21 at 14:50
  • Nice catch! This GitLab instance was apparently configured without a container registry. Thanks for all your help. – LiveByTheCode Oct 14 '21 at 17:34