How to export a scoped variable for other (non-TBC) jobs?

Question

Thank you for your work. I am Working on Terraform using your template, and need to set my CUSTOM_ENVIRONMENT_NAME variable differently depending on the environment, in your included jobs but also in others' and mine.

There are different logics to differentiate the environment on my pipeline:

your logic that is based on the branch BUT both production and staging share the master branch
an Orange job I include that is based on the gitlab project (one project = one environment)
my own jobs for which I am yet to precise the logic

So far I found different possible mechanisms:

using $CI_ENVIRONMENT_NAME -- but I need to set the environment: name: foo on each job and I am including many different ones that I don't really want to overwrite
using $CI_COMMIT_REF_NAME -- but in your template the production and staging share the same branch
exporting an artifact in a prerequired job, cf. https://docs.gitlab.com/ee/ci/variables/#pass-an-environment-variable-to-another-job
duplicating jobs with the extends keyword and setting CI_ENVIRONMENT_NAME or CUSTOM_ENVIRONMENT_NAME manually
using your scoped variables syntax -- but it only applies in your jobs

I would like to use a combination of two: using your scoped variables and exporting them to other jobs using e.g. an artifact

Is there a way for me to export the scoped variable (cf. https://docs.gitlab.com/ee/ci/variables/#pass-an-environment-variable-to-another-job) from the tf-tflint job (which is running at the very beginning of my pipeline)?

Here is a sample gitlab-ci file:

include:
  # terraform
  - project: 'to-be-continuous/terraform' # template principal de TBC
    ref: '2.3.0'
    file: '/templates/gitlab-ci-terraform.yml'
  - project: 'to-be-continuous/custom/devops-store' # template additionnel pour utiliser les runners RSC plutôt que DIOD/GIN
    ref: '1.1.2'
    file: '/templates/gitlab-ci-terraform-dos.yml'

  # PDN
  - project: "dixsiptal/newdelivery/templates/common"
    ref: "1.0.30"
    file: "templates/gitlab-ci-ansible.yml"
  - project: "dixsiptal/newdelivery/templates/common-token"
    ref: "1.0.1"
    file: "settoken.yml"
  - project: "dixsiptal/newdelivery/templates/pli/pdn"
    ref: "2.1.2"
    file: "templates/gitlab-ci.yml"

  # PLI docker
  # NB : common et common-token sont également des pré-requis au PDN
  - project: "dixsiptal/newdelivery/templates/pli/docker"
    ref: "3.1.0"
    file: "templates/gitlab-ci.yml"


variables:
  # les lignes scoped suivantes permettent de définir ENV_NOMMAGE_IPANEMA différemment selon l'environnement, grâce au template TBC
  # ENV_NOMMAGE_IPANEMA sera alors utilisable depuis tous les jobs
  # cf. https://to-be-continuous.gitlab.io/doc/usage/#scoped-variables
  scoped__ENV_NOMMAGE_IPANEMA__if__CI_ENVIRONMENT_NAME__equals__production: production
  scoped__ENV_NOMMAGE_IPANEMA__if__CI_ENVIRONMENT_NAME__equals__staging: maintenance
  scoped__ENV_NOMMAGE_IPANEMA__if__CI_ENVIRONMENT_NAME__equals__integration: e2e
  scoped__ENV_NOMMAGE_IPANEMA__if__CI_ENVIRONMENT_NAME__startswith__review: review

  # For the docker and the PDN template:
  INVENTORY: "./inventories/${ENV_NOMMAGE_IPANEMA}/terraform_inventory.ini"
  # or, if it is not possible to interpolate ENV_NOMMAGE_IPANEMA at this stage:
  # scoped__INVENTORY__if__CI_ENVIRONMENT_NAME__equals__production: "inventories/production/terraform_inventory.ini"
  # scoped__INVENTORY__if__CI_ENVIRONMENT_NAME__equals__staging: "inventories/maintenance/terraform_inventory.ini"
  # scoped__INVENTORY__if__CI_ENVIRONMENT_NAME__equals__integration: "inventories/e2e/terraform_inventory.ini"
  # scoped__INVENTORY__if__CI_ENVIRONMENT_NAME__startswith__review: "inventories/review/terraform_inventory.ini"

  # for the PDN template:
  PDN_PROJET: "${CI_SERVER_HOST}/wanctr/sandbox/iac/pdn_inventories/pdn_inventory_${ENV_NOMMAGE_IPANEMA}.git" # doit être différencié selon l'environnement. Imposé par le template PDN
  # same thing would apply here
  # scoped__PDN_PROJET__if__CI_ENVIRONMENT_NAME__equals__integration: "${CI_SERVER_HOST}/wanctr/sandbox/iac/pdn_inventories/pdn_inventory_e2e.git"
  # scoped__PDN_PROJET__if__CI_ENVIRONMENT_NAME__startswith__review: "${CI_SERVER_HOST}/wanctr/sandbox/iac/pdn_inventories/pdn_inventory_review.git"
  # etc.

  # terraform
  TF_PROJECT_DIR: "./terraform"
  TF_OUTPUT_DIR: "../ansible/inventories" # relatif à TF_PROJECT_DIR
  TF_BINARY_VERSION: "1.0.10"
  TF_BRMC_PROVIDER_VERSION: "2.8.1_2.7.0" # peut avoir la forme v1_v2 par exemple 2.8.1_2.7.0 pour avoir 2 versions de providers simultanément utile pour upgrade) (cf. la liste des versions disponibles sur https://gitlab.tech.orange/wanctr/sandbox/iac/terrabrmc/container_registry/18687)
  TF_IMAGE: "registry.${CI_SERVER_HOST}/wanctr/sandbox/iac/terrabrmc:${TF_BINARY_VERSION}-${TF_BRMC_PROVIDER_VERSION}"
  #TF_EXTRA_OPTS: ""
  TF_INIT_OPTS: "-upgrade" # pour upgrade la version du provider le cas échéant

  TF_REVIEW_ENABLED: "true" # permet l'utilisation des envs de test, créées automatiquement aux push sur des branches autres que "master" et "develop"
  TF_REVIEW_EXTRA_OPTS: "-var-file=values-review.tfvars"

  TF_INTEG_ENABLED: "true" # e2e/qualif
  TF_INTEG_EXTRA_OPTS: "-var-file=values-e2e.tfvars"

  TF_STAGING_ENABLED: "true" # maintenance
  TF_STAGING_EXTRA_OPTS: "-var-file=values-maintenance.tfvars"

  TF_PROD_ENABLED: "true" # production
  TF_PROD_EXTRA_OPTS: "-var-file=values-production.tfvars"
  
  BRMC_HOST: "https://brmc.si.fr.intraorange"
  BRMC_TENANT: "vsphere.local"
  BRMC_USERNAME: "$BRMC_API_ACCOUNT_USERNAME"
  BRMC_PASSWORD: "$BRMC_API_ACCOUNT_PASSWORD"

  # ansible utilisé par templates communs DESI https://gitlab.tech.orange/dixsiptal/newdelivery/templates/common/-/blob/1.0.30/README.md
  ANSIBLE_PATH: "./ansible" # chemin vers les ressources ansible
  ANSIBLE_USER: "ansible"
  ODE_ANSIBLE_VERSION: "2.9" # ou 2.10 ?

  # docker https://gitlab.tech.orange/dixsiptal/newdelivery/templates/pli/docker/-/blob/3.1.0/README.md
  # INVENTORY is used here
  .ansible:
    extends: .ansible-runner-ode # indique au template gitlab-ci du PLI docker d'utiliser ODE pour le déploiement
  DOCKER_INVENTORY_SECTION: "cluster"
  DOCKER_REDHAT_VG: "docker_vg"
  # etc.

  # PLI PostgreSQL
  # ...

  # ODE CLI
  ODE_CLI_VERSION: "2.1.8" # ou "2.3.0" ?

  # custom-traefik (one of my own)
  ANSIBLE_REQUIREMENTS: "${ANSIBLE_PATH}/requirements.yml"
  # ...
  CONFIG_APPLICATIVE_MAIN_INVENTORY_DIR: "config_applicative/inventories/${ENV_NOMMAGE_IPANEMA}"
  # ...


pre-test-scoped: # testing the dotenv artifact variable setting
  stage: test
  rules:
  - when: always
  script:
    - echo "ENV_NOMMAGE_IPANEMA_envfile=value_from_other_job" >> justtesting.env
  artifacts:
    reports:
      dotenv: justtesting.env

# echoing preset variables for testing
test-scoped:
  stage: test
  image: dockerfactory.tech.orange/ode:$ODE_CLI_VERSION
  variables:
    ENV_NOMMAGE_IPANEMA_rules_by_commitrefname: "review" # valeur par défaut
  environment: # Ideally I would like not to indicate the env name each time for concision and simplicity when importing templates
    name: testscoped
  needs: ["pre-test-scoped"]
  rules:
  - when: on_success
  - if: $CI_COMMIT_REF_NAME == "master"
    variables:
      ENV_NOMMAGE_IPANEMA_rules_by_commitrefname: "production" # et maintenance ?..
  - if: $CI_COMMIT_REF_NAME == "develop"
    variables:
      ENV_NOMMAGE_IPANEMA_rules_by_commitrefname: "E2E"
  - if: $CI_COMMIT_REF_NAME == "scoped-variables"
    variables:
      ENV_NOMMAGE_IPANEMA_rules_by_commitrefname: "scoped-variables-review-test"

  - if: $CI_ENVIRONMENT_NAME == "production"
    variables:
      ENV_NOMMAGE_IPANEMA_rules_by_cienvname: "production" # et maintenance ?..
  - if: $CI_ENVIRONMENT_NAME == "staging"
    variables:
      ENV_NOMMAGE_IPANEMA_rules_by_cienvname: "maintenance"
  - if: $CI_ENVIRONMENT_NAME == "integration"
    variables:
      ENV_NOMMAGE_IPANEMA_rules_by_cienvname: "E2E" # et maintenance ?..
  - if: $CI_ENVIRONMENT_NAME == "review"
    variables:
      ENV_NOMMAGE_IPANEMA_rules_by_cienvname: "review"

  script:
    - echo $CI_COMMIT_REF_NAME
    - echo $CI_ENVIRONMENT_NAME
    - echo $ENV_NOMMAGE_IPANEMA_tfscoped
    - echo $ENV_NOMMAGE_IPANEMA_rules_by_commitrefname
    - echo $ENV_NOMMAGE_IPANEMA_rules_by_cienvname
    - echo $ENV_NOMMAGE_IPANEMA_projectenvset # set in the gitlab project with a scope
    - echo $ENV_NOMMAGE_IPANEMA_envfile


custom-traefik: # job supplémentaire
  image: dockerfactory.tech.orange/ode:$ODE_CLI_VERSION
  when: manual
  stage: "Gestion"

  script:
  # ...
  # this job uses scoped varibles

export-inventory-file:
  image: dockerfactory.tech.orange/ode:$ODE_CLI_VERSION
  when: manual
  stage: "Gestion"

  script:
  # ...
  # this job uses scoped varibles

If I'm understanding you right, you want to pass the dotenv artifact to pass variables, but only have those variables applied to certain jobs in subsequent stages. Is that understanding correct? It may also be useful to provide a minimal CI configuration file and describe what variables you want present in which jobs. — sytech, Mar 29 '22 at 16:25
I want these variables to be valued in all jobs. Their values depend on the environment type. The first one is $ENV_NOMMAGE_IPANEMA and is equivalent to gitlab-ci's $CI_ENVIRONMENT_NAME (or environment_type in the terraform code) but with custom names. Others are concatenated strings which interpolate $ENV_NOMMAGE_IPANEMA. My gitlab-ci being quite complex and in french, I would need some editing to share it... — Dylan, Mar 30 '22 at 07:51
Do you try to use cache? I mean you could add prepare stage and generate .env file. Also you could use workflow sections with var declarations. — Vers_us, Mar 30 '22 at 12:18
What are "workflow sections with var declarations"? I am not sure if you got my point correctly. I am trying to export the scoped variable from the TBC tf-tflint job to make it available to all other jobs — Dylan, Mar 30 '22 at 12:51
@sytech "to be continuous" is a set of open-source, advanced and composable GitLab CI/CD templates. See: https://to-be-continuous.gitlab.io/doc/ Scoped variables is a TBC specific feature to limit/override environment variables depending on the job context. See: https://to-be-continuous.gitlab.io/doc/usage/#scoped-variables — pismy, Mar 31 '22 at 07:10

pismy · Answer 1 · 2022-03-31T07:12:49.197

You can of course mimic to-be-continuous scoped variables feature with the following piece of code in your .gitlab-ci.yml:

.base-scripts: &base-scripts |
  set -e

  function log_info() {
      echo -e "[\\e[1;94mINFO\\e[0m] $*"
  }

  function log_warn() {
      echo -e "[\\e[1;93mWARN\\e[0m] $*"
  }

  function log_error() {
      echo -e "[\\e[1;91mERROR\\e[0m] $*"
  }

  function unscope_variables() {
    _scoped_vars=$(env | awk -F '=' "/^scoped__[a-zA-Z0-9_]+=/ {print \$1}" | sort)
    if [[ -z "$_scoped_vars" ]]; then return; fi
    log_info "Processing scoped variables..."
    for _scoped_var in $_scoped_vars
    do
      _fields=${_scoped_var//__/:}
      _condition=$(echo "$_fields" | cut -d: -f3)
      case "$_condition" in
      if) _not="";;
      ifnot) _not=1;;
      *)
        log_warn "... unrecognized condition \\e[1;91m$_condition\\e[0m in \\e[33;1m${_scoped_var}\\e[0m"
        continue
      ;;
      esac
      _target_var=$(echo "$_fields" | cut -d: -f2)
      _cond_var=$(echo "$_fields" | cut -d: -f4)
      _cond_val=$(eval echo "\$${_cond_var}")
      _test_op=$(echo "$_fields" | cut -d: -f5)
      case "$_test_op" in
      defined)
        if [[ -z "$_not" ]] && [[ -z "$_cond_val" ]]; then continue; 
        elif [[ "$_not" ]] && [[ "$_cond_val" ]]; then continue; 
        fi
        ;;
      equals|startswith|endswith|contains|in|equals_ic|startswith_ic|endswith_ic|contains_ic|in_ic)
        # comparison operator
        # sluggify actual value
        _cond_val=$(echo "$_cond_val" | tr '[:punct:]' '_')
        # retrieve comparison value
        _cmp_val_prefix="scoped__${_target_var}__${_condition}__${_cond_var}__${_test_op}__"
        _cmp_val=${_scoped_var#"$_cmp_val_prefix"}
        # manage 'ignore case'
        if [[ "$_test_op" == *_ic ]]
        then
          # lowercase everything
          _cond_val=$(echo "$_cond_val" | tr '[:upper:]' '[:lower:]')
          _cmp_val=$(echo "$_cmp_val" | tr '[:upper:]' '[:lower:]')
        fi
        case "$_test_op" in
        equals*)
          if [[ -z "$_not" ]] && [[ "$_cond_val" != "$_cmp_val" ]]; then continue; 
          elif [[ "$_not" ]] && [[ "$_cond_val" == "$_cmp_val" ]]; then continue; 
          fi
          ;;
        startswith*)
          if [[ -z "$_not" ]] && [[ "$_cond_val" != "$_cmp_val"* ]]; then continue; 
          elif [[ "$_not" ]] && [[ "$_cond_val" == "$_cmp_val"* ]]; then continue; 
          fi
          ;;
        endswith*)
          if [[ -z "$_not" ]] && [[ "$_cond_val" != *"$_cmp_val" ]]; then continue; 
          elif [[ "$_not" ]] && [[ "$_cond_val" == *"$_cmp_val" ]]; then continue; 
          fi
          ;;
        contains*)
          if [[ -z "$_not" ]] && [[ "$_cond_val" != *"$_cmp_val"* ]]; then continue; 
          elif [[ "$_not" ]] && [[ "$_cond_val" == *"$_cmp_val"* ]]; then continue; 
          fi
          ;;
        in*)
          if [[ -z "$_not" ]] && [[ "__${_cmp_val}__" != *"__${_cond_val}__"* ]]; then continue; 
          elif [[ "$_not" ]] && [[ "__${_cmp_val}__" == *"__${_cond_val}__"* ]]; then continue; 
          fi
          ;;
        esac
        ;;
      *)
        log_warn "... unrecognized test operator \\e[1;91m${_test_op}\\e[0m in \\e[33;1m${_scoped_var}\\e[0m"
        continue
        ;;
      esac
      # matches
      _val=$(eval echo "\$${_target_var}")
      log_info "... apply \\e[32m${_target_var}\\e[0m from \\e[32m\$${_scoped_var}\\e[0m${_val:+ (\\e[33;1moverwrite\\e[0m)}"
      _val=$(eval echo "\$${_scoped_var}")
      export "${_target_var}"="${_val}"
    done
    log_info "... done"
  }


# Generic base job
.base-job:
  before_script:
    - *base-scripts
    - unscope_variables

variables:
  SOME_VAR: "this is the default value"
  scoped__SOME_VAR__if__CI_JOB_NAME__equals__job1: "this is value for job#1"
  scoped__SOME_VAR__if__CI_JOB_NAME__equals__job2: "this is value for job#2"

# Then have all your jobs inherit from .base-job
job1:
  extends: .base-job
  stage: stage1
  script:
    - echo "$SOME_VAR"

job2:
  extends: .base-job
  stage: stage2
  script:
    - echo "$SOME_VAR"

All the magic is implemented in the unscope_variables() function.

/!\ Now some warnings:

late evaluation of $INVENTORY and $PDN_PROJET (both using scoped variable $ENV_NOMMAGE_IPANEMA in their value) won't work
using $CI_ENVIRONMENT_NAME as the test condition in your scoped variables will only work in jobs with a defined environment (this value is unset elsewhere). You'd probably better use the $environment_type variable, set by the Terraform deploy jobs and propagated as a dotenv artifact to the downstream pipeline

Thanks, but I am not satisfied with this option. 1st it forces me to add an "extends" tag to all jobs. 2nd it adds a big chunk of "obscure" (for people not comfortable with shell) code. I was hoping I could simply use the values of TBC's scoped variables in other jobs. I think I'll try the $CI_ENVIRONMENT_NAME with environment: name: xxx in each job. — Dylan, Mar 31 '22 at 09:37

How to export a scoped variable for other (non-TBC) jobs?

1 Answers1