How to store the variable key and value that has been extracted from json into another variable with same format in azure pipeline?

Question

I have a variable template

var1.yml
variables:
- name: TEST_DB_HOSTNAME
  value: 10.123.56.222
- name: TEST_DB_PORTNUMBER
  value: 1521
- name: TEST_USERNAME
  value: TEST
- name: TEST_PASSWORD
  value: TEST
- name: TEST_SCHEMANAME
  value: SCHEMA  
- name: TEST_ACTIVEMQNAME
  value: 10.123.56.223
- name: TEST_ACTIVEMQPORT
  value: 8161

When I run the below pipeline

resources:
  repositories:
  - repository: templates
    type: git
    name: pipeline_templates
    ref: refs/heads/master


trigger:
- none
variables:
  - template: templates/var1.yml@templates
pool:
  name: PoolA
steps:
- pwsh: |
    Write-Host "${{ convertToJson(variables) }}"

I get the output

{
  build.sourceBranchName: master,
  build.reason: Manual,
  system.pullRequest.isFork: False,
  system.jobParallelismTag: Public,
  system.enableAccessToken: SecretVariable,
  TEST_DB_HOSTNAME: 10.123.56.222,
  TEST_DB_PORTNUMBER: 1521,
  TEST_USERNAME: TEST,
  TEST_PASSWORD: TEST,
  TEST_SCHEMANAME: SCHEMA,
  TEST_ACTIVEMQNAME: 10.123.56.223,
  TEST_ACTIVEMQPORT: 8161
}

How can I modify the pipeline to extract only the key value from the result set that starts with "Test_" and store into another variable in the same format so that I could be used in other tasks in the same pipeline ? OR iterate through the objects that has keys "Test_" and get the value for the same ?

Answer 1

The output you have shown is invalid JSON and cannot be transformed with JSON. Assuming that it were valid JSON:

{
  "build.sourceBranchName": "master",
  "build.reason": "Manual",
  "system.pullRequest.isFork": "False",
  "system.jobParallelismTag": "Public",
  "system.enableAccessToken": "SecretVariable",
  "TEST_DB_HOSTNAME": "10.123.56.222",
  "TEST_DB_PORTNUMBER": 1521,
  "TEST_USERNAME": "TEST",
  "TEST_PASSWORD": "TEST",
  "TEST_SCHEMANAME": "SCHEMA",
  "TEST_ACTIVEMQNAME": "10.123.56.223",
  "TEST_ACTIVEMQPORT": 8161
}

then you can use the to_entries or with_entries filters of jq to get an object containing only those keys which start with "TEST_":

with_entries(select(.key|startswith("TEST_")))

This will give you a new object as output:

{
  "TEST_DB_HOSTNAME": "10.123.56.222",
  "TEST_DB_PORTNUMBER": 1521,
  "TEST_USERNAME": "TEST",
  "TEST_PASSWORD": "TEST",
  "TEST_SCHEMANAME": "SCHEMA",
  "TEST_ACTIVEMQNAME": "10.123.56.223",
  "TEST_ACTIVEMQPORT": 8161
}

Answer 2

The convertToJson() function is a bit messy, as the "json" it creates is not, in fact, a valid json.

There are several possible approaches I can think of:

1. Use convertToJson() to pass the non-valid json to a script-step, convert it to a valid json and then extract the relevant values. I have done this before and it typically works, if you have control over the data in the variables. The downside is that there is risk that the conversion to valid json can fail.
2. Create a yaml-loop that iterates the variables and extract the ones that begins with Test_. You can find examples of how to write a loop here, but basically, it would look like this:

- stage:
  variables:
    firstVar: 1
    secondVar: 2
    Test_thirdVar: 3
    Test_forthVar: 4
  jobs:
  - job: loopVars
    steps:
    - ${{ each var in variables }}:
      - script: |
          echo ${{ var.key }}
          echo ${{ var.value }}
        displayName: handling ${{ var.key }}

3. If applicable to your use case, you can create complex parameters (instead of variables) for only the Test_ variables. Using this, you could use the relevant values directly and would not need to extract a subset from your variable list. Note however, that parameters are inputs to a pipeline and can be adjusted before execution. Example:

parameters:
- name: non-test-variables
  type: object
  default:
    firstVar: 1
    secondVar: 2
- name: test-variables
  type: object
  default:
    Test_thirdVar: 3
    Test_forthVar: 4

You can use these by referencing ${{ parameters.Test_thirdVar }} in the pipeline.