Issue with single quotes running Azure CLI command

Question

My script snippet is as below:

End goal to accomplish is to create a Azure DevOps variable group and inject key-values from another variable group into it(the newly created Azure DevOps Variable Group)

set -x
echo "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" | az devops login --organization https://dev.azure.com/tempcompenergydev
az devops configure --defaults organization=https://dev.azure.com/tempcompenergydev project=Discovery
export target_backend=automation
export target_backend="tempcomp.Sales.Configuration.Spa ${target_backend}"
export new_env="abc"

values=(addressSearchBaseUrl addressSearchSubscriptionKey cacheUrl calendarApiUrl checkoutBffApiUrl cpCode)

az_create_options=""

for ptr in "${values[@]}"
do
    result=$(
      az pipelines variable-group list --group-name "${target_backend}" | jq '.[0].variables.'${ptr}'.value'
        
    )
    printf "%s\t%s\t%d\n" "$ptr" "$result" $?

    # add the variable and value to the array
    az_create_options="${az_create_options} ${ptr}=${result}"
done

az pipelines variable-group create \
    --name "test ${new_env}" \
  --variables "${az_create_options}"

However, when the above command executes, I get an unexpected output as below:

+ az pipelines variable-group create --name 'test abc' --variables ' addressSearchBaseUrl="https://qtruat-api.platform.tempcomp.com.au/shared" addressSearchSubscriptionKey="xxxxxxxxxxxxxxxxxxx" cacheUrl="https://tempcompdstqtruat.digital.tempcomp.com.au/app/config" calendarApiUrl="https://qtruat-api.platform.tempcomp.com.au/sales/calendar/v1;rev=deadpool/AvailableDates/EnergyMovers/" checkoutBffApiUrl="https://qtruat-api.platform.tempcomp.com.au/sales/checkout-experience/v1;rev=deadpool/" cpCode="1067076"'
 cpCode "1067076"   0
 WARNING: Command group 'pipelines variable-group' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
 {
   "authorized": false,
   "description": null,
   "id": 1572,
   "name": "test abc",
   "providerData": null,
   "type": "Vsts",
   "variables": {
     "addressSearchBaseUrl": {
       "isSecret": null,
       "value": "\"https://qtruat-api.platform.tempcomp.com.au/shared\" addressSearchSubscriptionKey=\"xxxxxxxxxxxxxxxxxxxxxxxxx\" cacheUrl=\"https://tempcompdstqtruat.digital.tempcomp.com.au/app/config\" calendarApiUrl=\"https://qtruat-api.platform.tempcomp.com.au/sales/calendar/v1;rev=deadpool/AvailableDates/EnergyMovers/\" checkoutBffApiUrl=\"https://qtruat-api.platform.tempcomp.com.au/sales/checkout-experience/v1;rev=deadpool/\" cpCode=\"1067076\""
     }
   }
 }
 ##[section]Finishing: Bash Script

On a side note, If I run manually, I do get correct response. Example below:

az pipelines variable-group create --name "test abc" --variables addressSearchBaseUr="https://qtruat-api.platform.tempcomp.com.au/shared"  addressSearchSubscriptionKey="xxxxxxxxxxxxxxxxxxxxxxxxx"  cacheUrl="https://tempcompdstqtruat.digital.tempcomp.com.au/app/config"  calendarApiUrl="https://qtruat-api.platform.tempcomp.com.au/sales/calendar/v1;rev=deadpool/AvailableDates/EnergyMovers/"  checkoutBffApiUrl="https://qtruat-api.platform.tempcomp.com.au/sales/checkout-experience/v1;rev=deadpool/"  cpCode="1067076"

Output:

 + az pipelines variable-group create --name 'test abc' --variables addressSearchBaseUr=https://qtruat-api.platform.tempcomp.com.au/shared addressSearchSubscriptionKey=xxxxxxxxxxxxxxxxxxx cacheUrl=https://tempcompdstqtruat.digital.tempcomp.com.au/app/config 'calendarApiUrl=https://qtruat-api.platform.tempcomp.com.au/sales/calendar/v1;rev=deadpool/AvailableDates/EnergyMovers/' 'checkoutBffApiUrl=https://qtruat-api.platform.tempcomp.com.au/sales/checkout-experience/v1;rev=deadpool/' cpCode=1067076
 WARNING: Command group 'pipelines variable-group' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
 {
   "authorized": false,
   "description": null,
   "id": 1573,
   "name": "test abc",
   "providerData": null,
   "type": "Vsts",
   "variables": {
     "addressSearchBaseUr": {
       "isSecret": null,
       "value": "https://qtruat-api.platform.tempcomp.com.au/shared"
     },
     "addressSearchSubscriptionKey": {
       "isSecret": null,
       "value": "xxxxxxxxxx"
     },
     "cacheUrl": {
       "isSecret": null,
       "value": "https://tempcompdstqtruat.digital.tempcomp.com.au/app/config"
     },
     "calendarApiUrl": {
       "isSecret": null,
       "value": "https://qtruat-api.platform.tempcomp.com.au/sales/calendar/v1;rev=deadpool/AvailableDates/EnergyMovers/"
     },
     "checkoutBffApiUrl": {
       "isSecret": null,
       "value": "https://qtruat-api.platform.tempcomp.com.au/sales/checkout-experience/v1;rev=deadpool/"
     },
     "cpCode": {
       "isSecret": null,
       "value": "1067076"
     }
   }
 }
 ##[section]Finishing: Bash Script

Answer 1

Work Around

I am not a bash pro, but I found a solution that should work for you. I think the core of the issue is that when you print out your array there, it is treating the set of --variables arguments like they're one long string. So you're getting this...

az pipelines variable-group create --name "test ${new_env}" --variables ' addressSearchBaseUrl="val" addressSearchSubscriptionKey="val" ...'

instead of this...

az pipelines variable-group create --name "test ${new_env}" --variables addressSearchBaseUrl="val" addressSearchSubscriptionKey="val" ...

Bad: I think you could get around this using eval instead of printing out arguments but using eval in almost any situation is advised against.

Good: Instead, I thought this problem could be tackled a different way. Instead of doing a batch where all variables are created at once, this script will copy the desired variables to a new group one at a time:

set -x
echo "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" | az devops login --organization https://dev.azure.com/tempcompenergydev
az devops configure --defaults organization=https://dev.azure.com/tempcompenergydev project=Discovery
export target_backend=automation
export target_backend="tempcomp.Sales.Configuration.Spa ${target_backend}"
export new_env="abc"

values=(addressSearchBaseUrl addressSearchSubscriptionKey cacheUrl calendarApiUrl checkoutBffApiUrl cpCode)

az_create_options=()

for ptr in "${values[@]}"
do
   result=$( az pipelines variable-group list --group-name "${target_backend}" | jq '.[0].variables.'${ptr}'.value' )
   echo "$result"
   # Result from this az command always wraps the response in double quotes. They should be removed.
   stripQuote=$( echo $result | sed 's/^.//;s/.$//' )
   printf "%s\t%s\t%d\n" "$ptr" "$stripQuote" $?
  
   # add the variable and value to the array
   # When adding this way you ensure each key|value pair gets added as a separate element
   az_create_options=("${az_create_options[@]}" "${ptr}|${stripQuote}")
   # ^ Note on this: The | is used because there needs to be a way to separate
   #   the key from the value. What you're really trying to do here is an
   #   Associative Array, but not all versions of bash support that out of the
   #   box. This is a bit of a hack to get the same result. CAUTION if your
   #   variable value contains a | character in it, it will cause problems
   #   during the copy.
done
 
# Create the new group, save the group id (would be best to add a check to see if the group by this name exists or not first)
groupId=$( az pipelines variable-group create  --name "test ${new_env}"  --variables bootstrap="start" | jq '.id' )
 
 
for var in "${az_create_options[@]}"
do
   # Split the key|value pair at the | character and use the set to create the new variable in the new group
   # Some check could also be added here to see if the variable already exists in the new group or not if you want to use this create or update an existing group
   arrVar=(${var//|/ })
   echo "Parsing variable ${arrVar[0]} with val of ${arrVar[1]}"
   az pipelines variable-group variable create \
       --id "${groupId}" \
       --name "${arrVar[0]}" \
       --value "${arrVar[1]}"
done
 
# This is needed cause the az command is dumb and won't let you create a group in the first place without a --variables argument. So I had it create a dummy var and then delete it
az pipelines variable-group variable delete --id "${groupId}" --name "bootstrap" --yes

Notes to call out

1. I would suggest using an Associative Array instead of the array I listed here if possible. However, Associative Arrays are only supported in bash v4 or higher. Use bash --version to see if you'd be able to use them. See this guide as an example of some ways you could work with those if you're able.

2. If using my method, be wary that any | character in a variable value that's being copied will cause the script to fail. You may need to pick a different delimiter to split on.

3. The output of the az pipelines variable-group list command will give the values of the variables wrapped in ". If you try to turn around and throw the exact result you get from the list at a variable create command, you will get a bunch of variables in your group with values like

   {
     "variable": {
       "isSecret": null,
       "value": "\”value\”"
     }
   }
   

   instead of

   {
     "variable": {
       "isSecret": null,
       "value": "value"
     }
   }
   

4. The az command is dumb and won't let you create a new variable group in the first place without a --variables argument. So I had it create a dummy var bootstrap="start" and then delete it.

5. As I mentioned, there may be a better way to print out the array that I'm missing. Comments on my post are encouraged if you know a better way.

Answer 2

This might help....

There is an open issue on how Azure hosted agents authenticate back to ADO using the az-pipelines command.

Feel this is related but if it's not feel free to respond and I'll remove the answer.