Azure Service Connection not working across multiple subscriptions

Question

I have recently tried to set up a service connection to a second subscription, I have it working on the first subscription fine and has been working for seven months and still is working. But when I give the Service Principal Application access to the new subscription the pipeline fails and says it can not see the Resource Group. This is on the Terraform Initialise step.

I have given contributor access to the Service Principal on the Subscription Level and also for good measure have given contributor access to the Service Principal on the actual resource group its self.

This is the error that I am getting:

Initializing the backend...

Successfully configured the backend "azurerm"! Terraform will automatically
use this backend unless the backend configuration changes.
╷
│ Error: Failed to get existing workspaces: Error retrieving keys for Storage Account "nsterraformstatestorage": storage.AccountsClient#ListKeys: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceGroupNotFound" Message="Resource group 'TerraformBackendForCICTesting' could not be found."
│ 
│ 
╵

##[error]Error: The process '/opt/hostedtoolcache/terraform/1.0.4/x64/terraform' failed with exit code 1

I have followed this post in the past to setup the service connection: https://sabirmohamed.com/how-to-create-a-service-connection-in-the-azure-devops/

I do believe both subscriptions are in the same tenant.

I must be missing something as I am sure one service principal can work across multiple subscriptions.

Answer 1

If you create the service connection at the subscription scope level (that is done following the link), the service connection is bound to the subscription you enter when you create it.

If you instead create a management group that contains both subscriptions and create the service connection using that management group as the scope, then it should work on both subscriptions.

Personally, I don't like giving service principals and service connections too wide access, so I normally create at least one per subscription.

EDIT (due to poor formatting in the comments)

Service connections can be set at the task level if the task need one:

steps:
  - task: AzureKeyVault@1
    displayName: KeyVault
    inputs:
      azureSubscription: ${{ parameters.serviceConnectionName }}
      KeyVaultName: ${{ parameters.keyVaultName }}

EDIT 2:

In a Terraform task the service connection is set like this:

environmentServiceNameAzureRM: 'YourServiceConnectionName'