Bicep to Deploy Data Factory Managed Virtual Network

Question

I'm trying to create a bicep module that will deploy a data factory along with a managed vnet. Here's what I have:

param dfName string
 param sqlId string
    
 resource df 'Microsoft.DataFactory/factories@2018-06-01' = {
   name: dfName
   location: resourceGroup().location
   identity: {
     type: 'SystemAssigned'
   }
 }
    
 resource integrationRuntime 'Microsoft.DataFactory/factories/integrationRuntimes@2018-06-01' = {
   name: '${dfName}/managedVnetIr' 
   properties: {
     type: 'Managed'
     typeProperties: {
       computeProperties: {
         location: 'AutoResolve'
         dataFlowProperties: {
           computeType: 'General'
           coreCount: 8
           timeToLive: 0
         }
       }
     }
   }
   dependsOn: [
     df
   ]
 }
    
 resource managedVnet 'Microsoft.DataFactory/factories/managedVirtualNetworks@2018-06-01' = {
   name: '${dfName}/vnet'
   properties: { 
   }
   dependsOn: [
     integrationRuntime
   ]
 }
    
 resource managedPrivateEndpoint 'Microsoft.DataFactory/factories/managedVirtualNetworks/managedPrivateEndpoints@2018-06-01' = {
   name: '${dfName}/vnet/pe'
   properties: {
     privateLinkResourceId:sqlId
     groupId: 'sql'
   }
   dependsOn: [
     managedVnet
   ]
 }
    
 output dfId string = df.identity.principalId

When this module is run, I get the following error:

"status": "Failed", "error": { "code": "ResourceNotFound", "message": "Resource not found. ResourceId: '/subscriptions/8210b2ab-404f-40a5-baba-1cde6d89c670/resourceGroups/rg-contactcentre-dev-001/providers/Microsoft.DataFactory/factories/df-ccsurvey-dev-001/managedvirtualnetworks/vnet'." }

I've also tried the following (based on answer from AnsumanBal-MT)

param dfName string
param sqlId string
param vnetName string

resource df 'Microsoft.DataFactory/factories@2018-06-01' = {
  name: dfName
  location: resourceGroup().location
  identity: {
    type: 'SystemAssigned'
  }
}

resource integrationRuntime 'Microsoft.DataFactory/factories/integrationRuntimes@2018-06-01' = {
  parent: df
  name: '${dfName}-managedVnetIr' 
  properties: {
    type: 'Managed'
    typeProperties: {
      computeProperties: {
        location: 'AutoResolve'
        dataFlowProperties: {
          computeType: 'General'
          coreCount: 8
          timeToLive: 0
        }
      }
    }
  }
}

resource managedVnet 'Microsoft.DataFactory/factories/managedVirtualNetworks@2018-06-01' = {
  parent:df
  name: vnetName
  properties: { 
  }
  dependsOn: [
    integrationRuntime
  ]
}

resource managedPrivateEndpoint 'Microsoft.DataFactory/factories/managedVirtualNetworks/managedPrivateEndpoints@2018-06-01' = {
  parent:managedVnet
  name: '${vnetName}-sql-pe'
  properties: {
    privateLinkResourceId:sqlId
    groupId: 'sql'
  }
  dependsOn: [
    managedVnet
  ]
}

output dfId string = df.identity.principalId

but this gives the following error:

{ "status": "Failed", "error": { "code": "ResourceDeploymentFailure", "message": "The resource operation completed with terminal provisioning state 'Failed'." } }

Can anyone spot what I've done wrong or direct me to a working sample please?

Hello @Rob Bowman, I tested for SQL database and added an update to answer , Please let me know if that works.. — Ansuman Bal, Oct 15 '21 at 16:28
Hello @Rob Bowman, I updated the code as per your scenario and the configurations you need . I have tested it and it was deployed Successfully . Please use the whole code which i have provided in update :2 section . It will create a vnet, then sql server and adf and finally managed privateendpoint for sql in adf. — Ansuman Bal, Oct 19 '21 at 15:20

Ansuman Bal · Accepted Answer · 2021-10-19T15:23:41.873

For creating a managed virtual network on the Data Factory , you have to reference to an existing Vnet in your resource group.

Update:1

While testing for creating a managed private endpoint for sql database I encountered the same error as you , using your code it failed after 1hr 18mins with provisioning failed .

As I was testing for SQL server , I found two issues which are the groupId should be sqlServer and also the managed vnet for adf won't be able to communicate with sql server as it is not added to firewall and virtual networks.

To solve the issue you need to follow the below two steps :

If you are referencing Microsoft.SQL/Servers please change the groupID to sqlServer and if you are referencing 'Microsoft.Synapse/Workspaces' you can keep it as sql. You can refer this Microsoft Document for private endpoint sub-resources name.
Please add the existing virtual network that you are using to create managed virtual network for ADF in the SQL server . (If you are referencing synapse then go to Synapse>>Networking>>Allow Azure services and resources to access this workspace )

After the above 2 steps are done , the deployment will succeed.

Update:2

Scenario: Create a SQL Server with Vnet and then reference the vnet and sql to create adf managed virtual network and private endpoint.

Please use the below code which I have tested as per your requirement :

param serverName string = uniqueString('sql', resourceGroup().id)
param sqlDBName string = 'SampleDB'
param administratorLogin string
@secure()
param administratorLoginPassword string
param virtualNetworkName string = 'azure_mysql_vnet'
param subnetName string = 'azure_mysql_subnet'
param virtualNetworkRuleName string = 'AllowSubnet'
param vnetAddressPrefix string = '10.0.0.0/16'
param subnetPrefix string = '10.0.0.0/16'
param dfName string

resource virtualNetworkName_resource 'Microsoft.Network/virtualNetworks@2020-06-01' = {
  name: virtualNetworkName
  location: resourceGroup().location
  properties: {
    addressSpace: {
      addressPrefixes: [
        vnetAddressPrefix
      ]
    }
  }
}

resource virtualNetworkName_subnetName 'Microsoft.Network/virtualNetworks/subnets@2020-06-01' = {
  parent: virtualNetworkName_resource
  name: subnetName
  location: resourceGroup().location
  properties: {
    addressPrefix: subnetPrefix
  }
}

resource serverName_resource 'Microsoft.Sql/servers@2020-02-02-preview' = {
  name: serverName
  location: resourceGroup().location
  properties: {
    administratorLogin: administratorLogin
    administratorLoginPassword: administratorLoginPassword
  }
}

resource serverName_sqlDBName 'Microsoft.Sql/servers/databases@2020-08-01-preview' = {
  parent: serverName_resource
  name: sqlDBName
  location: resourceGroup().location
  sku: {
    name: 'Standard'
    tier: 'Standard'
  }
}

resource serverName_virtualNetworkRuleName 'Microsoft.Sql/servers/virtualNetworkRules@2021-02-01-preview' = {
  parent: serverName_resource
  name: virtualNetworkRuleName
  properties: {
    virtualNetworkSubnetId: virtualNetworkName_subnetName.id
    ignoreMissingVnetServiceEndpoint: true
  }
}

 resource df 'Microsoft.DataFactory/factories@2018-06-01' = {
   name: dfName
   location: resourceGroup().location
   identity: {
     type: 'SystemAssigned'
   }
 } 

 resource integrationRuntime 'Microsoft.DataFactory/factories/integrationRuntimes@2018-06-01' = {
   parent: df
   name: '${dfName}-managedVnetIr' 
   properties: {
     type: 'Managed'
     typeProperties: {
       computeProperties: {
         location: 'AutoResolve'
         dataFlowProperties: {
           computeType: 'General'
           coreCount: 8
           timeToLive: 0
         }
       }
     }
   }
 } 
 resource managedVnet 'Microsoft.DataFactory/factories/managedVirtualNetworks@2018-06-01' = {
   parent:df
   name: virtualNetworkName
   properties: { 
   }
   dependsOn: [
     integrationRuntime
   ]
 }
    
 resource managedPrivateEndpoint 'Microsoft.DataFactory/factories/managedVirtualNetworks/managedPrivateEndpoints@2018-06-01' = {
   parent:managedVnet
   name: '${virtualNetworkName}-${serverName}-pe'
   properties: {
     privateLinkResourceId: serverName_resource.id
     groupId: 'sqlServer'
   }
   dependsOn: [
     managedVnet
   ]
 }

Output:

Note: After the deployment succeeds , you need to manually approve the private endpoint request which is in pending state from SQL server as shown below:

I made the amendment as you suggested. My bicep module now receives the name of an existing vnet and uses this in the name property of the manged vnet resource. The deployment ran for 1h 18 minutes but failed when deploying type "managedPrivateEndpoints" with the error: he resource operation completed with terminal provisioning state 'Failed'. — Rob Bowman, Oct 15 '21 at 10:19
Hi, it's deploying the endpoint for sql . The sql id is the output from a bicep module that gets called earlier in the sequence in order to create the sql instance: resource sqlServerName_resource 'Microsoft.Sql/servers@2015-05-01-preview' = { name: sqlServerName location: resourceGroup().location properties: { administratorLogin: sqlAdminUsername administratorLoginPassword: sqlAdminPwd } } output sqlId string = sqlServerName_resource.id — Rob Bowman, Oct 18 '21 at 16:25
Hi @AnsumanBal-MT I’m afraid it doesn’t work. Please seem my first comment in response to your answer — Rob Bowman, Oct 18 '21 at 17:40
@RobBowman, I had updated the answer with new code and update of few steps to follow after the SQL is deployed .. let me try with sequence deployment and what needs to be configured while deploying the SQL server and also after that I will try deploying adf with managed private endpoint for SQL server as sub resource .I will try to update you after testing.. Let me know if I am wrong about the scenario you are trying .. — Ansuman Bal, Oct 18 '21 at 18:19
I think you are correct re the scenario. I have a "Main" bicep template that receives the name of a pre-existing vnet as a param. Main calls a "Sql" template when returns the id of the created sql instance. It then passes this along with the vnet name to a "DataFactory" bicep template which tries to create: data factory, integration runtime, managed vnet,, managed private endpoint — Rob Bowman, Oct 19 '21 at 13:35
Thanks for the update. I'm off the project now but should be able to test early next week. — Rob Bowman, Oct 20 '21 at 12:59
Np!! Glad to be of Help!! @RobBowman, I tested it and its working for me .. so hoping that this will surely resolve your issue .. — Ansuman Bal, Oct 20 '21 at 13:08

Bicep to Deploy Data Factory Managed Virtual Network

1 Answers1