So I'm hoping to be able to move some ETL pipelines over from Azure Databricks over to Azure Synapse Analytics, but I'm running into some issues with a vnet integration.
The case is as follows: We have a vnet that is peered to a VNG-vnet that provides access to on prem SQL servers. For these ETLs I need to be able to access the above mentioned SQL servers.
Azure Databricks solves this problem with the workspace letting me assign specific subnets instead of managing the networking for me:
resource "azurerm_subnet_network_security_group_association" "databricks_pub_sub_nsg_assos" {
subnet_id = azurerm_subnet.databricks_pub_sub.id
network_security_group_id = azurerm_network_security_group.databricks_nsg.id
}
resource "azurerm_subnet_network_security_group_association" "databricks_priv_sub_nsg_assos" {
subnet_id = azurerm_subnet.databricks_priv_sub.id
network_security_group_id = azurerm_network_security_group.databricks_nsg.id
}
resource "azurerm_databricks_workspace" "databricks" {
name = "newcorp-databricks"
resource_group_name = azurerm_resource_group.main_rg.name
managed_resource_group_name = "databricks-resources-rg"
location = azurerm_resource_group.main_rg.location
sku = "premium"
custom_parameters{
public_subnet_name = azurerm_subnet.databricks_pub_sub.name
private_subnet_name = azurerm_subnet.databricks_priv_sub.name
virtual_network_id = var.iver_vnet_id
}
depends_on = [azurerm_network_security_group.databricks_nsg]
}
Basically I'm wondering if anyone know of any way to inject a vnet (subnet) that my Azure Synapse Spark cluster can run in?
If not: can anyone think of a clever way around where we still keep the ETL (or ELT if you've got a storage account solution in mind?) in code -> not fond of the Azure Data Factory drag/drop/config solution.
