Azure : Web App that connects to Cosmos DB using Private Endpoint returns 'The SSL connection could not be established, see inner exception.'

Question

I have a Web App deployed in Azure that connects to Cosmos DB using Private Endpoint throws the following error

'The SSL connection could not be established, see inner exception.'

The remote certificate is invalid according to the validation procedure: RemoteCertificateNameMismatch

Cosmos DB

Private Endpoint

Private DNS Entry

Web App configuration

Stack Trace:

FBAuthDemoAPI.Controllers.FamilyController:    at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)at System.Net.Http.HttpConnectionPool.ConnectAsync(

Answer 1

I think the issue is with the CosmosDB Endpoint URL pointing to https://family.privatelink.documents.azure.com/, while it should be pointing to the 'normal' DNS name without the privatelink portion.

If DNS is set up correctly for privatelink, then resolving the normal public endpoint would return a private IP address - see below example for a storage account endpoint:

Reference: https://github.com/dmauser/PrivateLink/tree/master/DNS-Integration-Scenarios

The certificate error is likely to be coming from the mismatch in the name between the endpoint that is configured on the WebApp (the privatelink DNS name) and the certificate that is reflecting the normal public endpoint name.

So, if family.documents.azure.com is your CosmosDB account name, then that name should be used for connecting to the database.

In order to test if the name resolution works on your WebApp and test if the CosmosDB endpoint name resolves properly, you can use the built-in console.

To get to the Azure portal-hosted console from your app, go to Tools > Console. In native Windows apps, the tools ping, nslookup, and tracert won't work through the console because of security constraints (they work in custom Windows containers). To fill the void, two separate tools are added. To test DNS functionality, we added a tool named nameresolver.exe. The syntax is: nameresolver.exe hostname

Reference: https://learn.microsoft.com/en-us/azure/app-service/overview-vnet-integration#troubleshooting

So if if the URL below is the endpoint of your CosmosDB and DNS for Private Endpoints is set up properly, this command should return a private IP address:

nameresolver.exe family.documents.azure.com

Please also note that your WebApp needs to be properly integrated into the VNET using VNET Integration, otherwise this will not work (but I would expect a different error if that was not the case).

Answer 2

I don't know what went wrong. Everything started working after I recreated everything.

sub=poc-hubspoke
rgName=SpokeToSpoke
location=eastus
hubVNetName=vnet-hub-$sub
prodVnetName=vnet-prod-$sub
devVnetName=vnet-dev-$sub
onpremVnetName=vnet-onprem-$sub
rgLogName=Logs
loganalytics="loganalytics"

# Create Resource Group
az group create --name $rgName --location $location

# Create Azure Hub VNET
az network vnet create -g $rgName --name $hubVNetName --address-prefixes 10.11.0.0/16 --location $location

# Create Azure Prod VNET
az network vnet create -g $rgName --name $prodVnetName --address-prefixes 10.13.0.0/16  --location $location 
az network vnet subnet create -g $rgName --vnet-name $prodVnetName --name Management --address-prefix 10.13.1.0/24
az network vnet subnet create -g $rgName --vnet-name $prodVnetName --name WebApp --address-prefix 10.13.2.0/24 --delegations Microsoft.Web/serverFarms
az network vnet subnet create -g $rgName --vnet-name $prodVnetName --name Database --address-prefix 10.13.3.0/24
az network vnet subnet create -g $rgName --vnet-name $prodVnetName --name PrivateEndPoint --address-prefix 10.13.4.0/24  --disable-private-endpoint-network-policies true

# Prod Subnet NSG 
az network nsg create -g $rgName -n Prod-Management-subnet -l $location -o table
az network nsg create -g $rgName -n Prod-WebApp-subnet -l $location -o table
az network nsg create -g $rgName -n Prod-Database-subnet -l $location -o table
az network nsg create -g $rgName -n Prod-PrivateEndPoint-subnet -l $location -o table

az network vnet subnet update -g $rgName --vnet-name $prodVnetName --name Management --network-security-group Prod-Management-subnet
az network vnet subnet update -g $rgName --vnet-name $prodVnetName --name WebApp --network-security-group Prod-WebApp-subnet 
az network vnet subnet update -g $rgName --vnet-name $prodVnetName --name Database --network-security-group Prod-Database-subnet
az network vnet subnet update -g $rgName --vnet-name $prodVnetName --name PrivateEndPoint --network-security-group Prod-PrivateEndPoint-subnet

# Peering
az network vnet peering create -g $rgName --name HUBtoProd --vnet-name $hubVNetName --remote-vnet $prodVnetName --allow-vnet-access --allow-forwarded-traffic --allow-gateway-transit

az network vnet peering create -g $rgName --name ProdtoHUB --vnet-name $prodVnetName --remote-vnet $hubVNetName --allow-vnet-access --allow-forwarded-traffic --allow-gateway-transit

# Create Cosmos Database
CosmosDBRG="CosmosDBRG"
cosmosdbAccount="familydbaccount"
cosmosdbName="FamilyDB"
cosmosCollection="Family"
az group create --name CosmosDBRG --location $location
az cosmosdb create --name $cosmosdbAccount --resource-group $CosmosDBRG
az cosmosdb sql database create --account-name $cosmosdbAccount --resource-group $CosmosDBRG --name $cosmosdbName
az cosmosdb sql container create --account-name $cosmosdbAccount --resource-group $CosmosDBRG --database-name $cosmosdbName  --name Family  --partition-key-path "/address/zipcode"  --throughput 400

privateZoneRG="privateZoneRG"
az group create --name $privateZoneRG --location $location
az network private-dns zone create --name "privatelink.documents.azure.com" --resource-group $privateZoneRG

prodVnetID=$(az network vnet show --resource-group $rgName --name $prodVnetName --query id -o tsv)
az network private-dns link vnet create -n prodVnetPrivateDNSLink -g $privateZoneRG --zone-name "privatelink.documents.azure.com" -v $prodVnetID -e false

privateEndPointRG="privateEndPointRG"
cosmosPEName="FamilyDBAccountPE"
privateendpointconnectionname="FamilyDBPEConnection"
cosmosDBPEZoneGroup="cosmosDBPEZoneGroup"
az group create --name $privateEndPointRG --location $location
privateendpointcosmosid=$(az cosmosdb show  --name $cosmosdbAccount --resource-group $CosmosDBRG --query 'id' --output tsv)
prodVnetID=$(az network vnet show --resource-group $rgName --name $prodVnetName --query id -o tsv)
prodPrivateEndPointSubNetID=$(az network vnet subnet show --resource-group $rgName --name PrivateEndPoint --vnet-name $prodVnetName --query id -o tsv)

az network private-endpoint create --name $cosmosPEName --resource-group $privateEndPointRG --subnet $prodPrivateEndPointSubNetID  --private-connection-resource-id $privateendpointcosmosid --location $location --connection-name $privateendpointconnectionname --group-id Sql 

privateZoneID=$(az network private-dns zone show --name "privatelink.documents.azure.com" --resource-group $privateZoneRG --query "id" -o tsv)
az network private-endpoint dns-zone-group create --name $cosmosDBPEZoneGroup --resource-group $privateEndPointRG --endpoint-name $cosmosPEName --private-dns-zone $privateZoneID --zone-name privatelink.documents.azure.com

appServicePlan="MyAppServicePlan"
webAppRG="WebApps"
prodWorkLoadSubNetID=$(az network vnet subnet show --resource-group $rgName --name WebApp --vnet-name $prodVnetName --query id -o tsv)
az group create --name $webAppRG --location $location
az appservice plan create -g $webAppRG -n $appServicePlan --sku B1
az webapp create -g $webAppRG -p $appServicePlan -n $webAppName --subnet $prodWorkLoadSubNetID