When you are integrating an Azure function app by default virtual network uses DNS server only.You can use custom DNS server like below:

I created function app with vnet integeration like below

You can make use of configuring DNS setting for vnet integeration:
In your virtual network -> under setting, DNS server -> select Custom and enter the IP addresses of your custom DNS servers and Save.

The Function App should use the specific DNS servers that have been set up in the VNet once you configured the DNS settings in the networking setup of the Function App
Note that The DNS settings may not reach the Function App right away. It takes few minutes, if you're still having trouble resolving
the address, you might need to restart the Function App in order for it to take up the updated DNS settings

You can make use of following commands in the Kudu DebugConsole https://.scm.azurewebsites.net/DebugConsole
to see if the hostname is returning the right IP address. Use nameresolver domain IPofDNSServer and Nameresolver.exe, an alternative to nslookup, if it fails.
Ensure you have your DNS cache is cleared by using this ipconfig /flushdns
your debug console
Additionally, you can create a DNS zone and add a record set to authorise your server:

References:
Azure Functions networking options | Microsoft Learn
Integrate Azure DNS with your Azure resources - Azure DNS | Microsoft Learn