In a couple of cases where we had this issue, it is mostly to do with network connectivity.
Few things which helped us figure out the root cause -
Access the "https://xyz/.well-known/openid-configuration" route from a browser on the server.
If you are not able to access the url then it means that the server is unable to connect to the Idserver installation. This is a network level issue.
If you are able to access the url from the server where the relying application is hosted, but the relying application is throwing an error -> it means that, a proxy is configured on the server. The browser automatically uses the proxy, where as you have to set the proxy in the relying party application as below in the startup.cs
var request = WebRequest.Create(uri);
var myProxy = new WebProxy {Address = new Uri("proxy uri")};
request.Proxy = myProxy;
var response = request.GetResponse();
This will ensure that all the http requests originating from the code will also use the same proxy.
If the above doesnt help, Check if the IIS where Idserver is installed, allows TLS 1.0 and 1.1. THis is disabled on some servers for security purposes. If that is the case, use the below code to make ur application use tls 1.2 and the call will succeed
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;