5

Running an ASP.Net Core application based on IdentityServer 4 on an AWS cluster, after 14-21 days all outgoing HTTPS requests take 5min or more to finish.

After an investigation found that the WinHTTP Web Proxy Auto-Discovery Service dies with the following error: The endpoint mapper database entry could not be created. not sure if this is the issue or a side effect of something else that breaks lower in the OS.

Environment:

  • ELB with with an autoscaling group with two instances behind it, issue occurs on both
  • Windows Server 2019

Notes:

  • It is a custom AMI but the only difference with the instances that do not experience this is the product is running on it.
  • Happens on both .Net core 2.2 and 3.1 based versions of the application
  • Happens on both versions based on IdentityServer4 v2 and v3
  • The app is deployed self contained
  • The app uses Kestrel without a reverse proxy in front of it

Tried:

  • Enabled schannel: no errors in the schannel logs set on trace
  • No other notable errors in any of the EventViewer logs
  • Restarting the instance fixes the problem
  • When running curl.exe -v to an HTTPS site on the machine the request gets stuck on schannel: checking server certificate revocation
  • The clocks on the instances are OK
  • Restarting the app process does not resolve the issue
  • Disabling the HTTPClient in the code using the WinHTTP Web Proxy Auto-Discovery Service service still breaks the service.
MarkovskI
  • 1,489
  • 2
  • 21
  • 25
  • Did you find a root cause for this? we are seeing an identical issue – James Skinner Nov 04 '20 at 10:22
  • @JamesSkinner No, still investigating with Microsoft support. – MarkovskI Nov 10 '20 at 21:11
  • Have got the same issue with Windows Server 2019 Datacenter in the AWS environment. The best solution I've seen so far is to restart PC automatically every time when this service got crashed [here](https://blog.wxperts.com/2020/11/20/aws-2019-server-winhttp-web-proxy-auto-discovery-service-errors/) – Alexey Andrushkevich Nov 30 '20 at 10:19
  • 2
    One temporary workaround (that works for us) is to run the service as a regular user, the issue does not occur in this case, previously it was running as a windows service under the `System` account (which is the default) – MarkovskI Nov 30 '20 at 11:04
  • We are still seeing this issue but only on the same server - we have since been running 6 others in identical environments which have had no issues. The only difference in this case will be the an updated AMI so potentially there was just a bug that has now been fixed. – James Skinner Dec 01 '20 at 11:19

2 Answers2

2

For those running into this.

The root of this issue is the WinHTTP autoproxy (wpad) service which is used to autoconfigure proxy settings, WinHTTP 5.1 implements the Web Proxy Auto-Discovery (WPAD) protocol also known as autoproxy, and it must support DHCP. Now this is where the problem occurs, the AWS DHCP does not support DHCP option 252 (used by WPAD) which in turn causes the issue.

The resolution is to disable the WinHTTP service as it is turned on by default in Windows.

MarkovskI
  • 1,489
  • 2
  • 21
  • 25
0

I am seeing this exact same problem on a bare-bones AWS Lightsail instance of Server 2019. I am simply using IIS as a reverse proxy with the URL Rewrite module.

Here are some details about my environment/scenario in case it helps narrow down some common factors.

I have zero .Net applications running on this box, it is simply a reverse proxy.

I have 2 web applications, and 2 websites setup in IIS, each is a reverse proxy for a different back-end API.

The websites in IIS on the reverse proxy use the same CA issued wildcard certificate as the back-end API's.

The back-end API's are also hosted in IIS (Server 2012 R2).

The back-end API's run in a PCI compliant environment so IIS Crypto has been used to tighten up encryption protocols/algo's. I have not used IIS Crypto on the reverse proxy yet, so as I type this I'm starting to wonder if there's an SSL aspect to this.

Phil Sayers
  • 193
  • 1
  • 7