0

I am trying to work on a web site bug that, at the moment, only manifests in production. Since I can't take down that environment I'm trying to create a copy that is as close as possible. The site is Kentico 11-based, running on Windows Server 2019 Datacenter on an Amazon EC2 t2 instance, backed by an RDS SQL Server Web Edition instance. To create the test environment I make a backup dump of the database and restore it to a different database on the instance, and create an AMI of the EC2 and use that launch a new EC2 instance.

This process has worked dozens of times in the past but this time, and I've done it four times so far today, the web site on the new instance will never return anything but a generic 404 status (that is, the minimal 404 page returned by IIS itself, not from Kentico). Oddly, these requests to the copy are not being logged in its IIS request log. I cannot see anything that's different between the production instance and the copy except for details like the IP address and database connection string, and I cannot imagine any reason why there would be any difference; I created the AMI only minutes before launching the new instance. I've spent an entire day trying again and Googling until my fingers are numb...Does anyone have any ideas or suggestions at all on why the copy isn't working as expected or how to get it to?

EDIT: Wondering what was responding to HTTP requests if IIS was not, I took a look at the response header, which includes this: Server: Microsoft-HTTPAPI/2.0. I found this: https://docs.microsoft.com/en-us/windows/win32/http/http-api-start-page which really only deepens the mystery. What is getting in front of IIS and how? And again, since this instance is launched from an AMI based on a working site, how did this change come to be?

EDIT: I updated the title for more clarity

Don R
  • 143
  • 2
  • 11
  • You include zero information about the environment you have, such as the CMS software you're using and it's configuration, nor any log messages from it or the webserver. It's *impossible* for us to guess at why it's not working. – vidarlo Jun 10 '22 at 22:08
  • I'm a little confused by your comment? I said it's running on Kentico 11, and that not even the web server is logging anything from requests. – Don R Jun 10 '22 at 22:30
  • If it's neither in access logs or error logs; are you sure the requests are even hitting your webserver? – vidarlo Jun 10 '22 at 22:59
  • I'm quite sure they are not hitting the web server, that is, not hitting IIS. – Don R Jun 10 '22 at 23:31
  • So you have some other web server returning a 404. What's your DNS records like? Do you have some proxy in front of your site? What happens if you try to access the site locally from where it's hosted? – vidarlo Jun 10 '22 at 23:33
  • The DNS record is just a CNAME aliasing the AWS node. There are no proxies or anything like that. The site responds exactly the same if I try to hit it from the machine it runs on or from anywhere else. – Don R Jun 10 '22 at 23:38
  • "not being logged in its IIS request log" and "Server: Microsoft-HTPAPI/2.0" indicate the same thing that something else hooks to Windows HTTP API and hijacks those requests. Due to the complexity of Windows HTTP API, nobody can tell what's that without accessing this machine. More details in https://halfblood.pro/everything-you-might-need-about-iis-server-header-58ca268547be – Lex Li Jun 11 '22 at 15:15

1 Answers1

0

After some digging around with netstat and tasklist and determining that the System process was what was responding to requests on ports 80 and 443, I had a thought I ought to have had earlier, and looked to see what happens when making a localhost request on the production server being copied. To my surprise it was the same thing: A 404 returned by something identifying itself as Microsoft-HTTPAPI/2.0...Which finally put the thought into my head that it might be related to the hostname in the IIS port binding.

Sure enough, when I edited those bindings to match the DNS name of the copy host, rather than the name of the server being copied, requests were handled by IIS and everything was back to normal as far as web service.

Another cautionary tale about a developer with little admin experience having to try to solve an admin-side mystery.

Don R
  • 143
  • 2
  • 11