C# HttpRequest returning 404 error despite being browser-accessible

Question

I am currently overseas and testing a vanilla HttpRequest that I know works outside of the country currently am in (Indonesia).

It seems when I try access a particular URL via a browser, it works perfectly. When I try replay exactly the same request via my C# program the same request returns a 404 response. . In fact, even when using a HTTP debugger that sniffs and replays the successful request, the 404 error will still occur.

Interestingly, when I try the browser request with Fiddler running, the browser then gets the 404 response which then seems to redirect to a "censorship" web-page.

Some points to note:

• There is nothing sensitive about the URL in question. All urls such as http://info.cern.ch/ seem to yield a 404.

• The link works perfectly when requested through a browser (except when Fiddler is running).

• The link is http, not https so there are no SSL certificates involved.

• I have tried matching the headers as noted in this post. In fact, as mentioned, if I replay the entire request via a HTTP debugger, I still get a 404 error despite the browser request succeeding.

• I have tried leaving the .Proxy property as-is and also setting to null but both do not work (I can see Proxy is automatically picking up some settings but not sure where from).

• The code is being executed under Winforms .NET Framework v4.6 like so:

  HttpWebRequest request = (HttpWebRequest) WebRequest.Create("http://info.cern.ch/");
                request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36";
                request.Accept =
                    "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9";
  
                request.Proxy = null; // The request fails even if this line is commented
         response = (HttpWebResponse) request.GetResponse();
  

Does anyone know what would cause this behavior and how to ensure my C# application can access the URL and/or emulate the browser request?

Answer 1

I built a little console app (Target framework=.Net 5.0, but it shouldn't make a difference, copied/pasted your code ... and I was UNABLE to reproduce the problem.

The problem does NOT appear to be "your code". Nor the "UserAgent" header or "Proxy" (either of which could potentially cause HTTP 404 errors).

Sample code:

using System;
using System.Net;

namespace TestHttpWebRequest
{
    class Program
    {
        static void Main(string[] args)
        {
            try
            {
                HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://info.cern.ch/");
                request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36";
                request.Accept =
                    "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9";

                request.Proxy = null; // The request fails even if this line is commented
                var response = (HttpWebResponse)request.GetResponse();
                Console.WriteLine("response: StatusCode=" + response.StatusCode + ", StatusDescription=" + response.StatusDescription);
            } catch (Exception ex)
            {
                Console.WriteLine("ERROR: " + ex.Message);
                Console.WriteLine(ex.StackTrace);
            }
        }
    }
}

Sample output:

response: StatusCode=OK, StatusDescription=OK

SUGGESTIONS:

• Try Accept: */* (who knows - it might actually work!)

• Verify the problem is occurring on any arbitrary URL (e.g. "http://info.cern.ch/"), not just the host you were originally trying to connect to.

• Try moving your client to a different network (e.g. install the test app on your laptop, and try it out from a local Starbucks wifi).

• Analyze the traffic with Wireshark