Cannot download some https home pages using HttpWebRequest

Question

I'm just trying to download some site information, from various retail sites using C#, just the plain HTML of the home page. I have an issue with some sites which use https, some work fine while for others I get the following exception

The underlying connection was closed: An unexpected error occurred on a send.

The Inner Exception has

The handshake failed due to an unexpected packet format.

I THINK it has something to do with TLS but for the life of me I can't figure it out. I've been all over Stack Overflow and the HttpWebRequest documentation.

Here's the sample code below I using to call the sites, and I'd be grateful if anyone had any insights on this, it's driving me crazy.

public HttpWebResponse GetWebResponse(HtmlVerb verb, Uri uri)
{

    var request = System.Net.WebRequest.CreateHttp(uri);
    request.Method = verb.ToString();
    request.ProtocolVersion = HttpVersion.Version11;
    request.CookieContainer = new CookieContainer();
    request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
    request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0";
    request.Timeout = (int)TimeSpan.FromSeconds(10).TotalMilliseconds;
    request.AllowAutoRedirect = true;
    request.ServerCertificateValidationCallback = delegate { return true; };
    request.Headers.Add("Accept-Language", "en-US,en;q=0.5");
    request.Headers.Add("Accept-Encoding", "gzip, deflate");
    request.ContentType = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
    request.KeepAlive = false;

    return request.GetResponse() as HttpWebResponse;
}

Kind Regards

Did you specify the port in the URI, as here: http://stackoverflow.com/questions/16895484/getting-handshake-failed-unexpected-packet-format-when-using-webclient-uploa ? — Siderite Zackwehdex, Apr 17 '16 at 11:46
Try diagnosing what really happens using https://blogs.technet.microsoft.com/netmon/p/downloads/ or https://www.wireshark.org/ — Siderite Zackwehdex, Apr 17 '16 at 11:56

score 0 · Answer 1 · answered Apr 17 '16 at 12:22

For anyone else experiencing this issue, I have found the following solution to work. If you include the following lines of code at the start of your Application to initialise the application context, then these problem sites which use https now will download correctly.

private const string DisableCachingName = @"TestSwitch.LocalAppContext.DisableCaching";
private const string DontEnableSchUseStrongCryptoName = @"Switch.System.Net.DontEnableSchUseStrongCrypto";
static void Main(string[] args)
{
    AppContext.SetSwitch(DisableCachingName, true);
    AppContext.SetSwitch(DontEnableSchUseStrongCryptoName, false);

Cannot download some https home pages using HttpWebRequest

1 Answers1