Unable to Capture Data From Website

Question

I am trying to capture the JSON served by the following URL: https://www.gasunietransportservices.nl/en/shippers/balancing-regime/sbs-and-pos/graphactualjson/M3G

I have found various articles and solutions for my issue, I will walk through each of them below. None of them appear to work.

Simple Case

HttpWebRequest req =
                (HttpWebRequest)
                    WebRequest.Create(
                        "https://www.gasunietransportservices.nl/en/shippers/balancing-regime/sbs-and-pos/graphactualjson/M3G");

            var resp = req.GetResponse();

Give the following Web Exception:

An unhandled exception of type 'System.Net.WebException' occurred in System.dll

Additional information: The underlying connection was closed: An unexpected error occurred on a send.

Next Try: TLS Version

From ..The underlying connection was closed: An unexpected error occurred on a receive, I tried the simple answer initially, setting the TLS Version and Expect100. Code becomes (same result with or without KeepAlive):

    ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
    ServicePointManager.Expect100Continue = true;

    HttpWebRequest req =
        (HttpWebRequest)
            WebRequest.Create(
                "https://www.gasunietransportservices.nl/en/shippers/balancing-regime/sbs-and-pos/graphactualjson/M3G");

    req.KeepAlive = false;

    var resp = req.GetResponse();

Giving exception:

An unhandled exception of type 'System.Net.WebException' occurred in System.dll

Additional information: The underlying connection was closed: An unexpected error occurred on a send.

Try 3: MSDN Solutions

Looking at the following MSD Article 'https://support.microsoft.com/en-us/help/915599/you-receive-one-or-more-error-messages-when-you-try-to-make-an-http-re' (linked to by another answer on other thread, the following is suggested, all of which provide the same exception already stated earlier:

Resolution A: Update to latest framework.

I have tried this up to .Net 4.5 with same result. Article originally written for .Net 1.1. I am unable to go up any further versions due to restrictions with my environments.

Resolution D: Set KeepAlive to false

As shown in excerpt above this makes no difference.

Resolution E: Set MaxIdleTime property

I have tried this as well with no difference in the result. Code becomes:

ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
            ServicePointManager.Expect100Continue = true;

            ServicePointManager.MaxServicePoints = 4;
            ServicePointManager.MaxServicePointIdleTime = 1000;

            HttpWebRequest req =
                (HttpWebRequest)
                    WebRequest.Create(
                        "https://www.gasunietransportservices.nl/en/shippers/balancing-regime/sbs-and-pos/graphactualjson/M3G");

            req.KeepAlive = false;

            var resp = req.GetResponse();

Resultion F: Amend Server Timeout

The web server is an external and out of my control.

Resolution 0: HTTP-100 Header

I have tried the code listed above both with an without the 'Expect100Continue property. Same results.

I have tried all solutions above, both jointly and in isolation with no success. The code provided should run 'as is' if anyone is able to provide any suggestions.

Finally, I tried using the WebClient class, as opposed to HttpWebRequest - again identical results.

Thanks.

Edit 1 - Proxy I am able to access the URL from a browser on the same machine. I have also added the following code:

req.Proxy = new WebProxy("http://MY-PROXY-URL");
            req.Proxy.Credentials = CredentialCache.DefaultCredentials;

Outcome remains unaffected.

Edit 2 - Same code, Separate Environment

Using code from 'Next Try: Tls Version', however running from my server, produces a different exception: System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel.

Edit 3 - Inner Exception

Complete error trace from initial code on my desktop:

System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host\r\n   at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)\r\n   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)\r\n   --- End of inner exception stack trace ---\r\n   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)\r\n   at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)\r\n   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)\r\n   at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)\r\n   at System.Ne
t.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)\r\n   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)\r\n   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)\r\n   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)\r\n   at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)\r\n   at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)\r\n   at System.Net.ConnectStream.WriteHeaders(Boolean async)\r\n   --- End of inner exception stack trace ---\r\n   at System.Net.HttpWebRequest.GetResponse()\r\n   at GasUnieSBS.Program.Main(String[] args) in c:\\dev\\Sandpit\\MessagingPlay\\GasUnieSBS\\Program.cs:line 37"

`Resolution E` works fine for me in a Console app. Can you try that code in a Console app and let us know if it works for you? — mjwills, Aug 29 '17 at 13:17
@mjwills: Same exception. Net 4.5 Console app. I'm wandering if this could be aproxy issue from within my organisation? — GinjaNinja, Aug 29 '17 at 13:19
@GinjaNinja You can use curl command on the terminal to check if you get the contents on your machine. Just to make sure !. If you dont get it, then it means your having a proxy issue. — Hassan Abbas, Aug 29 '17 at 13:25
Thinking about this, I can access it from same machine via a browser, which I think rules out proxy issue? This is the only URL I've noticed issues with. — GinjaNinja, Aug 29 '17 at 13:34
See edit 1, proxy specified and makes no difference. Replacing URL with **https://www.google.com**, code produces no error. (there is an HTTPS in the google URL, doesn't display in comment — GinjaNinja, Aug 29 '17 at 13:40
I have proxy at work, the second try "Next Try: TLS Version" gives no error in console app. — Zasypin N.V., Aug 29 '17 at 13:47
4.5. Can it be firewall, that reason of dropping the connection? — Zasypin N.V., Aug 29 '17 at 13:58
Not sure, I'm getting to the end of my depth of knowledge in HTTPS comms. I think the fact browser works and google HHTP site works via same code would rule this out — GinjaNinja, Aug 29 '17 at 13:59

score 0 · Answer 1 · answered Aug 31 '17 at 15:13

Hope this helps someone else.

By releasing the code sample ("Next Try - TLS Version) onto Windows 10 fixed the issue. Everything I have read sounds like Windows 7 and .Net 4.5 should support full TLS1.2.

However, a scan of the target URL shows they are very tied down on what they allow. One of our security engineers suggested that this is the 2nd similar issue we have had and it relates to our own Windows-7 build config.

I have still not solved the issue on server (Windows 2008 R2).

Unable to Capture Data From Website

1 Answers1