0

I tried to use WinHTTP Request to retrieve a webpage (HTTPS website) in VB.Net and for some reason it was only returning the partial HTML, is there any length restriction on the number of characters it could take? If so, can I get the content after, say, 10000th character?

The relevant code is here:

         oRequest = New WinHttp.WinHttpRequest            
            oRequest.Open("GET", sQueryURL, False)            
            oRequest.SetTimeouts(0, 600000, 0, 0)
            oRequest.Send()
            If oRequest.Status = "200" Then                
                Debug.Print(oRequest.ResponseText)
            Else

            End If
AZhu
  • 1,312
  • 6
  • 22
  • 40

1 Answers1

0

It's been a while since I've used WinHttpRequest, but I believe as soon as you read ResponseText, WinHttpRequest will abandon processing the response. Since you're diving right in reading the response I'd guess the full response hasn't arrived by the time you print ResponseText!

I think you have 2 options to try:

  1. Use WinHttpRequest.WaitForResponse() to wait for the entire response to be ready
  2. Use WinHttpRequest.ResponseStream to process the response in chunks (you'll need to convert the chunks from bytes to readable text)

I can't tell if you're using VB.Net, but if you are: consider using System.Web.HttpRequest. The interface is pretty much the same and you will have an easier time finding working examples and advice.

Rocjoe
  • 459
  • 4
  • 13
  • Tried to re-write using HTTPRequest class, still has no luck, it always stopped at 249th line, about 2.4k characters.... – AZhu Jun 16 '11 at 17:21
  • 1
    The 2 options above still apply to System.Web.HttpRequest. Larger responses require time to download, so making your code wait for HttpRequest to fully process is important. Try this example instead: http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.getresponse.aspx note it uses StreamReader to process the whole response. – Rocjoe Jun 16 '11 at 18:11
  • 1
    My first hunch is correct, the response from the server is incomplete. Specifically the header contains a "Transfer-Encoding: chunked". The only way to proceed is to add additional code that can tease out the chunked size and go back to the server and ask for the next chunk, until all chunks have been sent. I haven't been able to locate a working example to point you to. I suggest looking for alternatives, maybe this one: http://code.google.com/p/c-sharp-http-client/ but I'll leave it up to you to decide if it does the job! – Rocjoe Jun 16 '11 at 21:48
  • @Rocjoe: So basically there is no way in the existing/default library to get the chunk size? And what header parameter is the chunk size? Many thanks for your help! – AZhu Jun 16 '11 at 22:08
  • @zhuanyi well the chunk size is encoded immediately after the HTTP header. Try running your example URL through Fiddler, then look at the "RAW" view for the response and you'll see the http header followed by two carriage returns, then a number in base-16. That number is the chunk size. – Rocjoe Jun 17 '11 at 00:40
  • @Rocjoe: So I would assume this chunk size is constant right (i.e., it is part of their server setup), so how can I request the next chunk after the first one is received? or rather, how can I request the next chunk directly, skipping the first one? is there any code in winhttprequest or httpwebrequest that does something like this? Many thanks for your help! – AZhu Jun 17 '11 at 03:19
  • 1
    @zhuanyi Unfortunately that's the part where I get stuck too. The best answer I've found describes the logic, but doesn't apply it in running code: http://stackoverflow.com/questions/318489/chunked-encoding-implementation-in-net-or-at-least-pseudo-code ...So there is a way forward, but you might save some time/energy by looking for free alternative Http libraries. – Rocjoe Jun 17 '11 at 12:29
  • The "as you read ResponseText, WinHttpRequest will abandon processing the response" in the answer is not true, or at least not exact, as I'm sure that at least for short (single chunk) responses if it's called immediately after the send the ResponseText property waits for it, 300-400ms in my test. If I precede it with a "sleep 300ms" command it takes only <100ms, proving that it waits for it. – 6diegodiego9 Sep 09 '21 at 10:53