0

I have been stuck on this for a couple of weeks now and this is a follow on from SO question Delphi REST Debugger Returns Error 429 Too Many Requests but Browser Returns JSON as Expected

I was wanting to get the content of a url response using the TNetHTTPRequest and TNetHTTPClient components. I was continually getting 429 errors “too many requests”. When using Firefox Inspect Element to look at network and storage, I discovered that I needed to receive cookies and then send those cookies with my request. Unfortunately, one of the cookies essential to the website content seems to be dependent (I think) on the execution of javascript. I went back to first principles and dropped a TWebbrowser on a form (VCL) and sure enough browser shows a javascript error “Expected Identifier”.

When I use the TWebbrowser in FMX it does not throw an error it just does not return the website contents at all and remains blank. I need FMX as I will be in a cross platform mobile environment.

The URL is https://shop.coles.com.au/a/national/home

I use Delphi Community Edition 10.3.3 Rio.

The URL returns perfectly in commercial browsers Firefox, Safari, Chrome and even CEF4Delphi. Unfortunately, I can’t use CEF as I need cross platform.

I would like to know how to get the website content returned to the browser (or even better NetHTTPClient) without script errors and how to access the browsers current cookies.

Any help will be most appreciated.

Thanks,

John.

JohnT
  • 31
  • 5
  • Difficult to help because lack of technical details. – fpiette Sep 28 '20 at 06:37
  • @fpiette what would you like? I really do want help, but I don't know what you would like. do you want me to drop a browser component and a button on a form and publish the form here somehow for you to compile? It would probably take less time to just do that without me? What would you like from me? – JohnT Sep 28 '20 at 07:09
  • Why do you need to load that page anyway? You should access web services, not HTML pages. – Olivier Sep 28 '20 at 07:16
  • @Olivier I'm not sure I understand the nature of the question. The reason to access that page is to allow the user to interact with the website seamlessly from within my app. As for the HTML..... I would also like to know what was on the page being interacted with. If the HTML is not possible, then that's fine but I do need the webpage to load into the browser component in my app. – JohnT Sep 28 '20 at 07:27
  • An app usually queries web services. Is the website your partner? Do they know you're developing an app to interact with them? – Olivier Sep 28 '20 at 07:33
  • @JohnT If you publish some minimal code which reproduce the issue then we can easily test it, understand it and try to fix. Your question is not very clear. A some point to seems to compare/confuse accessing a webpage and a webservice. Sending a request to a webservice is not the same as getting a webpage using a browser. – fpiette Sep 28 '20 at 09:58
  • is the usual ie7 mode problem, use Amigojack's link – whosrdaddy Sep 28 '20 at 10:15
  • @whosrdaddy Many Thanks. I'll give it a try. – JohnT Sep 29 '20 at 01:08

1 Answers1

0

URL returns perfectly in commercial browsers ... without script errors and how to access the browsers current cookies

If you'd inspect the network traffic (F12 > Network, then requesting your URL) or use uMatrix (to block everything that doesn't belong to the domain by default) you'd see the JS does at least one XHR to amazonaws.com. Your HTTP transfer alone (as done by TNetHTTP*) works fine and you get the same resource that each internet browser gets.

However, you don't operate with what you got (in contrast to the internet browser, which also automatically parses the HTML, sees JS resources, and executes them). TWebbrowser does not what you take for granted most likely due to security settings (try to get an error console in there, preferably F12 again). You need to do the same: parse the HTML resource for JS URIs, requesting those and executing what you get, while still providing the same cookie environment.

For executing JS you could use Chakra or mORMot or BESEN. It's challenging at first, but the more you understand about HTTP (including cookies) and a JS engine, the more you'll see why "things work" in one situation and not in another. There's a reason why an internet browser is a very complex software and not just a downloader.

As per this forcing IE11 Quirks mode might cure your problem already when using TWebBrowser:

TBrowserEmulationAdjuster.SetBrowserEmulationDWORD(TBrowserEmulationAdjuster.IE11_Quirks);
AmigoJack
  • 5,234
  • 1
  • 15
  • 31
  • Many Thanks for such a detailed answer. I appreciate it very much. I'll have a look at all you have suggested. Cheers. – JohnT Sep 29 '20 at 01:07