Requests.get() returns different HTML to Chrome Dev Tools

Question

In the interests of educational web-scraping I'm trying to parse some html but something I can't explain is happening. When I view the page's source code via developer tools in Chrome I see

<link rel="preload" href="/static/bundles/es6/Consumer.js/54df4d9114a3.js" as="script" type="text/javascript" crossorigin="anonymous" />

but when I load it in Python via requests.get(url, headers) I get

<link rel="preload" href="/static/bundles/metro/Consumer.js/54df4d9114a3.js" as="script" type="text/javascript" crossorigin="anonymous" />

The difference is es6 is metro. What may be causing this? What could cause the same url to return different static html?

I'm using an identical User-Agent string to what's shown in Dev Tools, so I suspect I could be missing some other header information.

headers = {
    'Access-Control-Allow-Origin': '*',
    'Access-Control-Allow-Methods': 'GET',
    'Access-Control-Allow-Headers': 'Content-Type',
    'Access-Control-Max-Age': '3600',
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36'
    }
html_content = requests.get(url, headers).text

I'm aware of this question, with an almost identical title, Requests.get showing different HTML than Chrome's Developer Tool, but it doesn't answer the question and I don't want to use Selenium or a Web Driver. I'm after speed.

The server may react on some difference in the sent header, probably the user agent. — Michael Butscher, Oct 29 '20 at 03:09
#Michael Butscher, the user agent string is identical, is there more to it than that? — livin_amuk, Oct 29 '20 at 03:21
You should see in the Chrome developer tools that the browser usually sends some more header lines to the server. — Michael Butscher, Oct 29 '20 at 03:34
@Michael Butscher, I've updated the question with the full list of browser header lines. Does anything stand out to you? — livin_amuk, Oct 29 '20 at 04:31
Hard to tell but "cookie" and the "accept..." header lines are worth a try. — Michael Butscher, Oct 30 '20 at 04:23

Requests.get() returns different HTML to Chrome Dev Tools

0 Answers0