In Python 3, requests.get() gives incomplete json response

Question

I am trying to get data using requests.get(). And the response data is large(containing 10000 mongodb records). But the response I get is almost always broken. Very few times I got the correct result.

Example:
Should be like this:

[
    {
        "_id":"5a72c839c634133e1e9ab502",
        "data":{"today_wh":13500},
        "dts":"2018-02-01T07:56:31.000Z",
        "ts":1517471791
    },
    {
        "_id":"5a72c839c634133e1e9ab503",
        "data":{"today_wh":13500},
        "dts":"2018-02-01T07:57:06.000Z",
        "ts":1517471826
    }
]

Comes like this:

[
    {
        "_id":"5a72c8ecc634133e1e9ab51b",
        "data":{"today_wh":13700},
        "dts":"2018-02-01T08:00:01.000Z",
        "ts":1517472001
    },
    {
        "_id":

What to do to get the whole result?

show the code you use to issue and receive the request. – Patrick Artner Feb 21 '18 at 11:18 — Patrick Artner, Feb 21 '18 at 11:18
@martinho - Is this issue fixed? – Amal Dev S I Jul 24 '20 at 07:05 — Amal Dev S I, Jul 24 '20 at 07:05

score 0 · Answer 1 · answered Feb 21 '18 at 11:23

The default User-Agent set by requests is 'User-Agent': 'python-requests/2.7.6'. Try to simulate as it is coming from a browser and not a script. Try simulating a User-Agent as follows:

import requests
url = "http://example.com/"
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36',
    'Content-Type': 'text/html',
}
response = requests.get(url, headers=headers)
html = response.text

In Python 3, requests.get() gives incomplete json response

1 Answers1