Python http.client giving empty string when I use response.read().decode()

Question

I am using http.client to hit a POST api and get the json response. My code is working correctly when I print response.read(). However, for some reason, this response is limited to only 20 results and the total count of results is over 20,000. I want to get the complete response in a variable using response.read().decode(), I am hoping that the variable will contain the complete json string. The issue is that I am getting an empty string when I used decode(). How do I get this done? How do I get the complete results?

import http.client

host = 'jooble.org'
key = 'API_KEY'

connection = http.client.HTTPConnection(host)
#request headers
headers = {
    "Content-type": "application/json"}
#json query
body = '{ "keywords": "sales", "location": "MA"}'
connection.request('POST','/api/' + key, body, headers)
response = connection.getresponse()
print(response.status, response.reason)
print(response.read())
print(response.read().decode())

The API code is provided by the website itself. Not sure how to convert this code into a version using requests module. Will look into it but it would be preferred if I can solve it using http.client — Mohammad Haris, Jun 23 '22 at 22:39
You're calling `response.read()` twice. The first call reads the entire response, there's nothing left for the second call to read, so there's nothing to decode. — Barmar, Jun 23 '22 at 22:42

score 2 · Accepted Answer · answered Jun 23 '22 at 22:49

2

Don't call response.read() twice. response is a stream, so each call to read() continues from where the previous one ended. Since the first call is reading the entire response, the second one doesn't read anything.

If you want to print the encoded and decoded response, assign response.read() to a variable, then decode that.

data = response.read()
print(data)
print(data.decode())

But this can be done more simply using the requests module.

import requests

host = 'jooble.org'
key = 'API_KEY'

body = { "keywords": "sales", "location": "MA"}

response = requests.post(f'https://{host}/api/{key}', json=body)

print(response.content)

Note that in this version body is a dictionary, not a string. The json parameter automatically converts it to JSON.

answered Jun 23 '22 at 22:49

Barmar

741,623
53
500
612

Thanks. This is helpful. I am still getting 20 results. Maybe I can write the results to a file and that will do the job? Will explore this. – Mohammad Haris Jun 23 '22 at 22:56
Many APIs limit the the number of results you can get in each call. Check the documentation to see if you can specify the number to return, and how to get the next batch. You'll need to call it in a loop. – Barmar Jun 23 '22 at 22:57
Thanks. 1 thing I have also noticed is that the results page has 20 results initially, when I scroll down more, the other (probably 20 more) results load and get displayed. Could this be a reason for this? If yes, can this be solved via some sort of timeout? Not sure.. – Mohammad Haris Jun 23 '22 at 23:23
Do you have a link to the API documentation? It should mention pagination. – Barmar Jun 23 '22 at 23:48
Unfortunately, documentation for this API is not available publicly – Mohammad Haris Jun 24 '22 at 15:01
Then you're on your own. Search it for parameters used for pagination. – Barmar Jun 24 '22 at 15:27

Python http.client giving empty string when I use response.read().decode()

1 Answers1