0

My python script needs to establish two HTTPConnection from the httplib library, each connecting to a different domain to GET data. I also have to make multiple requests to those domains multiple times. The data returned from requests is all in JSON.

import httplib, json
... ...

# Establish connection to DOMAIN1
conn1 = httplib.HTTPConnection("DOMAIN1")
conn1.request("GET", "URL1")
objs1 = json.loads(conn1.getresponse().read())
print json.dumps(objs1, indent=4)
#-- here, objs1 is nicely printed out in python console --#

#Establish connection to DOMAIN2
conn2 = httplib.HTTPConnection("DOMAIN2")
conn2.request("GET", "URL2")
objs2 = json.loads(conn2.getresponse().read())
print json.dumps(objs2 , indent=4)
#-- here, objs1 is nicely printed out in python console --#

# Reuse the first connection to DOMAIN1 and call the same URL
conn1.request("GET", "URL1")
objs3 = json.loads(conn1.getresponse().read()) #-- here, socket.error is thrown out! The program does not go any further --#
print json.dumps(objs3, indent=4)

... ...

I suspect it's due to multiple connections to different domains as even I removed the connection to DOMAIN2 and put in a few more connections to DOMAIN1 for testing purpose, my code worked!

Any idea to solve the problem?

alextc
  • 3,206
  • 10
  • 63
  • 107
  • if you use DOMAIN2 everywhere then what happens? – akonsu Jan 27 '15 at 23:16
  • Can you use urllib2 instead of httplib? – Paul Rooney Jan 27 '15 at 23:19
  • @PaulRooney. Actually I tried using urllib2 for connecting to DOMAIN2 but still failed due to the same error. I think urllib was developed on top of httplib, wasn't it? – alextc Jan 27 '15 at 23:24
  • As @akonsu basically said, this could easily be a domain problem, not a Python problem. Try connecting to _only_ `DOMAIN2`. If needed, try connecting to `DOMAIN2` with your Web browser, and on the command line with `telnet` (to port 80), `ping`, and `traceroute`. The Python standard library isn't the problem. (I just tried it, using the first two domains I thought of --- they worked fine.) – Kevin J. Chase Jan 27 '15 at 23:25
  • Can you get an errno when the socket.error is thrown? – Paul Rooney Jan 27 '15 at 23:26
  • @KevinJ.Chase Sorry I edited my question with a bit more clarification. The problem is it actually failed when re-calling DOMAIN1 after successfully fetching and printing out the JSON data from DOMAIN2. – alextc Jan 27 '15 at 23:28
  • 1
    you need to re-establish the connection `conn1` – akonsu Jan 27 '15 at 23:31
  • The server will close the connection after some inactivity, even with keep-alive enabled. Timeout depends on the server but some use only few seconds. Thus it can just be that conn1 timed out out while doing conn2. The first time it will detect this is probably in the read. – Steffen Ullrich Jan 27 '15 at 23:37
  • @akonsu Yeah it worked after I re-created the connection. – alextc Jan 27 '15 at 23:39
  • 1
    This answer might help http://stackoverflow.com/questions/13652875/reusing-httplib-httpconnection-in-python-2-7. My point is you can reconnect rather than recreate. – Paul Rooney Jan 27 '15 at 23:52

0 Answers0