Python: interrupting urllib2.urlopen() with SIGTERM

Question

I'm using urllib2.urlopen() to open sometimes potentially large files. I have a signal handler to catch SIGTERM, but is it possible to actually interrupt urlopen() when it's downloading a big file to close my program immediately, without waiting for the call to finish?

Answer 1

urlopen returns a file-like object. Data is only sent over the network when you make a .read() request on this object. (Your OS does some buffering of network data, so this isn't strictly true, but it's close enough for all practical purposes.)

So simply use the .read() method's capability to read data in chunks using a loop, perhaps 16K or 64K at a time, rather than retrieving the whole file at once. In your signal handler, then, you can close the file-like object and the file will stop downloading after the current chunk finishes. The smaller the chunk you use, the less latency there will be in stopping the download.

I'd use a global variable to hold the reference to file-like object so it is accessible in your signal handler; in this case it seems like the simplest solution.

If you should happen to try to read from the file-like object after closing it, you will get an exception, which you can handle gracefully.