How to download a file pushed to a browser using python?

Question

I want to download a zip file using python.

With this type of url, http://server.com/file.zip this is quite simple by using urllib2.urlopen and writing it in a local file.

But in my case I have this type of url: http://server.com/customer/somedata/download?id=121&m=zip, the download is launched after a form validation.

It could be useful to precise that in my case I want to deploy it on heroku, so I can't use spynner that is built with C++. This download is launched after a scraping that uses scrapy.

From a browser the download works well, I get a good zip file with its name. Using python I just get html and header data...

Is there any way to get a file from this type of url in python ?

possible duplicate of [Scrapy, define a pipleine to save files?](http://stackoverflow.com/questions/7123387/scrapy-define-a-pipleine-to-save-files) — Blender, Apr 11 '13 at 17:19
What does the response look like? Is it an HTML page or is it the actual file that you're after, just as a Scrapy `Response` object? — Blender, Apr 11 '13 at 17:27
This is not duplicate, here I can't get .body like in a pdf file. — Corentin Geoffray, Apr 11 '13 at 17:40
I didn't find a solution with scrapy, so I used: data = urllib2.urlopen(url) file = open('name.zip','wb') file.write(data) to get this response — Corentin Geoffray, Apr 11 '13 at 18:00
@CorentinGeoffray: Scrapy rejects many filetypes by default from being crawled. You might want to read the documentation to see how you can remove those files from the blacklist. — Blender, Apr 11 '13 at 19:48
I don't find this blacklist. seems not to be possible easily... — Corentin Geoffray, Apr 12 '13 at 02:31

score 1 · Accepted Answer · edited Apr 11 '13 at 17:31

1

This Site is serving JavaScript which then invokes the download. You have no choice but to: a) evaluate the JavaScript in a simulated Browser environment or b) parse manually what the JS does, and re-implement that in python. e.g. string extraction of the URL and download key, possibly invoking an AJAX request, and finally download the file

I generally recommend Mechanize for webpage related automation, but it cannot deal with JavaScript either, so I guess you can stick with Scrapy if you want to go for plan b).

edited Apr 11 '13 at 17:31

kindall

178,883
35
278
309

answered Apr 11 '13 at 17:22

ch3ka

11,792
4
31
28

Mechanize doesn't support JavaScript. – Blender Apr 11 '13 at 17:24
I know, but maybe cookies are needed – ch3ka Apr 11 '13 at 17:24
Scrapy takes care of cookies, sessions, redirects, etc., so I don't think Mechanize will add any extra functionality. – Blender Apr 11 '13 at 17:26

score 0 · Answer 2 · answered Apr 11 '13 at 17:24

When you do the download in the browser, open up the network tab of the developer console and record what HTTP method (probably POST), the POST parameters, the cookie, and everything else that is part of the validation; then use a library to replicate that.

How to download a file pushed to a browser using python?

2 Answers2