12

I am trying to download a file from a website using python and mechanize. My current code successfully logs on to the website and opens the page that contains the download link.

The download link is: https://www.lendingclub.com/browse/browseNotesRawDataV2.action

The info for the link is: 

Link(base_url='https://www.lendingclub.com/browse/browse.action', url='/browse/browseNotesRawDataV2.action', text='', tag='a', attrs=[('class', 'master_pngfix'), ('id', 'browseDownloadAllLink'), ('href', '/browse/browseNotesRawDataV2.action')])

I use the follow_link method to click on the link:

br = mechanize.Browser()
br.follow_link(url='/browse/browseNotesRawDataV2.action')

However, nothing happens and no file is downloaded. When I open the link in my browser when I'm logged on, it pauses for a few seconds and downloads the file.

How can I download the file using Python?

user1137778
  • 1,111
  • 3
  • 14
  • 26
  • 2
    Did you enable logging? Anything you've seen there? Did you try "retrieve" instead of follow_link? – niko Jun 12 '12 at 17:54
  • How do you enable logging? I'm using Eclipse on a mac. I tried retrieve and nothing happened. Is there some way to indicate a filename to save as? – user1137778 Jun 12 '12 at 19:53
  • 2
    You can turn on logging with `import logging; logging.basicConfig(level=logging.DEBUG)` – David Wolever Jun 12 '12 at 20:53
  • I turned on logging but I don't see any errors. – user1137778 Jun 12 '12 at 23:37
  • What else do you see in the log? Any HTTP 404, 403, 500 errors? Does it say anything about a file beeing downloaded?. As for the parameters, you can look them up using `pydoc mechanize` or `help(mechanize.Browser.retrieve)` from an interactive python shell. – niko Jun 13 '12 at 08:46
  • I had to enable logging manually in Eclipse. The file was being stored in a temp location, so I just added this code to give it a filename: `br.retrieve('https://www.lendingclub.com/browse/browseNotesRawDataV2.action','loans.csv')[0]` – user1137778 Jun 13 '12 at 16:25

1 Answers1

33

For anyone who's interested, this was the solution:

br.retrieve('https://www.lendingclub.com/browse/browseNotesRawDataV2.action','l‌​oans.csv')[0]
user1137778
  • 1,111
  • 3
  • 14
  • 26
  • Just to follow up on @user1137778's answer, make sure to provide the filename (the second argument in the example), otherwise, a temporary file is created and subsequently deleted. For example, br.retrieve(yourcustomurl, f'{yourcustomfilename}.mp3')[0] – Yellowjacket11 Dec 29 '20 at 22:34