0

I am writing a script for my employer to get certain data from their own site. For a long list of reasons, I need to get the data off the site as it is shown. I have discovered, some of that data is retrieved via a js call...

In retrospect, I should have gone with mechanize, but I went with twill (thinking I would have access to mechanize since twill describes itself as a thin-wrapper for mechanize) the project is just one post-call away from completion and I desperately do not want to rewrite all my twill based code for mechanize.

I need to submit a post request that is not attached to a form. So I need mechanize (or a similar lib). According to twill's docs, there should be a mechanize browser object retrievable by the below code: 

#after logging in and successfully loading pages
b = get_browser()
mb = b._browser

I get: AttributeError: 'TwillBrowser' object has no attribute '_browser'

Looking at the source it seems twill's browser is not using mechanize's browser anymore?

So my questions are:

  1. Is there still a mechanize browser inside twill.
  2. If so, how do I get it?
  3. If not, how can I get the cookies from twill to mechanize so I can make request using those cookies?

ideally I'd so something like:

cjar = 'cookies.yaml'
save_cookies(cjar)
mb.open(url, data=data, cookies=cjar)

But I know I need a little more magic than just adding cookies=filename :)

Alex Nich
  • 23
  • 1
  • 7

1 Answers1

0
  1. Is there still a mechanize browser inside twill.
  2. If so, how do I get it?

No. It is no longer built into twill. While pip installing twill still includes mechanize into twill's files, twill's code never imports those modules.

  1. If not, how can I get the cookies from twill to mechanize so I can make request using those cookies?

From what I've read, there does not appear to be a simple or trivial way to do this. You can save_cookies from twill (but you can't customize how they are saved.). Mechanize will need help to read the cookie files, and when it loads them it will need to ignore persistance, etc. Mechanize does give you the ability to do this, but I haven't found any usable code for this.
In my case, it was easier to write a method that would log me in in both twill and mechanize.

Alex Nich
  • 23
  • 1
  • 7