python mechanize session not saved

Question

I'm trying to use python mechanize to retrive the list of apps on iTunes connect. Once this list is retrieved, further work will be done with those links.

Logging in succeeds but then when i follow the "Manage Your Applications" link I get redirected back to the login page. It is as if the session gets lost.

import mechanize
import cookielib
from BeautifulSoup import BeautifulSoup
import html2text

filename = 'itunes.html'

br = mechanize.Browser()
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)

br.set_handle_equiv(True)
br.set_handle_redirect(True)
br.set_handle_referer(True)
br.set_handle_robots(False)

br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)

br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]

br.open('https://itunesconnect.apple.com/WebObjects/iTunesConnect.woa')
br.select_form(name='appleConnectForm')
br.form['theAccountName'] = username
br.form['theAccountPW'] = password

br.submit()

apps_link = br.find_link(text='Manage Your Applications')
print "Manage Your Apps link = ", apps_link
req = br.follow_link(text='Manage Your Applications')

for app_link in br.links():
    print "link is ", app_link

Any ideas what could be wrong?

score 0 · Answer 1 · answered Mar 19 '12 at 22:47

0

You need to save/load the cookiejar

answered Mar 19 '12 at 22:47

John La Rooy

295,403
53
369
502

When should the cookie jar be saved/loaded? – Inn0vative1 Mar 19 '12 at 22:55
You should load it early, probably straight after you create it, and save it when the program exits – John La Rooy Mar 19 '12 at 23:00
The first time it runs the load would fail since the file doesn't exist, so you might want a try/except around that – John La Rooy Mar 19 '12 at 23:01
That doesnt seem to be solving the problem. If the cookie is retrieved when the user logs in, isn't it already available in memory in the Browser object during this running instance, without having to save or load from disk? Also when i turned on `debug_http` to print http headers, i see several cookies are set with requests. The initial login succeeds but trying to access subsequent links fails – Inn0vative1 Mar 20 '12 at 16:54

score 0 · Accepted Answer · answered Mar 21 '12 at 07:29

Figured this out after further investigation. This was due to a known bug in cookielib documented here: http://bugs.python.org/issue3924

Basically some sites (notably itunesconnect), set the cookie version as a string not an int. Which causes an error in cookielib since it does not deal with that condition. The fix at the bottom of that issue thread worked for me.

python mechanize session not saved

2 Answers2