Python: mechanize randomly stops the program infinitely

Question

I am writing some code that uses mechanize to access a website, but often times, when I run the Python code, it halts indefinitely at a line in which I used mechanize.ParseResponse. It is not giving me an error, and in stead, I have to interrupt it via CTRL+C. Also, I believe I am using the correct arguments for the method. However, I am confused as to why my program would suddenly stop running. Any idea?

As extra background, I am running on a Mac.

Any help would be much appreciated!

Edit: The following is my code

Note: I have called python bikes.py and it halts occasionally at the following line:

form = mechanize.ParseResponse(response, backwards_compat=False)

At times, it will also halt at:

text = response.read()

# bikes.py
import re
import webbrowser
import mechanize
import urllib

brands = ["cannondale", "felt", "fuji", "giant", "specialized", "trek"]
keywords = ["52", "53", "54", "shimano", "sora", "tiagra", "105", "ultegra", \
"road", "allez", "defy"]
avoid = ["bmx", "mountain", "kids", "fixie", "jacket", "clothing", "fixed gear", \
"hybrid", "mtb"]

def openLink(text):
    text = text.lower()
    open = False
    for word in avoid:
        if word in text:
            return False
    for word in keywords:
        if word in text:
            open = True

    return open

def scourPage(text, fileRead, fileWrite):
    links = re.findall(r'class="row".+?href="(.+?)"', text)

    for link in links:
        if "http:" in link:
            url = link
        else:
            url = homePage + link

        page = urllib.urlopen(url)
        pageText = page.read()
        title = re.search(r'"postingtitle">.{0,10}<span.+?>[\s\'"]+(.+?)[\s\'"]{0,10}</h2>', \
        pageText, re.DOTALL)
        body = re.search(r'"postingbody">(.+?)</section>', pageText, re.DOTALL)
        openBody = False
        openTitle = False

        if body != None:
            body = body.group(1)
            openBody = openLink(body)

        if title != None:
            title = title.group(1)
            openTitle = openLink(title)

        if (openTitle and openBody) and (url not in fileRead) and (title not in fileRead):
            fileWrite.write(title + "\n" + url + "\n")

        fileWrite.close()

homePage = "http://sfbay.craigslist.org"
request = mechanize.Request(homePage)
response = mechanize.urlopen(request)
forms = mechanize.ParseResponse(response, backwards_compat=False)
form = forms[0]

request = form.click()
response = mechanize.urlopen(request)
emptySearch = response.geturl()
request = mechanize.Request(emptySearch)
response = mechanize.urlopen(request)
forms = mechanize.ParseResponse(response, backwards_compat=False)
form = forms[0]

form["catAbb"] = ["bik"]
form["maxAsk"] = "500"
form.find_control("hasPic").items[0].selected = True

for brand in brands:
    form["query"] = brand

    request = form.click()
    response = mechanize.urlopen(request)
    text = response.read()

    fileR = open('bikes.txt', 'r').read()
    fileA = open('bikes.txt', 'a')

    scourPage(text, fileR, fileA)

    fileA.close()

    next = re.findall(r'class="nplink next".{0,50}<a href=\'(.+?)\'>', text, re.DOTALL)

    while len(next) != 0:
        text = urllib.urlopen(next[0]).read()

        fileR = open('bikes.txt', 'r').read()
        fileA = open('bikes.txt', 'a')

        scourPage(text, fileR, fileA)

        fileA.close()

        next = re.findall(r'class="nplink next".{0,50}<a href=\'(.+?)\'>', text, re.DOTALL)

This code combs through Craigslist ads in an attempt to weed out those that I do not desire. In this case, I'm trying to find a road bike, and avoid any mountain bikes and other listings.

UPDATE:

After waiting quite a long time, I finally keyboard interrupted the run again, and it stopped at the form = mechanize.ParseResponse(response, backwards_compat=False) line. I tried running it again and got this error:

Traceback (most recent call last):
  File "bikes.py", line 97, in <module>
    forms = mechanize.ParseResponse(response, backwards_compat=False)
  File "build/bdist.macosx-10.8-intel/egg/mechanize/_form.py", line 945, in ParseResponse
  File "build/bdist.macosx-10.8-intel/egg/mechanize/_form.py", line 981, in _ParseFileEx
  File "build/bdist.macosx-10.8-intel/egg/mechanize/_form.py", line 758, in feed
  File "build/bdist.macosx-10.8-intel/egg/mechanize/_sgmllib_copy.py", line 110, in feed
  File "build/bdist.macosx-10.8-intel/egg/mechanize/_sgmllib_copy.py", line 192, in goahead
  File "build/bdist.macosx-10.8-intel/egg/mechanize/_form.py", line 654, in handle_charref
  File "build/bdist.macosx-10.8-intel/egg/mechanize/_form.py", line 149, in unescape_charref
ValueError: unichr() arg not in range(0x10000) (narrow Python build)

Can you please give us the code? – svineet Jul 27 '13 at 07:34 — svineet, Jul 27 '13 at 07:34

score 0 · Answer 1 · answered Jul 31 '13 at 20:56

0

Your while loop could be going to infinite, which explains the behaviour. Did you check it isn't?

The runtime error you get when you CTRL-C your code doesn't necessarily mean the code is broken.

answered Jul 31 '13 at 20:56

djas

973
8
24

I have placed print statements throughout my for loop, and it is not stopping at the "text = response.read()" line. I'm pretty sure the while loop is working alright, because if it were infinite, it would be printing a lot of statements, but it isn't. I think it has to do with the urllib, but that's just a guess. – Zhouster Jul 31 '13 at 23:30

Python: mechanize randomly stops the program infinitely

1 Answers1