0

I want to open a URL with Python code but I don't want to use the "webbrowser" module. I tried that already and it worked (It opened the URL in my actual default browser, which is what I DON'T want). So then I tried using urllib (urlopen) and mechanize. Both of them ran fine with my program but neither of them actually sent my request to the website!

Here is part of my code:

finalURL="http://www.locationary.com/access/proxy.jsp?ACTION_TOKEN=proxy_jsp$JspView$SaveAction&inPlaceID=" + str(newPID) + "&xxx_c_1_f_987=" + str(ZA[z])

print finalURL

print ""

br.open(finalURL)

page = urllib2.urlopen(finalURL).read()

When I go into the site, locationary.com, it doesn't show that any changes have been made! When I used "webbrowser" though, it did show changes on the website after I submitted my URL. How can I do the same thing that webbrowser does without actually opening a browser?

I think the website wants a "GET"

jacob501
  • 345
  • 2
  • 5
  • 17
  • Isn't there an '?' in `proxy.jspACTION_TOKEN=` like `proxy.jsp?ACTION_TOKEN=` – Paulo Scardine Dec 23 '11 at 16:30
  • Do you know if the service wants a `POST` or a `GET` request? – Wayne Werner Dec 23 '11 at 16:30
  • Not specifically related to your problem, but you might want to note that `urllib.urlopen()` has been deprecated, and is removed it Python 3.0. Use [`urllib2.urlopen()`](http://docs.python.org/library/urllib2.html#urllib2.urlopen) – Amadan Dec 23 '11 at 16:35
  • 1
    This is a good primer for urllib2: http://www.voidspace.org.uk/python/articles/urllib2.shtml#fetching-urls – sgallen Dec 23 '11 at 16:38
  • Try this: write a program with both `webbrowser` and `urllib2`, and have the webbrowser update with one set of data, then change the data slightly with the urllib2 request. Does one request work while the other does not? Then try swapping the requests - this will make sure that your requests are good. – Wayne Werner Dec 23 '11 at 16:51
  • 1
    Is an HTTPError raised? Can you provide an example of a url, minus the variables, that works using the 'webbrowser' module? If I attempt to construct a url based on your submission (http://www.locationary.com/access/proxy.jsp?ACTION_TOKEN=proxy_jsp$JspView$SaveAction&inPlaceID=3674484&xxx_c_1_f_987=ZA[z]) and put it in my actual browser I get: {"manifest":{"errorTimeout":0,"succeed":true,"errorCode":0,"serverVersion":"1.0","type":"locaaccess"},"saveResult":{"message":"Yellowpages.com incomplete. Please enter a value or press \"cancel\".","placeOpenedState":0,"isSucess":false}} – sgallen Dec 23 '11 at 17:15
  • I'm also wondering if you clear cookies on your browser and then attempt your code, does the "webbrowser" module version work? – sgallen Dec 23 '11 at 17:17

2 Answers2

1

I'm not sure what OS you're working on, but if you use something like httpscoop (mac) or fiddler (pc) or wireshark, you should be able to watch the traffic and see what's happening. It may be that the website does a redirect (which your browser is following) or there's some other subsequent activity.

Start an HTTP sniffer, make the request using the web browser and watch the traffic. Once you've done that, try it with the python script and see if the request is being made, and what the difference is in the HTTP traffic. This should help identify where the disconnect is.

Kirsten Jones
  • 2,681
  • 1
  • 17
  • 20
0

A HTTP GET doesn't need any specific code or action on the client side: It's just the base URL (http://server/) + path + optional query.

If the URL is correct, then the code above should work. Some pointers what you can try next:

  1. Is the URL really correct? Use Firebug or a similar tool to watch the network traffic which gives you the full URL plus any header fields from the HTTP request.

  2. Maybe the site requires you to log in, first. If so, make sure you set up cookies correctly.

  3. Some sites require a correct "referrer" field (to protect themselves against deep linking). Add the referrer header which your browser used to the request.

  4. The log file of the server is a great source of information to trouble shoot such problems - when you have access to it.

Aaron Digulla
  • 321,842
  • 108
  • 597
  • 820