Posting form data with requests and python

Question

Posting form data isn't working and since my other post about this wasn't working, I figured I would try to ask the question again so maybe I can get another perspective. I am currently trying to get the requests.get(url, data=q) to work. When I print, I am getting a page not found. I have resorted just to set variables and join them to the entire URL to make it work but I really want to learn this aspect about requests. Where am I making the mistake? I am using the HTML tag attributes name=search_terms and name=geo_location_terms for the form.

search_terms = "Bars"
location = "New Orleans, LA"
url = "https://www.yellowpages.com"
q = {'search_terms': search_terms, 'geo_locations_terms': location}
page = requests.get(url, data=q)
print(page.url)

score 2 · Accepted Answer · answered Aug 21 '18 at 01:50

2

You have few little mistakes in your code:

Check form's action parameter. Then url = "https://www.yellowpages.com/search"
Second parameter is geo_location_terms not geo_locations_terms.
You should pass query parameters in requests.get as params not as request data (data).

So, the final version of code:

import requests

search_terms = "Bars"
location = "New Orleans, LA"
url = "https://www.yellowpages.com/search"
q = {'search_terms': search_terms, 'geo_location_terms': location}
page = requests.get(url, params=q)
print(page.url)

Result:

https://www.yellowpages.com/search?search_terms=Bars&geo_location_terms=New+Orleans%2C+LA

answered Aug 21 '18 at 01:50

Lev Zakharov

2,409
1
10
24

So when using `requests.get()` use ‘params’ and when `requests.post()` use `data` to pass the values? – Kamikaze_goldfish Aug 21 '18 at 03:36
@Kamikaze_goldfish not really, but usually it's so. Check [this answer](https://stackoverflow.com/questions/24535920/difference-between-data-and-params-in-python-requests). – Lev Zakharov Aug 21 '18 at 08:47

score 1 · Answer 2 · answered Aug 21 '18 at 02:55

Besides the issues pointed by @Lev Zakharov, you need to set the cookies in your request, like this:

import requests

search_terms = "Bars"
location = "New Orleans, LA"
url = "https://www.yellowpages.com/search"

with requests.Session() as session:
    session.headers.update({
        'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36',
        'Cookie': 'cookies'
    })

    q = {'search_terms': search_terms, 'geo_locations_terms': location}
    response = session.get(url, params=q)

    print(response.url)
    print(response.status_code)

Output

https://www.yellowpages.com/search?search_terms=Bars&geo_locations_terms=New+Orleans%2C+LA
200

To get the cookies you can see the requests using some Network listener for instance using Chrome Developer Tools Network tab, then replace the value 'cookies'

Posting form data with requests and python

2 Answers2