Automate the boring stuff (can't get BeautifulSoup to parse HTML)

Question

This script takes a keyword(s), searches it on Google and then opens up tabs of the results in the browser. The script returns an empty array at the select method and I'm confused as to why. I checked the HTML of the search results and the CSS selector seems like it should work.

#! /usr/bin/env python3

import webbrowser, sys, requests, bs4, pyperclip

if len(sys.argv) > 1:
    address = ' '.join(sys.argv[1:])
else:
    address = pyperclip.paste()

res = requests.get('https://www.google.com/search?q=' + address)

soup = bs4.BeautifulSoup(res.text, "lxml")

linkElems = soup.select('.r a')
numOpen = min(5, len(linkElems))
for i in range(numOpen):
    webbrowser.open('http://google.com' + linkElems[i].get('href'))

this has come up multiple times and has multiple answers on StackOverflow e.g. https://stackoverflow.com/search?q=soup.select%28%27.r+a%27%29+boring — QHarr, Nov 05 '19 at 00:15

score 0 · Answer 1 · answered Nov 04 '19 at 22:36

Try setting a User-Agent in your headers:

from bs4 import BeautifulSoup
import requests

url = "https://www.google.com/search?q=python"

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:18.0) Gecko/20100101 Firefox/18.0"
}

response = requests.get(url, headers=headers)
assert response.status_code == 200

soup = BeautifulSoup(response.text, "html.parser")

for element in soup.select(".r a"):
    print(element)

Automate the boring stuff (can't get BeautifulSoup to parse HTML)

1 Answers1