0

I need to scrape the "People also ask box" from Google for questions and answers.

I make a search on google, then scrape it with BeautifulSoup.

import requests
from bs4 import BeautifulSoup
import html2text
import urllib.request

link = "https://www.google.com/search?client=firefox-b-d&source=hp&ei=v0mUXPu2ApTljwS6iLnABA&ei=lAyVXMPFCsaUsgXqmZT4DQ&q=what+is+java"

headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
page = requests.get(link ,headers = headers)
soup = BeautifulSoup(page.content, 'html.parser')
#For answers :
mydivs = soup.find_all('div', class_="ILfuVd NA6bn")

The results are an empty list. I checked in the html file and the answers are in fact under that class. What is wrong with my code?

Karl Knechtel
  • 62,466
  • 11
  • 102
  • 153
welu
  • 329
  • 3
  • 12
  • 1
    The class is randomly named, specifically to avoid this kind of web scraping. Whatever value you set in your script won't be the value that gets used in the HTML next time. – Karl Knechtel Jun 08 '22 at 18:45
  • 1
    Does this answer your question? [How to scrape dynamic content from a website?](https://stackoverflow.com/questions/55709463/how-to-scrape-dynamic-content-from-a-website) – Karl Knechtel Jun 08 '22 at 18:46

3 Answers3

1

Google's frontpage updates when you enter text in the search box, so you won't be able to get the results while making a simple request to the search page.

You can go to https://google.com in your browser, open the Development Tools panel (usually F12) and watch the Network tab to see the underlying requests being made to the autocomplete API.

It looks like the endpoint is https://www.google.com/complete/search?q=yourQueryHere&client=psy-ab, which is easier to query than an HTML page:

query = "what is java"
res = requests.get("https://google.com/complete/search?client=psy-ab&q=" + query)
print(res)

Also, Google probably doesn't want people to scrape this so you will probably hit rate-limiting if you do too many requests.

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
rpld
  • 86
  • 6
  • Checking the inspecter, I can see the results exist under the class="LGOjhe" in the html file. – welu Mar 27 '19 at 17:04
  • The HTML code may be updated by a script running on the page, after you made the initial request. There is probably a script catching inputs in the search bar, making requests to the above endpoint and updating the HTML document – rpld Mar 27 '19 at 17:20
1

people-also-ask might help you.

pip install people-also-ask

Usage example:

people_also_ask.get_related_questions("coffee", 5)

['How did coffee originate?',
    'Is coffee good for your health?',
  'Who brought coffee America?',
    'Who invented coffee?',
    'Why is coffee bad for you?',
    'Why is drinking coffee bad for you?']
Tuan
  • 11
  • 1
1
  1. To get answers you can use selenium click methods or other libraries that can simulate clicks.
  2. Extract directly from Javascript:enter image description here
  3. Using Google Related Questions API from SerpApi. It's a paid API with a free trial. Check playground.

Code and example:

from serpapi import GoogleSearch
import os

params = {
  "engine": "google",
  "q": "what is java",
  "api_key": os.getenv("API_KEY"),
}

search = GoogleSearch(params)
results = search.get_dict()

for q_and_a in results['related_questions']:
  print(f"Question: {q_and_a['question']}\nAnswer: {q_and_a['snippet']}\n")

Output:

Question: What is Java and why do I need it?
Answer: Java is a programming language and computing platform first released by Sun Microsystems in 1995. There are lots of applications and websites that will not work unless you have Java installed, and more are created every day. Java is fast, secure, and reliable.

Question: What is Java used for?
Answer: One of the most widely used programming languages, Java is used as the server-side language for most back-end development projects, including those involving big data and Android development. Java is also commonly used for desktop computing, other mobile computing, games, and numerical computing.Apr 12, 2019

Question: What is Java in simple words?
Answer: Java is a high-level programming language developed by Sun Microsystems. Instead, Java programs are interpreted by the Java Virtual Machine, or JVM, which runs on multiple platforms. ... This means all Java programs are multiplatform and can run on different platforms, including Macintosh, Windows, and Unix computers.Apr 19, 2012

Question: What is Java and its types?
Answer: The types of the Java programming language are divided into two categories: primitive types and reference types. The primitive types (§4.2) are the boolean type and the numeric types. The numeric types are the integral types byte , short , int , long , and char , and the floating-point types float and double .

Disclaimer, I work for SerpApi.

Dmitriy Zub
  • 1,398
  • 8
  • 35