BeautifulSoup find_all function return an empty list []

Question

Problem

I want to do web scraping in this website using BeautifulSoup package, but what I'm actually getting is empty list from find_all function.

Here is a screenshot that illustrates the problem:

What I've tried

This is the most simplified version of my code that I've been able to get, which still produces the problem I described above.

from bs4 import BeautifulSoup
import requests

html_text = requests.get('https://play.google.com/store/apps/details?id=com.google.android.googlequicksearchbox&hl=en').text
soup = BeautifulSoup(html_text, 'lxml')
reviews = soup.find_all('div', class_="d15Mdf bAhLNe")
print(reviews)

... Basically, what the code is doing is scraping all the reviews on the play store website and from class class_ = "d15Mdf bAhLNe".

Also when I try other combinations like soup.find_all({class : d15Mdf bAhLNe}) the result is the same.

Related Topic When I printed the soup I catch it seemed like the scrape worked cause I got an HTML file

Question

So my basic question is, why find_all function can't find the right values? what I am missing?

Answer 1

Looking at the webpage and with a quick look with Postman, I can tell then that exact same content you are looking for is probably Javascript generated.

In fact, the lastest and closest div before your target one should be with class W4P4ne or Njo8s

reviews = soup.find_all('div', class_="W4P4ne")
reviews2 = soup.find_all('div', class_="Njo8s")
print(reviews)
print(reviews2)

After that, you can see that the content becomes funky

<c-data id="i23" jsdata=" OKeYaf;_;9"></c-data>

Using Postman the same behaviour is observe, the 2 target class cant be found.

I suggest you to look on this answer Here

Quick example using Selenium

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.firefox.options import Options

# Config Change depending on your needs
options = Options()
options.binary_location = r"binary_path"
browser = webdriver.Firefox(options=options, executable_path="driver_path")

# Get the data
url = 'https://play.google.com/store/apps/details?id=com.google.android.googlequicksearchbox&hl=en'
browser.get(url)
res = browser.find_elements(By.XPATH, '//div[@class="d15Mdf bAhLNe"]')
print(res)

Answer 2

Actually, the url is loaded dynamically by javascript that's why only bs4 can't grab content. So I use selenium with bs4. Here is the minimal working solution:

CODE:

from bs4 import BeautifulSoup
import time
from selenium import webdriver

driver = webdriver.Chrome('chromedriver.exe')
driver.maximize_window()
time.sleep(8)

url = 'https://play.google.com/store/apps/details?id=com.google.android.googlequicksearchbox&hl=en'
driver.get(url)
time.sleep(10)

soup = BeautifulSoup(driver.page_source, 'lxml')
reviews = soup.find_all('div', class_="d15Mdf bAhLNe")
for review in reviews:
    all_reviews = review.select_one('span[jsname="bN97Pc"]').get_text()
    print('all_reviews:' + all_reviews, sep="\n",end ="\n\n")


driver.close()

Output:

all_reviews:Since the latest update, on my Pixel2, Google search no longer works. It only shows my history - it no longer shows suggestions as I type, and when I'm finished typing and hit search, there's just a blank white screen - no error notification, no 
progress bar, nothing. I don't want to have to go to C...Full Review

all_reviews:Endlessly frustrating when I get alerts for news articles I'm interested in, but opening them from the alerts doesn't take me right to the article, and usually 
it's either nowhere to be found, or buried several pages into the newsfeed. I get it, 
you want me to endlessly scroll through everything. St...Full Review

all_reviews:It's hard enough using a cell phone when you're my age, 66 years old. Take that and then try to learn apps and that is an obstacle in itself. Now I'm suffering 
from a problem with Google app which I've always depended on. Recently when I go to search, the screen goes back to the home screen and a gr...Full Review

all_reviews:Bug in the recent update: I can no longer make Google searches unless I use the voice search. No error message or anything, the page just doesn't load at all. I would be happy to provide screenshots and further information!

Answer 3

If you print out soup instead of reviews, you will see that the HTML content you got is different from the HTML content on the live website.

Because you're not a browser, the script that create the content dynamically is not doing its job. See more detailed answer here: How can a scraped HTML be different from the source code?