AttributeError: 'NoneType' object has no attribute 'text' (USING requests AND BeautifulSoup)

Question

`import requests
from bs4 import BeautifulSoup

class Temperature:
    def temperature(place):
        placeTemp = []
        placeWeather = "Weather in", place
        url = f"https://www.google.com/search?q={placeWeather}"
        r = requests.get(url)
        soup = BeautifulSoup(r.text, "html.parser")
        placeTemp.append(soup.find("div", class_="wob_t q8U8x").text)
        print(placeWeather,"is",placeTemp)

t = Temperature.temperature("Delhi")
print(t)`

I am making a program that finds the temperature of a place using requests and BeautifulSoup. Whenever I run the code I get this error:

placeTemp.append(soup.find("div", class_="wob_t q8U8x").text)
AttributeError: 'NoneType' object has no attribute 'text'

I've tried changing the class_ part but it always gives the same error.

`placeWeather = "Weather in", place` that's a tuple, not a string — Fractalism, Jan 29 '23 at 15:45
The tag you searched for is not in the source code. Google loads it with JavaScript — Jurakin, Jan 29 '23 at 15:49

score 0 · Answer 1 · answered Jan 29 '23 at 16:06

It is a bit tricky and you always should take a look into your soup first.

To avoid consent redirect, send cookies:

cookies = {'CONSENT':"YES+"}
To get your values, select your elements more specific:

list(soup.select_one('div:has( > span:has(>:-soup-contains("Weather"))) ~ div > div').stripped_strings)

import requests

from bs4 import BeautifulSoup


params = {'q':'weather in Delhi', 'hl': 'en'}
headers = {'User-Agent': 'Mozilla/5.0'}
cookies = {'CONSENT':"YES+"}

url = 'https://www.google.com/search'


soup = BeautifulSoup(requests.get(url, params=params, headers=headers, cookies=cookies).content)

list(soup.select_one('div:has( > span:has(>:-soup-contains("Weather"))) ~ div > div').stripped_strings)

Output

Simply slect value by index:

['14°C', 'Sunday 21:32\nCloudy', 'More on weather.com']

Jurakin · Answer 2 · 2023-02-03T15:00:13.830

It is possible to get the temperature by only using beautiful soup. Google uses https://weather.com to show weather. You can use it, to get temperature:

import requests
from bs4 import BeautifulSoup

class Temperature:
    def temperature(place):
        session = requests.session()
        session.headers.update({"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"})

        placeTemp = []
        
        placeWeather = f"Weather in {place}"
        url = f"https://www.google.com/search?q={placeWeather}"

        page = session.get(url)
        page.raise_for_status()
        soup = BeautifulSoup(page.content, "html.parser")

        url = soup.find("div", class_="YfftMc").find("a", href=True).get("href")
        
        page = session.get(url)
        page.raise_for_status()
        soup = BeautifulSoup(page.content, "html.parser")
        
        placeTemp.append(soup.find("span", {"data-testid": "TemperatureValue"}).text)
        print(placeWeather,"is",placeTemp)

        return placeTemp

t = Temperature.temperature("Delhi")
print(t)

Outputs:

Weather in Delhi is ['14°']
['14°']

score 0 · Answer 3 · answered Jan 29 '23 at 16:11

Try using selenium instead. I am not sure but Google loads the data using JavaScript so, BeautifulSoup will not work.

Here is an example code for you :-

from selenium import webdriver
from selenium.webdriver.common.by import By

class Temperature:
    def temperature(place):
        placeWeather = f"Weather in {place}"
        url = f"https://www.google.com/search?q={placeWeather}"
        driver = webdriver.Firefox()
        driver.get(url)
        placeTemp = driver.find_element(By.ID,"wob_tm").text

        print(placeWeather,"is",placeTemp)
        return placeTemp

t = Temperature.temperature("Delhi")

output:-

Weather in Delhi is 57

*BeautifulSoup will not work* quite a sweeping statement, but there are two examples that refute this statement. As long as the information is contained in the response, nothing stands in the way. With `selenium` you only make it easier in that sense, since the `page_source` corresponds to the one from the devTools. — HedgeHog, Jan 29 '23 at 18:34

AttributeError: 'NoneType' object has no attribute 'text' (USING requests AND BeautifulSoup)

3 Answers3

import requests

Output

Linked