0

I am trying to write my first program, which will web scrape from multiple websites. I want to consolidate prices for different times of products from different URLs. However, I am receiving an error. Below is my code:

#imports
import pandas as pd
import requests
from bs4 import BeautifulSoup

#Product Websites For Consolidation
url = ['https://www.aeroprecisionusa.com/ar15/lower-receivers/stripped-lowers?product_list_limit=all', 'https://www.aeroprecisionusa.com/ar15/lower-receivers/complete-lowers?product_list_limit=all']
for website in url:
    headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:71.0) Gecko/20100101 Firefox/71.0"}
    page = requests.get(website, headers=headers)
    soup = BeautifulSoup(page.content, 'html.parser')


#Locating All Products On Page
all_products_on_page = soup.find(class_='products wrapper container grid products-grid')
individual_items = all_products_on_page.find_all(class_='product-item-info')


#Breaking Down Product By Name And Price
aero_product_name = [individual_items.find(class_='product-item-link').text for individual_items in individual_items]
aero_product_price = [individual_items.find(class_='price').text for individual_items in individual_items]



Aero_Stripped_Lowers_Consolidated = pd.DataFrame(
    {'Aero Stripped Lower': aero_product_name,
     'Prices': aero_product_price,
     })

Aero_Stripped_Lowers_Consolidated.to_csv('MasterPriceTracker.csv')

Below is my error. It appears that Python has an issue with the ".text". However, Python was able to run it like this when I was web scraping only one URL

Traceback (most recent call last):
  File "C:/Users/The Rossatron/Documents/PyCharm_Projects/Aero Stripped Lower List/Master_Price_Tracker.py", line 16, in <module>
    aero_product_price = [individual_items.find(class_='price').text for individual_items in individual_items]
  File "C:/Users/The Rossatron/Documents/PyCharm_Projects/Aero Stripped Lower List/Master_Price_Tracker.py", line 16, in <listcomp>
    aero_product_price = [individual_items.find(class_='price').text for individual_items in individual_items]
AttributeError: 'NoneType' object has no attribute 'text'

I was wondering if anyone would mind helping me out for this. It seems like a simple mistake but I've been troubleshooting it for several hours.

Thanks in advance!

PythonLearner23
  • 45
  • 3
  • Maybe its the same problem like here: [link](https://stackoverflow.com/questions/56890528/nonetype-object-has-no-attribute-decode/56890978#56890978) – Lukas Nothhelfer Jan 09 '20 at 11:36
  • it's because the site doesn't contain the `individual_items.find(class_='price')`. It's returning None, and you can't use `.text` is there is nothing there. The page is likely dynamic, meaning that will show up if you look at the source html, but it's rendered after you open the page. Using requests won't render that. – chitown88 Jan 09 '20 at 11:38
  • I'm not sure, but I think there's a problem in the list comprehension: `[individual_items.find(class_='price').text for individual_items in individual_items]`. Try replacing it with `[individual_item.find(class_='price').text for individual_item in individual_items]`, so that the element name is not the same as the list variable name – Itamar Mushkin Jan 09 '20 at 11:38
  • `product-item-link` or `price` mustn’t exist. Why are you querying `individual_items` instead of a dingle item? – Peter Wood Jan 09 '20 at 11:40
  • as @ItamarMushkin, your list comprehensions are incorrect. change both to: – chitown88 Jan 09 '20 at 13:06
  • 1
    `aero_product_name = [item.find(class_='product-item-link').text for item in individual_items]` – chitown88 Jan 09 '20 at 13:06
  • 1
    `aero_product_price = [item.find(class_='price').text for item in individual_items]` – chitown88 Jan 09 '20 at 13:07

0 Answers0