Product description is not returning while scraping

Question

I tried to scrape product description from below url. But it is not returning that

https://www.mambo.com.br/arroz-integral-camil-1kg/p

My code below not return description text:

myurl = "https://www.mambo.com.br/arroz-integral-camil-1kg/p"    
agent = {'User-Agent': 'Magic Browser'}
req1 = requests.get(myurl, headers=agent)
soup2 = BeautifulSoup(req1.content, "html.parser")
for desc in soup2.findAll('div', {"class": "accordion__body ProductDescription"}):
    print(desc.text)

Please fix and help in code the issue.

Possible duplicate of [How to get multiple class in one query using Beautiful Soup](https://stackoverflow.com/questions/13572676/how-to-get-multiple-class-in-one-query-using-beautiful-soup) — GalAbra, Jul 29 '18 at 06:32
No. There is only on class as {"class": "accordion__body ProductDescription"}) But it is not returning the text — Saranaone, Jul 29 '18 at 06:41
When a class "name" contains a space, it indicates the element actually [has two classes](https://stackoverflow.com/questions/8722163/how-to-assign-multiple-classes-to-an-html-container) — GalAbra, Jul 29 '18 at 06:42

Andrej Kesely · Accepted Answer · 2018-07-29T06:51:40.183

The data is loaded dynamically through Ajax - the page itself doesn't contain any data.

You need to extract SKU (product number) from the main page and then call API located at https://www.mambo.com.br/api/ for JSON data (you can see all requests that the page is doing in Firefox/Chrome network inspector):

from bs4 import BeautifulSoup
import requests
import json

product_url = "https://www.mambo.com.br/api/catalog_system/pub/products/search/?fq=productId:{}"
url = "https://www.mambo.com.br/arroz-integral-camil-1kg/p"

r = requests.get(url)
soup = BeautifulSoup(r.text, 'lxml')
sku = soup.select_one('meta[itemprop="sku"]')['content']
data_json = json.loads(requests.get(product_url.format(sku)).text)

for p in data_json:
    print(p['description'])
# print(json.dumps(data_json, indent=4)) # this will print all data about the product

Output:

O arroz integral faz parte da linha de produtos naturais da Camil. É saudável, prático e gostoso. Melhor que saborear um prato delicioso é fazer isso com saúde!

EDIT:

Alternatively you can get description from <meta itemprop="description">, but I'm not sure if the description is complete in this tag:

url = "https://www.mambo.com.br/arroz-integral-camil-1kg/p"

r = requests.get(url)
soup = BeautifulSoup(r.text, 'lxml')
print(soup.select_one('meta[itemprop="description"]')['content'])

Prints:

O arroz integral faz parte da linha de produtos naturais da Camil. É saudável, prático e gostoso. Melhor que saborear um prato delicioso é fazer isso com saúde!

Fantastic!.. It worked like a charm. Thanks a ton!! – Saranaone Jul 29 '18 at 06:59 — Saranaone, Jul 29 '18 at 06:59

Product description is not returning while scraping

1 Answers1