0

I am unable to scrape multiple pages, but I am able to do it for single page.

    from urllib.request import urlopen
    from bs4 import BeautifulSoup

    file = "pyp.csv"
    f = open(file, "w")
    Headers = "product, description, img_url, price, amzn_link\n"
    f.write(Headers)
    for page in range(1,5):
    url = "https://www.homerungifts.com/gift-ideas-for-mom/page/{}/".format(page)
    html = urlopen(url)
    soup = BeautifulSoup(html,"html.parser")
    containers = soup.findAll("article", {"class":"repick_item small_post col_item inf_scr_item contain_im_grid"})

    for container in containers:

        for i in containers:
            try:
                product = i.container.div.div.a.text
                description = i.container.div.div.p.text
                img_url = i.container.a.img['data-src']
                price = i.container.div.span.span.ins.text
                amzn_link = i.container.findAll("a", {"class": "btn_offer_block re_track_btn"})[0]['href']
                print("product: " + product + "\n")
                print("description: " + str(description) + "\n")
                print("img_url: " + str(img_url) +"\n")
                print("price: " + str(price) + "\n")
                print("amzn_link: " + str(amzn_link) + "\n")
                f.write("{}".format(product) + ",{}".format(description).replace(",", " ")+ ",{}".format(img_url) + ",{}".format(price) + ",{}".format(amzn_link) + "\n")
            except: AttributeError
f.close()

output

F:\aaa\2>python ppp.py

F:\aaa\2>python ppp.py

F:\aaa\2>python ppp.py

F:\aaa\2>
scrappedcola
  • 10,423
  • 1
  • 32
  • 43
Kiran JC
  • 57
  • 1
  • 8
  • 4
    Debugging tip: if your script is ending without producing any errors or output, then remove any `try-except`s you have that silently discard error messages. Then when the program crashes, it will give you valuable diagnostic information. – Kevin Nov 28 '17 at 14:42
  • What is your desired output? – ashleedawg Nov 28 '17 at 14:43
  • i would like to extract the content like image url, product name etc from same category from different pages where as iam able to do it for single page . but i would like to loop the process – Kiran JC Nov 28 '17 at 14:50
  • Your `except` handles *all* exceptions by doing *nothing*. The `AttributeError` in that line is *very* confusing and has absolutely no effect whatsoever. – BlackJack Nov 29 '17 at 16:38
  • thank you Mr. BalckJack I got the output... – Kiran JC Nov 29 '17 at 17:11

0 Answers0