I've successfully scrapped a multi page library using “scrapy”. But when I added a yield block only the first web page was scrapped. The page count continued to increase but the web page didn't change What is wrong? Here is the structure of part of the program.
enter code here
.current page = 1 while current_ page < =- last_page product_container = #the container of all the books to be scraped from the current web page
enter code here
for product in product_container:
get:
book title,
authors into authorDic,
narrators into narratorDic
yield{
title': book_title,
'author': authorsDic,
'narrantor': narratorsDic,
}
current_page +=1
get next_ page
next_page.clickl()
'''
