0

I want to scrape the name of products in this web page, but i get an empty list. I used splash for dynamic webpage but the result is the same. Can someone tell me what to do ? any other solution ?

This is the webpage that i want to scrape : https://www.woolworths.com.au/shop/browse/drinks/cordials-juices-iced-teas/iced-teas

import scrapy

from shop.items import ShopItem

from scrapy_splash import SplashRequest


class Spider(scrapy.Spider) :

      name = 'productsspider'
      start_urls = ['https://www.woolworths.com.au/shop/browse/drinks/cordials-juices-iced-teas/iced-teas']

      def start_requests(self):
          for url in self.start_urls:
          yield SplashRequest(url=url, callback=self.parse, args={"wait":3})

      def parse(self, response):
          item = ShopItem()
          for p in response.css(".divng-tns-c32-15 product-grid.ng-trigger.ng-trigger-staggerFadeInOut"):
              item = productsItem()
              item["product_name"] = p.css(".ng-tns-c32-15.product-grid--tile.ng-star-inserted .ng-tns-c32-15.product-grid--tile.ng-star-inserted shared-product-tile section .shelfProductTile-content div header a::text").extract()
              yield item
DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
Devpy
  • 1
  • 4
  • Does the page look correct when you use the [view](https://docs.scrapy.org/en/latest/topics/commands.html#view) command or when rendering in Splash? Maybe the JavaScript isn't fully loaded or the `User-Agent` needs to be set like a real browser. – notorious.no Nov 17 '20 at 20:43
  • when a use view or splash, i see an empty white page. Do you know any solution that i can use to get the data or the whole data of the page ? – Devpy Nov 18 '20 at 22:12

0 Answers0