Stack Overflow
Stack Overflow

Questions tagged [scrapy-selenium]

29 questions
2
votes
0 answers

Scrapy-Selenium Middleware:Failed to scrape data from website

I am facing an issue here. I am trying to execute this code using scrapy-selenium but this is scraping nothing. What actually I am missing here. If it is because of 'page_source' then how to apply 'page_source' properly? Here is my code_ import…
Raisul Islam
  • 277
  • 2
  • 19
1
vote
0 answers

Scrapy CrawlProcess crawl() function sometimes hangs indefinitely

I'm using CrawlPrcoess to run a scrapy spider with a selenium downloader. Most of the time the code works as expected but sometimes it hangs indefinitely after executing the crawl() function. process =…
Arman Avetisyan
  • 349
  • 2
  • 10
1
vote
0 answers

taking a screenshot with scrapy-selenium

been trying to get scrapy-selenium to work for a while, I have the simple code from the tutorial and just trying to take a screenshot but for some reason I don't get a proper response, here is my code: from shutil import which SELENIUM_DRIVER_NAME…
Yaniv Assaf
  • 329
  • 1
  • 4
  • 6
0
votes
0 answers

scrapy_selenium scroll multiple time in web page

im trying to scroll more than once in same instance of webdriver using scrapy selenium . as sending another SeleniumRequest will refresh the page. here is my code : class QuotesSpider(scrapy.Spider): name = 'scss' def start_requests(self): url…
Low LiFe
  • 15
  • 2
0
votes
1 answer

how to write xpath for this

I writing a script to scrap data from a website "https://pfchangsmexico.com.mx/ubicaciones/" in which I want latitude longitude values of each restaurants enter image description here in the image I have highlighted the latitude and longitude values…
Prathamesh
  • 1
  • 1
0
votes
1 answer

Scrapy-selenium error: TypeError: WebDriver.__init__() got an unexpected keyword argument 'executable_path'

I am trying to set up scrapy-selenium to do some scraping: pip installed scrappy, scrapy-selenium; downloaded and put to my project directory chromedriver.exe, updated the setting.py: from shutil import which SELENIUM_DRIVER_NAME =…
densol96
  • 49
  • 3
0
votes
0 answers

ScrapyRequest with Rule LinkExtractor stops crawling after first page

I was trying to extract all urls related to my test domain. The designed page is Javascript page and it requires selenium to crawl through all urls corresponding to this domain. But the crawler stops after crawling one page. I need to collect all…
Hans M
  • 11
  • 1
0
votes
1 answer

Error in scraping an ecommerce website daraz.pk

I am trying to scrape daraz.pk and ran into this error .The spider scrapes all the values on the page until the last value because it returns None value and then the spider throws an NoneType object is not iterable . I have tried using exception…
Zain Asif
  • 9
  • 2
0
votes
0 answers

How do I load a XML Page thru Scrapy without getting 502 (Bad Gateway Error) with a Proxy

Hi I am seeking some help after going back and forth trying to figure this out. Summary: I wish to open up a URL and subsequently open the get request which turns out to be a XML like HTML Content. I need to scrape that whole response.body Sample:…
Suren Gunaseelan
  • 13
  • 3
0
votes
0 answers

scrapy_selenium, how to pass driver to parse

I'm running this code using scrapy_selenium but I'm not able to pass the driver to parse_page, can anyone identify what I'm doing wrong? class LSpider(scrapy.Spider): name = 'test' def start_requests(self): yield SeleniumRequest( …
Giovanna Palú
  • 1
  • 1
0
votes
0 answers

Scrapy: stale element reference: element is not attached to the page document

I'm trying to run this code using Scrapy-Selenium. Whenever I'm trying to execute this, selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document error is occurring. I…
Raisul Islam
  • 277
  • 2
  • 19
0
votes
1 answer

Spider closes without error messages and does not scrape all the pages in the pagination

I have created a pipeline to place all the data scraped into a SQLite database but my spider is not completing the pagination. This is what I get when the spider closes. I should get around 45k results and I am only getting 420. Why could this…
cesalomx
  • 25
  • 7
0
votes
2 answers

How to crawl Page with from a tab that needs to be clicked with scrapy-selenium

So I want to scrape data from this site, especially from the company details part: Site to crawl I got some help from a person to get it work with python playwright but I need to get this done with the python scrapy-selenium. I want to rewrite the…
TheGoldBerg
  • 21
  • 3
0
votes
1 answer

How to scrape data via scrapy python correctly from a dynamically(?) created table

I am currently trying to crawl the the Company Overview from alibaba.com. For instance: https://www.alibaba.com/product-detail/T14-series-original-air-pro-TWS_1600273931389.html?spm=a2700.galleryofferlist.normal_offer.d_title.4aa778f2ahtuBx&s=p For…
TheGoldBerg
  • 21
  • 3
0
votes
1 answer

Scrapy-Selenium Pagination

Can anyone help me? I'm practicing and I can't understand what I did wrong on pagination! It only returns the first page to me and sometimes an error comes up. When it works, it just returns the first page. "The source list for the Content Security…
Lucas Guidi
  • 65
  • 1
  • 10
1
2