Questions tagged [scrapy-selenium]
29 questions
0
votes
2 answers
Object of type Selector is not JSON serializable
I'm trying to scrape a dynamic website and I need Selenium.
The links that I want to scrape only open if I click on that specific element. They are being opened by jQuery, so my only option is to click on them because there is no href attribute or…
Jonathan Simpson
- 15
- 3
0
votes
0 answers
unable to run scrapy-selenium library with UI (with head)
I was using scrapy_selenium but I don't know how to run it with UI I followed this answer, but I am getting following error:
Traceback (most recent call last):
File "D:\Anaconda\envs\web scrapping\lib\site-packages\scrapy\crawler.py", line 192, in…
Iswar Chand
- 91
- 7
0
votes
0 answers
I am trying scrapy crawling spider using the general spider template with selenium , but when crawling to the product details page it give wrong data
Blockquote
I am trying to extract the href link and then i joined the url and then i give the seleniumRequest with the joined url
my code works and it crawls the data but at the result it produce the same repetative data
the data that are crawled…
0
votes
1 answer
Keep browser window open after scraping?
When using scrapy-selenium, how can one keep the browser window open after scraping has finished (or aborted)?
stefan.at.kotlin
- 15,347
- 38
- 147
- 270
-1
votes
1 answer
selenium script written in _init_ function not executing
I am trying to integrate selenium with scrapy to render javascript from a website. I have put the selenium automation code in a constructor, it performs a button click, and then the parse function scrapes the data from the page. But follwing errors…
vasu gupta
- 1
- 1
-1
votes
1 answer
How to crawl data from the new tab opening
I'm trying to crawl the detail of product of this webpage https://www.goo-net.com/php/search/summary.php by scrapy-selenium.
Because I want to crawl the detail information of each product, I crawled all url of product from the page. Then I use…
Đạt Trầm
- 11
-1
votes
1 answer
Scrapy-Selenium: Chrome Driver does not load page
I have two projects, one with Selenium and one using Scrapy-Selenium, which fits into a Scrapy spider program format but uses Selenium for automation.
I can get the Chromedriver to load the page I want for the basic Selenium program, but something…
Sun R
- 1
- 5
-1
votes
1 answer
Why Scrapy selenium providing the 'Null' and duplicate value at the same time?
I am trying to execute this script but I don't know why it is throwing 'Null' and duplicate value at the same time! My goal is to put the necessary value and click the search button, get all the 'href' from the page and collect the data, which is…
Raisul Islam
- 277
- 2
- 19
-1
votes
1 answer
Scrapy Selenium: Why pagination is not working for scrapy-selenium?
I am trying to get data using scrapy-selenium but there is some issue with the pagination. I have tried my level best to use different selectors and methods but nothing changes. It can only able to scrape the 1st page. I have also checked the other…
Raisul Islam
- 277
- 2
- 19
-1
votes
1 answer
Scrapy / Use Scrapy Selenium for the first request-page?
i have a running solution using scrapy_selenium for a site with javascript-loading.
As you can see in the code below the the SeleniumRequest is used when yielding the detailPage with parseDetails -
But what can i do when i need the SeleniumRequest…
Rapid1898
- 895
- 1
- 10
- 32
-1
votes
1 answer
Crawled 0 pages, scraped 0 items ERROR / webscraping / SELENIUM
So I've tried several things to understand why my spider is failing, but haven't suceeded. I've been stuck for days now and can't afford to keep putting this off any longer. I just want to scrape the very first page, not doing pagination at this…
cesalomx
- 25
- 7
-1
votes
1 answer
My Scrapy code is either filtering too much or scraping the same thing repeatedly
I am trying to get scrapy-selenium to navigate a url while picking some data along the way. Problem is that it seems to be filtering out too much data. I am confident there is not that much data in there. My problem is I do not know where to apply…
Reggie18
- 25
- 6
-1
votes
1 answer
How can I scrape a button that does not return a value
I am trying to scrape from the website https://tonaton.com/en/ads/ghana/electronics. There is a "next" button that I want to click and scrape the contents. The problem is the xpath or css selector of that button does not return any value in neither…
Danny Stringz
- 9
- 3
-2
votes
1 answer
How to Load more/show more pagination with scrapy-selenium
Getting response but scraping nothing!
import scrapy
from scrapy.selector import Selector
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
from time import sleep
class ProductSpider(scrapy.Spider):
…
Md. Fazlul Hoque
- 15,806
- 5
- 12
- 32