Stack Overflow
Stack Overflow

Questions tagged [scrapy-shell]

The Scrapy shell is an interactive shell where you can try and debug your scraping code very quickly, without having to run the spider.

It’s meant to be used for testing data extraction code, but you can actually use it for testing any kind of code as it is also a regular Python shell.

177 questions
1
vote
1 answer

why i can not open this url on scrapy shell?

when i ran scrapy shell https://partsouq.com/en/catalog/genuine/unit?c=Toyota&ssd=%24HQwdcgcAAwFNa3Y_IS9WZm1SC1BrWUo%24&vid=4463&cid=&uid=2535&q= I get nothing. the curser blinks like in the screenshot how to scrap this page?
Hat hout
  • 471
  • 1
  • 9
  • 18
1
vote
1 answer

How to stop Multithreading or How to give request one by one in Scrapy?

I tried to crawl data of a product in the format. 1) ADD CART 2) VIEW CART 3) REMOVE CART For single color product it is working perfectly but for multi-color product, Scrapy takes multi-threading so above process is not in order for each and every…
Vimal Annamalai
  • 139
  • 1
  • 2
  • 12
1
vote
1 answer

How to get the line number of a match with scrapy

Using the following example: $ scrapy shell http://doc.scrapy.org/en/latest/_static/selectors-sample1.html where selectors-sample1-html is: Example website …
Javier Enciso
  • 55
  • 1
  • 2
  • 11
1
vote
0 answers

scrapy 301 redirect for amazon

I'm trying to follow along with scrapy tutorial but using amazon as an example as opposed to dmoz being used in their online tutorials. When i run the following command, I noticed I get a 301 redirect. scrapy shell…
noobie
  • 11
  • 2
1
vote
1 answer

How do I extract text with the Xpath in Scrapy?

Alright, it seems like I've tried everything and just can't figure out what is going on here. The direct link to the site I am trying to scrape is…
Mxracer888
  • 341
  • 1
  • 4
  • 14
1
vote
1 answer

Why this inconsistent behaviour using scrapy shell printing results?

Load the scrapy shell scrapy shell "http://www.worldfootball.net/all_matches/eng-premier-league-2015-2016/" Try a selector: response.xpath('(//table[@class="standard_tabelle"])[1]/tr[not(th)]') Note: it prints results. But now use that selector as…
willdanceforfun
  • 11,044
  • 31
  • 82
  • 122
1
vote
1 answer

list spiders from scrapy shell and run individual spider

I want to access scrapy shell from within "scrapy project folder" in terminal. And want to list all the available spiders in my project. I also like to run individual spider and play with the response. Once i enter into scrapy shell i get following…
rosnk
  • 1,068
  • 1
  • 14
  • 36
1
vote
1 answer

Scrapy bot and shell return different results with same xpath query. Why?

When I execute the same xpath query in a scrapy bot and in the scrapy shell, I'm getting different results. Note: I'm merely trying to learn scrapy and so modifying some of the tutorial code. Please go slowly with me. The…
ryan71
  • 131
  • 2
  • 8
1
vote
2 answers

Scrapy response.xpath not returning anything for a query

I am using the scrapy shell to extract some text data. Here are the commands i gave in the scrapy shell: >>> scrapy shell…
Abhishek
  • 2,998
  • 9
  • 42
  • 93
0
votes
1 answer

Cannot find html element using css or xpath selectors in Scrapy

I'm using Scrapy to scrape this website. I want to grab all the div elements with class="data1". I'm using css and xpath selectors to do so. However, I cannot find these elements using css and xpath selectors even though I can see them in the html…
Chris
  • 3
  • 1
0
votes
1 answer

Why is scrapy shell returning an empty list when my XPath selector works as it should in the “Elements” tab of my Chrome browser?

The XPath selector in Scrapy shell response.xpath('//div[@class="chr-lot-header__bid-details"]//span[@class="chr-lot-header__value-field"] returns an empty list while the same XPath selector selects the right html tag in the "Elements" tab of my…
user2821
  • 13
  • 2
0
votes
0 answers

Python scrapy with SplashRequest the response css got additional element which is not expected

Response css returns value which is not expected. I see ["Digital Content & Devices", "Shop By Department", "Programs & Features", "Help & Settings"] on Amazon's sidebar menu. But when I tried scrapy shell import scrapy_splash req =…
ShihLun
  • 21
  • 1
  • 4
0
votes
0 answers

Why is response.xpath('') not printing anything?

I opened a bash prompt using Cygwin on windows 10, I then ran scrapy shell then it looked different before it did not have "in 1:"so I am not sure what changed, but then I tried to get a response using XPath, nothing printed and it just went to a…
Ryan Njonjo
  • 21
  • 2
0
votes
2 answers

Scrape the feature image from this website but it returns this `data:image/gif

Using Scrapy and Scrapy shell in python to scrape the feature image from this website https://www.thrillist.com/travel/nation/all-the-ways-to-cool-off-in-austin but it returns this data:image/gif;base64,R0 instead of src of the image, I need the…
Info Rewind
  • 145
  • 7
0
votes
2 answers

Why do I get a empty list in scrapy when I use response.css

The code I used was import scrapy class JobSpider(scrapy.Spider): name = 'job' start_urls = [ 'https://jobs.goodlifefitness.com/listjobs/' ] In the scrapy shell I put…
codingLearner99
  • 31
  • 2
1 2 3
…
11 12