Stack Overflow
Stack Overflow

Questions tagged [scrapy-shell]

The Scrapy shell is an interactive shell where you can try and debug your scraping code very quickly, without having to run the spider.

It’s meant to be used for testing data extraction code, but you can actually use it for testing any kind of code as it is also a regular Python shell.

177 questions
2
votes
2 answers

Scrapy shell with playwright

Is it possible to invoke Playwright in a Scrapy shell? I would like to use a shell to test my xpaths, which I intend to place in a spider that incorporates Scrapy Playwright. My scrapy settings file has the usual Playwright setup: # Scrapy…
Clemence Padya
  • 63
  • 6
2
votes
3 answers

Why am I getting Error 429 from a specific website on fetch() using Scrapy?

I'm very new to web scraping and and as a first project(in order to learn) I wanted to create a database for house prices. Later on I'm going to feed it to ML algorithms to see if I'm going to be able to predict the prices but I cannot fetch the…
Aras Uludağ
  • 113
  • 2
  • 7
2
votes
3 answers

How to use python requests with scrapy?

I am trying to use requests to fetch a page then pass the response object to a parser, but I ran into a problem: def start_requests(self): yield self.parse(requests.get(url)) def parse(self, response): #pass builtins.AttributeError:…
max will
  • 31
  • 1
  • 2
2
votes
1 answer

Change default browser while using scrapy's shell view function

While on a scrapy shell, when I try to use the view(response) function, instead of opening a browser, it opens Visual Studio Code. How to make it open on a browser? I read that webbrowser is the library used to view the page and that I could set the…
FTM
  • 1,887
  • 17
  • 34
2
votes
0 answers

How to use Scrapy Shell in WinPython and by using IPython?

I would like to learn scrapy and I can use scrapy shell in the regular Windows shell. As soon as I type scrapy shell 'url' in IPython i get the subsequent message: scrapy shell http://quotes.toscrape.com/ File "",…
Christian
  • 515
  • 1
  • 6
  • 17
2
votes
0 answers

Scrapy Shell has the Correct Output, but the script does not

So I'm very confused here. When I use the scrapy shell and input the xpath the correct data is returned, but when I set that same xpath equal to a variable within the script, it outputs a blank. I'm really not sure what is going on. import…
a1letterword
  • 307
  • 1
  • 4
  • 16
2
votes
4 answers

What should i do to enable cookies and use scrapy for this url?

I am using scrapy for a scrapying project with this url https://www.walmart.ca/en/clothing-shoes-accessories/men/mens-tops/N-2566+11 I tried to play with the url and open it in the shell, but it got 430 error, so i added some settings to the header…
Hat hout
  • 471
  • 1
  • 9
  • 18
2
votes
1 answer

Scrapy Shell: twisted.internet.error.ConnectionLost although USER_AGENT is set

When I try to scrape a certain web site (with both, spider and shell), I get the following error: twisted.web._newclient.ResponseNeverReceived: [
areiterer
  • 33
  • 1
  • 7
2
votes
2 answers

scrapy don't detect an html element but it is visible on source page

I have a request working normally on regular browsers but not on in scrapy shell. An entire HTML block get vanish as soon as I use "scrapy shell" or "scrapy crawl". I am not banned for sure. Here, below, is the issue on the github (with pictures)…
M. Mayouf
  • 31
  • 1
  • 3
2
votes
1 answer

Scrapy FormRequest , trying to send a post request (FormRequest) with currency change formdata

I've been trying to scrapy the following Website but with the currency changed to 'SAR' from the upper left settings form , i tried sending a scrapy request like this: r = Request(url='https://www.mooda.com/en/', cookies=[{'name': 'currency', …
M.nabil.H
  • 23
  • 4
2
votes
2 answers

AttributeError: 'module' object has no attribute 'DATABASE' when using scrapy shell

I am trying to run the scrapy shell in the root of my project, but I keep getting an obscure error regarding some sort of DATABASE setting. I'm not sure if this is an SQLAlchemy thing...or some problem with my schema definition? If I run scrapy…
tumultous_rooster
  • 12,150
  • 32
  • 92
  • 149
2
votes
1 answer

Scrapy Amazon absolute path but with no value returned

I am trying to use Scrapy some info from amazon.co.uk by using absolute path as below. but strangely no value returned. I am quite new to Scrapy: scrapy shell…
W.S.
  • 647
  • 1
  • 6
  • 19
2
votes
1 answer

scrapy xpath select elements by classname

I have followed How can I find an element by CSS class with XPath? which gives the selector to use for selecting elements by classname. The problem is when I use it it retrieves an empty result "[]" and I know by fact there is a div classed…
secuaz
  • 459
  • 3
  • 6
  • 15
2
votes
0 answers

scrapy keyError: z (freebsd)

I am trying to install scrapy 0.24 in freebsd (MariaDB) system but when I try to run it I have an "keyError: 'z'" which I don't know what it means... I tried to debug it with no success. File "/usr/local/bin/scrapy", line 9, in…
Manu
  • 21
  • 3
2
votes
2 answers

Scrapy Shell XPath

I am trying to get links and category from this http://www.npr.org/rss/#feeds news feed website. This is my xpath in scrapy shell: a = sel.xpath('//ul[@class="rsslinks"]/li/a/@href').extract() b =…
m0rpheu5
  • 600
  • 4
  • 16
1
2
3
…
11 12