7

I had multiple spiders in my project folder and want to run all the spiders at once, so i decided to run them using scrapyd service. I have started doing this by seeing here

First of all i am in current project folder

  1. I had opened the scrapy.cfg file and uncommented the url line after

    [deploy]

  2. I had run scrapy server command, that works fine and scrapyd server runs

  3. I tried this command scrapy deploy -l

    Result : default http://localhost:6800/

  4. when i tried this command scrapy deploy -L scrapyd i got following output

Result: 

Usage
=====
  scrapy deploy [options] [ [target] | -l | -L <target> ]

deploy: error: Unknown target: scrapyd

when i tried to deploy the project with this command scrapy deploy scrapyd -p default got following error

Usage
=====
  scrapy deploy [options] [ [target] | -l | -L <target> ]

deploy: error: Unknown target: scrapyd

I am really unable to identify whey scrapyd is showing the above errors, can lead me in to a correct way of how to deploy a project in to scrapyd

Thanks in advance..........

Edited Code:

After seeing the answer of Peter Kirby,i named target in scrapy.cfg and tried the following command in my project folder,

command:

scrapy deploy ebsite -p ebsite

then i got the below error

Building egg of ebsite-1341808241
'build/lib' does not exist -- can't clean it
'build/bdist.linux-x86_64' does not exist -- can't clean it
'build/scripts-2.7' does not exist -- can't clean it
zip_safe flag not set; analyzing archive contents...
Deploying ebsite-1341808241 to http://localhost:6800/addversion.json
Deploy failed: <urlopen error [Errno 111] Connection refused>

How to solve this.....

Shiva Krishna Bavandla
  • 25,548
  • 75
  • 193
  • 313

4 Answers4

10

From scrapyd service documentation: (http://scrapy.readthedocs.org/en/latest/topics/scrapyd.html?highlight=scrapyd)

You can define targets by adding them to your project’s scrapy.cfg file... Here’s an example of defining a new target scrapyd2 with restricted access through HTTP basic authentication:

[deploy:scrapyd2]
url = http://scrapyd.mydomain.com/api/scrapyd/
username = john
password = secret

Essentially what your error means is that your "target" name is not correct. If I remember correctly, the scrapy.cfg file sets the initial target name as "default". What you should be typing is something like: 

scrapy deploy default -p project_name
Peter Kirby
  • 1,915
  • 1
  • 16
  • 29
  • Thanks for u r reply, as u said i edited my code as above and i got the error mentioned above,can u tell me how to solve that – Shiva Krishna Bavandla Jul 09 '12 at 04:58
  • Try accessing the Scrapyd web interface. Go to http://localhost:6800 or type curl http://localhost:6800/listprojects.json and tell me what happens. It might be that Scrapyd is installed incorrectly. What OS are you on? – Peter Kirby Jul 10 '12 at 14:35
1

Just type scrapy deploy if you have no named targets and left settings at default!

Sjaak Trekhaak
  • 4,906
  • 30
  • 39
1

I got this error when I try to deploy my project without the scrapyd running, so simple run 

scrapyd

on another terminal fixed the error

shellbye
  • 4,620
  • 4
  • 32
  • 44
0

This is the scrapyd proc have no permission! You need kill the proc,then use root user,Just type:

scrapy server

then the new scrapyd will run.then you can do as scrapyd documention says.

gnemoug
  • 467
  • 1
  • 5
  • 21