2

Could you please help me figure out what I'm doing wrong ? Here are the steps:

  • followed the portia install manual found here https://github.com/scrapinghub/portia - all ok
  • created a new project, entered an url, tagged an item - all ok
  • clicked "continue browsing", browsed through site, items were being extracted as expected - all ok

Next I wanted to deploy my spider:

1st try : I tried to run, as the docs specified, scrapyd-deploy your_scrapyd_target -p project_name - got error - scrapyd wasn't installed fix: pip install scrapyd

2nd try : I launched scrapyd server, accessed http://localhost:6800/ -all ok

After a brief reading of scrapyd docs I found out I had to edit the file scrapy.cfg from my project : slyd/data/projects/new_project/scrapy.cfg added the following : 

[deploy:local]
url = http://localhost:6800/

Went back to the console, checked all is ok :

$:> scrapyd-deploy -l
local http://localhost:6800/

$:> scrapyd-deploy -L local
default

Seemed ok so i gave it another try :

$scrapyd-deploy local -p default
Packing version 1418722113
Deploying to project "default" in http://localhost:6800/addversion.json
Server response (200):
{"status": "error", "message": "IOError: [Errno 21] Is a directory: '/Users/Mike/www/portia/slyd/data/projects/new_project'"}

What am I missing ?

Mihai
  • 133
  • 1
  • 14
  • When executing `scrapyd-deploy`, you have to be positioned in the right directory, the project directory. Where are you located at that time? – bosnjak Dec 16 '14 at 11:56
  • I'm in the right spot. I opened an issue on github also with this problem : https://github.com/scrapinghub/portia/issues/128 – Mihai Dec 16 '14 at 12:26
  • Any suggestions are more than welcome... – Mihai Dec 16 '14 at 12:51
  • Check for caveats in the code, as described here: http://scrapyd.readthedocs.org/en/latest/deploy.html#egg-caveats Since its autogenerated, these shouldn't be an issue, but still.. – bosnjak Dec 16 '14 at 12:58
  • Can you paste the contents of `/var/log/scrapyd/scrapyd.log`? And `/var/log/scrapyd/scrapyd.out` if exists. Also `/var/log/scrapyd/scrapyd.err`. – bosnjak Dec 16 '14 at 13:07
  • Please set `debug = on` in your configuration file, under `[scrapy]`, and then try to deploy. Paste the debug output after that. – bosnjak Dec 16 '14 at 13:15
  • Some clues where exactly i should put that ? Again, I am trying to build a project built with Portia, from within the project folder i am trying to scrapyd-deploy and i'm getting that error. – Mihai Dec 16 '14 at 16:10
  • I am running in a virtual environment, so no system wide installs or anything... I added a "node" called `[scrapy]` in scrapy.cfg and below added `debug = on`. Tried to build, got the error, but no log was generated, and the error wasn't more explicit. tryed adding the same `debug = on` under the `[settings]` node of the same file .. same results. – Mihai Dec 16 '14 at 16:12

1 Answers1

0

For anyone who stumbles upon this issue, the fix is to deploy scrapyd in another directory other than that of the project. See details here : https://github.com/scrapinghub/portia/issues/128

Mihai
  • 133
  • 1
  • 14