Can't deploy scrapy to scrapyd server

Question

I'm trying to deploy spider created via Portia. Portia, scrapyd - all is latest versions.

I'm running scrapyd server just by command:

scrapyd

I'm getting this result on my machine locally:

$> cd PROJECT_PATH_HERE;
$> scrapyd-deploy
Packing version 1463420166
Deploying to project "retail" in http://X.X.X.X:6800/addversion.json
Server response (200):
{"status": "error", "message": "AttributeError: 'list' object has no attribute 'iteritems'", "node_name": "vaua0048313.online-vm.com"}

On the server side (Ubuntu 12.04.5 LTS):

root@server:~# scrapyd
2016-05-16 17:35:42+0000 [-] Log opened.
2016-05-16 17:35:42+0000 [-] twistd 16.1.1 (/usr/bin/python 2.7.3) starting up.
2016-05-16 17:35:42+0000 [-] reactor class: twisted.internet.epollreactor.EPollReactor.
2016-05-16 17:35:42+0000 [-] Site starting on 6800
2016-05-16 17:35:42+0000 [-] Starting factory <twisted.web.server.Site instance at 0x2f5d950>
2016-05-16 17:35:42+0000 [Launcher] Scrapyd 1.1.0 started: max_proc=12, runner='scrapyd.runner'
2016-05-16 17:36:06+0000 [_GenericHTTPChannelProtocol,0,78.111.181.96] Unhandled Error
        Traceback (most recent call last):
          File "/usr/local/lib/python2.7/dist-packages/twisted/web/http.py", line 1767, in allContentReceived
            req.requestReceived(command, path, version)
          File "/usr/local/lib/python2.7/dist-packages/twisted/web/http.py", line 768, in requestReceived
            self.process()
          File "/usr/local/lib/python2.7/dist-packages/twisted/web/server.py", line 183, in process
            self.render(resrc)
          File "/usr/local/lib/python2.7/dist-packages/twisted/web/server.py", line 234, in render
            body = resrc.render(self)
        --- <exception caught here> ---
          File "/usr/local/lib/python2.7/dist-packages/scrapyd/webservice.py", line 17, in render
            return JsonResource.render(self, txrequest)
          File "/usr/local/lib/python2.7/dist-packages/scrapyd/utils.py", line 19, in render
            r = resource.Resource.render(self, txrequest)
          File "/usr/local/lib/python2.7/dist-packages/twisted/web/resource.py", line 250, in render
            return m(request)
          File "/usr/local/lib/python2.7/dist-packages/scrapyd/webservice.py", line 79, in render_POST
            spiders = get_spider_list(project, version=version)
          File "/usr/local/lib/python2.7/dist-packages/scrapyd/utils.py", line 116, in get_spider_list
            raise RuntimeError(msg.splitlines()[-1])
        exceptions.RuntimeError: AttributeError: 'list' object has no attribute 'iteritems'

2016-05-16 17:36:06+0000 [-] "78.111.181.96" - - [16/May/2016:17:36:05 +0000] "POST /addversion.json HTTP/1.1" 200 135 "-" "Python-urllib/2.7"

What am I doing wrong ?

Can you post your slybot, scrapy and scrapyd version so we can try to recreate it? — Ruairi Fahy, May 17 '16 at 08:55
@RuairiFahy slybot==0.13.0b18, Scrapy==1.0.0, scrapyd-client==1.0.1 — artyomboyko, May 17 '16 at 10:20

score 0 · Answer 1 · answered Jul 05 '16 at 07:13

0

i think the problem comes from your setting.py. The ITEM_PIPELINES should be specified by the dict format below, when you are using scrapy version of 1.10.0,

ITEM_PIPELINES = {
    'coolscrapy.pipelines.ArticleDataBasePipeline': 5,
}

answered Jul 05 '16 at 07:13

xyqzki

1

score 0 · Answer 2 · answered Jul 22 '16 at 08:21

0

Modify ITEM_PIPELINES in Portia_Projects/spiders/setting.py to

ITEM_PIPELINES = {'slybot.dupefilter.DupeFilterPipeline':1}

answered Jul 22 '16 at 08:21

magicshine

1
1

Can't deploy scrapy to scrapyd server

2 Answers2