1

I'm trying to run two scrapy spiders simultaneously using scrapyd. I execute

curl http://localhost:6800/schedule.json -d project=myproject -d spider=somespider

result - {"status": "ok", "jobid": "6487ec79947edab326d6db28a2d86511e8247444"} then:

curl http://localhost:6800/schedule.json -d project=myproject -d spider=somespider2

result - 

    python2.7/site-packages/twisted/web/server.py", line 234, in render
            body = resrc.render(self)
        --- <exception caught here>
        File "/usr/local/lib/python2.7/json/decoder.py", line 383, in raw_decode
raise ValueError("No JSON object could be decoded")
            exceptions.ValueError: No JSON object could be decoded

and everytime I run curl for scrapyd - I get the same - exceptions.ValueError: No JSON object could be decoded. The only one way to proceed working with scrapyd - to restart it. How can I run multiple spiders with scrapyd? Why do I get that error?

I use scrapy 1.1, scrapyd 1.1, python 2.7.9

Rainmaker
  • 10,294
  • 9
  • 54
  • 89

2 Answers2

0

Your error has nothing to do the scrapyd, most probably the error is in your spider 'somespider2'. Just ensure that you are able to run 'somespider2' properly via command line.

Ketan Patel
  • 157
  • 1
  • 2
  • 6
  • So why the same error occurs when I run someSpider twise? (First time scrapyd returned job ID) Evenmore, I get the same error if I execute curl with 'listspiders' params twice. (First time scrapid returns correct result) – Rainmaker May 17 '16 at 15:23
0

The problem was in sqlite3 installed in my Centos6.5. Decision: need to add

self.conn.text_factory = bytes

to SqliteDict and SqlitePriorityQueue classes after connect creation in a scrapyd/sqlite.py b/scrapyd/sqlite.py

Rainmaker
  • 10,294
  • 9
  • 54
  • 89