4

I came across a situation where my scrapy code is working fine when used from command line but when I'm using the same spider after deploying (scrapy-deploy) and scheduling with scrapyd api it throws error in "scrapy.extensions.feedexport.FeedExporter" class.

  1. one is while initializing "open_spider" signal
  2. second is while initializing "item_scraped" signal and
  3. last while "close_spider" signal

1."open_spider" signal error 

2016-05-14 12:09:38 [scrapy] INFO: Spider opened
2016-05-14 12:09:38 [scrapy] ERROR: Error caught on signal handler: <bound method ?.open_spider of <scrapy.extensions.feedexport.FeedExporter object at 0x7fafb1ce4a90>>
Traceback (most recent call last):
  File "/home/jonsnow/venv/scrapy1/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/home/jonsnow/venv/scrapy1/lib/python2.7/site-packages/pydispatch/robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "/home/jonsnow/venv/scrapy1/lib/python2.7/site-packages/scrapy/extensions/feedexport.py", line 185, in open_spider
    uri = self.urifmt % self._get_uri_params(spider)
TypeError: float argument required, not dict

2."item_scraped" signal error:

2016-05-14 12:09:49 [scrapy] DEBUG: Scraped from <200 https://someurl.>
2016-05-14 12:09:49 [scrapy] ERROR: Error caught on signal handler: <bound method ?.item_scraped of <scrapy.extensions.feedexport.FeedExporter object at 0x7fafb1ce4a90>>
Traceback (most recent call last):
  File "/home/jonsnow/venv/scrapy1/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/home/jonsnow/venv/scrapy1/lib/python2.7/site-packages/pydispatch/robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "/home/jonsnow/venv/scrapy1/lib/python2.7/site-packages/scrapy/extensions/feedexport.py", line 210, in item_scraped
    slot = self.slot
AttributeError: 'FeedExporter' object has no attribute 'slot'

3."close_spider" signal error:

2016-05-14 12:09:49 [scrapy] INFO: Closing spider (finished)
2016-05-14 12:09:49 [scrapy] ERROR: Error caught on signal handler: <bound method ?.close_spider of <scrapy.extensions.feedexport.FeedExporter object at 0x7fafb1ce4a90>>
Traceback (most recent call last):
  File "/home/jonsnow/venv/scrapy1/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/home/jonsnow/venv/scrapy1/lib/python2.7/site-packages/pydispatch/robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "/home/jonsnow/venv/scrapy1/lib/python2.7/site-packages/scrapy/extensions/feedexport.py", line 193, in close_spider
    slot = self.slot
AttributeError: 'FeedExporter' object has no attribute 'slot'
  • Scrapy: version 1.1.0
  • scrapyd: version 1.1.0

also tried with scrapyd (1.1.0 with scrapy version 0.24.6)

Pawel Miech
  • 7,742
  • 4
  • 36
  • 57
MrPandav
  • 1,831
  • 1
  • 20
  • 24
  • Are you using a special FEED_URI? – paul trmbrth May 15 '16 at 10:05
  • yes...i am using FEED_URI... – MrPandav May 15 '16 at 10:14
  • The FEED_URI does not seem to please the feed exporter in open_spider, so slot is not set (exception in the first statement) and the other signal handlers fails because of it – paul trmbrth May 15 '16 at 10:31
  • i am using normal default settings...i am not disabling the feed uri , like other solution suggest..i am not doing anything out of the box – MrPandav May 15 '16 at 10:33
  • Would setting items_dir to empty in scrapyd.conf work? See https://github.com/scrapy/scrapyd/issues/31 – paul trmbrth May 15 '16 at 11:30
  • i ran the scrapyd from git master branch, and it has items_dir set to empty by default but it did not work as well – MrPandav May 15 '16 at 12:37
  • this was working before, created new virtual environment for scrapyd and ran this from there still same error..do you think directory permission will create this issue ?...i am running on linux and project is stored on Ntfs partition – MrPandav May 15 '16 at 12:40
  • working before as in when i have tried locally before 6-8 months.. since than not much has changed.. it has same system , same Python version and everything same – MrPandav May 15 '16 at 12:42
  • ironically, this is the same project that i have developed tried locally on scrapyd , deployed it on server , ran there for 8 months without single crash and had website change..fixed few xpaths nd tried locally and boom this error – MrPandav May 15 '16 at 12:47
  • I have the same problem as soon as i custom a FEED_EXPORTERS setting variable – miguelfg Jun 08 '16 at 14:52

2 Answers2

17

It happens when feedexporter is not able to write file, it happend to me when i had my previously exported csv file open in excel. try to close the opened exported file it will work fine

Jatin Gera
  • 814
  • 1
  • 11
  • 25
0

sometimes it happens even if file is closed or even result is not being saved in a file but there is some permission issue. try command with root access(sudo in linux and run as admin in windows), I have tried and the issue is resolved.

Ravendhar Lalwani
  • 1