Scrapyd Error: exceptions.AttributeError: 'dict' object has no attribute 'fields'

Question

I recently published a working scrape to scrapyd. I'm getting the error message below when I run the scrape.

I reviewed this closed issue: https://github.com/scrapy/scrapy/issues/86 and implemented the recommended fix per the docs: http://scrapyd.readthedocs.org/en/latest/config.html

My config file lives here: :/etc/scrapyd/conf.d/000-default

I cannot seem to disable the Feedexporter extension by setting items_dir to none. Is there a new way to disable feedexporter in scrapyd?

Config file:

[scrapyd]
http_port  = 6800
debug      = off
#max_proc  = 1
eggs_dir   = /var/lib/scrapyd/eggs
dbs_dir    = /var/lib/scrapyd/dbs
items_dir  =
logs_dir   = /var/log/scrapyd

Error Message:

2014-10-12 06:29:15-0500 [jsc] ERROR: Error caught on signal handler: <bound method ?.item_scraped of <scrapy.contrib.feedexport.FeedExporter object at 0x7fc879ff82d0>>
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/lib/pymodules/python2.7/scrapy/core/scraper.py", line 215, in _itemproc_finished
    item=output, response=response, spider=spider)
  File "/usr/lib/pymodules/python2.7/scrapy/signalmanager.py", line 23, in send_catch_log_deferred
    return signal.send_catch_log_deferred(*a, **kw)
  File "/usr/lib/pymodules/python2.7/scrapy/utils/signal.py", line 53, in send_catch_log_deferred
    *arguments, **named)
--- <exception caught here> ---
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 139, in maybeDeferred
    result = f(*args, **kw)
  File "/usr/lib/pymodules/python2.7/scrapy/xlib/pydispatch/robustapply.py", line 54, in robustApply
    return receiver(*arguments, **named)
  File "/usr/lib/pymodules/python2.7/scrapy/contrib/feedexport.py", line 190, in item_scraped
    slot.exporter.export_item(item)
  File "/usr/lib/pymodules/python2.7/scrapy/contrib/exporter/__init__.py", line 87, in export_item
    itemdict = dict(self._get_serialized_fields(item))
  File "/usr/lib/pymodules/python2.7/scrapy/contrib/exporter/__init__.py", line 71, in _get_serialized_fields
    field = item.fields[field_name]
exceptions.AttributeError: 'dict' object has no attribute 'fields'

One additional comment. I've tried to run the scrape with items_dir = 0 and items_dir = None. Neither work. — dfriestedt, Oct 12 '14 at 17:29
Appears to be a know issue with scrapyd: https://github.com/scrapy/scrapyd/issues/56 — dfriestedt, Oct 12 '14 at 19:41

score 1 · Answer 1 · answered Oct 12 '14 at 20:27

1

I was able to disable FeedExporter in scrapyd by adding the following line to settings.py

EXTENSIONS = {'scrapy.contrib.feedexport.FeedExporter': None}

Disabling FeedExporter solved the problem.

answered Oct 12 '14 at 20:27

dfriestedt

483
1
3
18

Scrapyd Error: exceptions.AttributeError: 'dict' object has no attribute 'fields'

1 Answers1