unable to deploy scrapy to scrapyd server

Question

I am trying to deploy my scrapy which connected to django project to scrapyd, but when I tried scrapyd-deploy JD -p JDSpider, it failed. It said No module named GradutionProject. It seems the scrapyd cannot detect "GradutionProject.settings" in settings.py. I have tried each combination of the path, but all failed. Could you please tell me possible solutions? Thanks in advance.

2019-03-24T01:02:02+0800 [_GenericHTTPChannelProtocol,12,127.0.0.1] Unhandled Error
        Traceback (most recent call last):
          File "d:\anacaonda\lib\site-packages\twisted\web\http.py", line 2190, in allContentReceived
            req.requestReceived(command, path, version)
          File "d:\anacaonda\lib\site-packages\twisted\web\http.py", line 917, in requestReceived
            self.process()
          File "d:\anacaonda\lib\site-packages\twisted\web\server.py", line 199, in process
            self.render(resrc)
          File "d:\anacaonda\lib\site-packages\twisted\web\server.py", line 259, in render
            body = resrc.render(self)
        --- <exception caught here> ---
          File "d:\anacaonda\lib\site-packages\scrapyd\webservice.py", line 21, in render
            return JsonResource.render(self, txrequest).encode('utf-8')
          File "d:\anacaonda\lib\site-packages\scrapyd\utils.py", line 20, in render
            r = resource.Resource.render(self, txrequest)
          File "d:\anacaonda\lib\site-packages\twisted\web\resource.py", line 250, in render
            return m(request)
          File "d:\anacaonda\lib\site-packages\scrapyd\webservice.py", line 86, in render_POST
            spiders = get_spider_list(project, version=version)
          File "d:\anacaonda\lib\site-packages\scrapyd\utils.py", line 137, in get_spider_list
            raise RuntimeError(msg.encode('unicode_escape') if six.PY2 else msg)
        builtins.RuntimeError: Traceback (most recent call last):
          File "d:\anacaonda\lib\runpy.py", line 193, in _run_module_as_main
            "__main__", mod_spec)
          File "d:\anacaonda\lib\runpy.py", line 85, in _run_code
            exec(code, run_globals)
          File "d:\anacaonda\lib\site-packages\scrapyd\runner.py", line 40, in <module>
            main()
          File "d:\anacaonda\lib\site-packages\scrapyd\runner.py", line 37, in main
            execute()
          File "d:\anacaonda\lib\site-packages\scrapy\cmdline.py", line 108, in execute
            settings = get_project_settings()
          File "d:\anacaonda\lib\site-packages\scrapy\utils\project.py", line 68, in get_project_settings
            settings.setmodule(settings_module_path, priority='project')
          File "d:\anacaonda\lib\site-packages\scrapy\settings\__init__.py", line 292, in setmodule
            module = import_module(module)
          File "d:\anacaonda\lib\importlib\__init__.py", line 126, in import_module
            return _bootstrap._gcd_import(name[level:], package, level)
          File "<frozen importlib._bootstrap>", line 978, in _gcd_import
          File "<frozen importlib._bootstrap>", line 961, in _find_and_load
          File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
          File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
          File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
          File "c:\users\柘宇\appdata\local\temp_~1\jdspider-1553360516-l5i44at6.egg\JDSpider\settings.py", line 24, in <module>
          File "d:\anacaonda\lib\site-packages\django\__init__.py", line 19, in setup
            configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
          File "d:\anacaonda\lib\site-packages\django\conf\__init__.py", line 56, in __getattr__
            self._setup(name)
          File "d:\anacaonda\lib\site-packages\django\conf\__init__.py", line 43, in _setup
            self._wrapped = Settings(settings_module)
          File "d:\anacaonda\lib\site-packages\django\conf\__init__.py", line 106, in __init__
            mod = importlib.import_module(self.SETTINGS_MODULE)
          File "d:\anacaonda\lib\importlib\__init__.py", line 126, in import_module
            return _bootstrap._gcd_import(name[level:], package, level)
          File "<frozen importlib._bootstrap>", line 978, in _gcd_import
          File "<frozen importlib._bootstrap>", line 961, in _find_and_load
          File "<frozen importlib._bootstrap>", line 936, in _find_and_load_unlocked
          File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
          File "<frozen importlib._bootstrap>", line 978, in _gcd_import
          File "<frozen importlib._bootstrap>", line 961, in _find_and_load
          File "<frozen importlib._bootstrap>", line 948, in _find_and_load_unlocked
        ModuleNotFoundError: No module named 'GraduationProject'

scrapy.cfg

[settings]
default = JDSpider.settings

[deploy:JD]
url = http://localhost:6800/
project = JDSpider

settings.py

import os
import sys
sys.path.append(os.path.dirname(os.path.abspath('.')))
# Do not forget the change iCrawler part based on your project name
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "GraduationProject.settings")

import django
django.setup()

Project hierarchy

Do you have a pipeline that uses django's ORM? – digenishjkl Jun 23 '19 at 19:13 — digenishjkl, Jun 23 '19 at 19:13

score 2 · Answer 1 · answered Jul 29 '19 at 15:06

2

I came with the problem today and found out that, you need to cd to the root dir of your scrapy project inside the django project, then start scrapyd.

This is what I've found today, the readme of this repo was helpful:Scrapyd-Django-Template

answered Jul 29 '19 at 15:06

leon

75
1
8

Please consider adding details, on SO this can be considered as link-only answer – Romil Patel Jul 29 '19 at 15:48

score 0 · Answer 2 · answered Dec 22 '21 at 08:25

0

you should add GraduationProject to PYTHONPATH

answered Dec 22 '21 at 08:25

Stephen

635
5
7

unable to deploy scrapy to scrapyd server

2 Answers2