1

I want to create a custom service for scrapyd, from the scrapy project but I keep getting error.

I created crawler/webservice.py:

from scrapyd.webservice import WsResource


class BackInTime(WsResource):
    def render_GET(self, txtrequest):
        return {
            'id': 'something works!'
        }

and then modified scrapyd.conf and added the line backintime.json = crawler.webservice.BackInTime under [services] tag.

Bu when I run the command scrapyd I keep getting the error:

...
  File "c:\personal\virtualenvs\crawlie\lib\site-packages\scrapyd\app.py", line 39, in application
    webservice = TCPServer(http_port, server.Site(webcls(config, app)), interface=bind_address)
  File "c:\personal\virtualenvs\crawlie\lib\site-packages\scrapyd\website.py", line 33, in __init__
    servCls = load_object(servClsName)
  File "c:\personal\virtualenvs\crawlie\lib\site-packages\scrapy\utils\misc.py", line 46, in load_object
    mod = import_module(module)
  File "C:\Users\user\Anaconda3\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import

  File "<frozen importlib._bootstrap>", line 971, in _find_and_load

  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "<frozen importlib._bootstrap>", line 994, in _gcd_import

  File "<frozen importlib._bootstrap>", line 971, in _find_and_load

  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked

builtins.ModuleNotFoundError: No module named 'crawler'


Failed to load application: No module named 'crawler'

Tried to follow some previous solutions but without any success:

How should I add the new service so scrapyd can import it?

George ȚUȚUIANU
  • 43
  • 1
  • 4

1 Answers1

0

Try adding the import module of your app to the environment variable PYTHONPATH. In my Dockerfile it looks like this:

ENV PYTHONPATH "${PYTHONPATH}:${APPDIR}/${APPNAME}"

On your local machine it might look like:

export PYTHONPATH="${PYTHONPATH}:/home/app/crawler"

After that, run Scrapyd.

Kyle F Hartzenberg
  • 2,567
  • 3
  • 6
  • 24
Arag Entum
  • 1