3

Scrapyd is service where we can eggify deploy our projects. However I am facing a problem. I have a Project named MyScrapers whose spider classes uses an import statement as follows:

from mylibs.common.my_base_spider import MyBaseSpider

The path to my_base_spider is /home/myprojectset/mylibs/common/my_base_spider

While setting environment variable PYTHONPATH=$HOME/myprojectset/, I am able to run MyScrapers using scrapy command: scrapy crawl MyScrapers.

But when I use scrapyd for deploying MyScrapers by following command: scrapy deploy scrapyd2 -p MyScrapers, I get the following error:

Server response (200): {"status": "error", "message": "ImportError: No module named mylibs.common.my_base_spider"}

Please tell how to make deployed project to use these libs?

fgb
  • 18,439
  • 2
  • 38
  • 52
Haider Mahmood
  • 31
  • 5

2 Answers2

1

You need to edit your setup.py and/or MANIFEST.in to declare that it has a mylibs package dependency.

See the distutils documentation for more information.

Francis Avila
  • 31,233
  • 6
  • 58
  • 96
  • Any idea about this error? {"status": "error", "message": "AttributeError: 'NoneType' object has no attribute 'module_name'"} – Haider Mahmood Nov 10 '12 at 21:20
  • For your info, here is my directory structure for MyScrapers' setup.py: /home/myprojectset/scrapy_projects/MyScrapers/setup.py and Here is path to mylibs: /home/myprojectset/scrapy_projects/mylibs/ how would I add mylibs to setup.py? – Haider Mahmood Nov 10 '12 at 21:34
  • Which means, my libs is outside my project. So, it cannot be deployed as an egg. But I need to use it in imports. Please response ASAP. – Haider Mahmood Nov 11 '12 at 07:52
  • Did you read the link? Did you try changing the `packages` argument? We can't tell you *exactly* what to do because you haven't shown us your `setup.py` and we can't replicate your build environment exactly, but the basic answer is the same--you need to declare the dependency. Once you do it will be bundled into the egg. – Francis Avila Nov 11 '12 at 15:39
  • Actually `mylibs` were way outside from the location where `setup.py` of my deploy-able project is present. I tried giving full paths to `mylibs` in `packages` property of `setup.py` but that didn't solve my problem. I found the answer by adding `mylibs` to `site-packages` of python by using `setup.py` inside `mylib` folder. That way I could import everything inside `mylib` in my projects. – Haider Mahmood Nov 11 '12 at 16:48
  • What you have done is create a separate `mylibs` distribution with the `mylibs` package in it. This is a solution, but it's not the same as making the `mylibs` package part of your scrapy project's distribution. In either case, you still need to make sure `mylibs` is declared as a dependency so that when you one day need to say `easy_install my_scrapy_project`, it will download and install `mylibs` as well. You should add what you did as an answer here for future visitors. – Francis Avila Nov 11 '12 at 18:30
  • Yes, I have added it as an answer. Thanks for providing help. :) – Haider Mahmood Nov 12 '12 at 13:37
0

I found the answer by adding mylibs to site-packages of python by using setup.py inside mylib folder. That way I could import everything inside mylib in my projects. Actually mylibs were way outside from the location where setup.py of my deploy-able project is present. setup.py looks for packages on same level and inside the folders where it is located.

Haider Mahmood
  • 31
  • 5