1

I am trying to deploy to ScrapingHub and here is the error I am getting...

Deploy log last 30 lines:
  File "/app/python/lib/python3.8/site-packages/scrapy/cmdline.py", line 142, in execute
    cmd.crawler_process = CrawlerProcess(settings)
  File "/app/python/lib/python3.8/site-packages/scrapy/crawler.py", line 280, in __init__
    super(CrawlerProcess, self).__init__(settings)
  File "/app/python/lib/python3.8/site-packages/scrapy/crawler.py", line 152, in __init__
    self.spider_loader = self._get_spider_loader(settings)
  File "/app/python/lib/python3.8/site-packages/scrapy/crawler.py", line 146, in _get_spider_loader
    return loader_cls.from_settings(settings.frozencopy())
  File "/app/python/lib/python3.8/site-packages/scrapy/spiderloader.py", line 60, in from_settings
    return cls(settings)
  File "/app/python/lib/python3.8/site-packages/scrapy/spiderloader.py", line 24, in __init__
    self._load_all_spiders()
  File "/app/python/lib/python3.8/site-packages/scrapy/spiderloader.py", line 46, in _load_all_spiders
    for module in walk_modules(name):
  File "/app/python/lib/python3.8/site-packages/scrapy/utils/misc.py", line 77, in walk_modules
    submod = import_module(fullpath)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/app/__main__.egg/sstkscrape/spiders/sstkscrape_spider.py", line 4, in <module>
  File "/app/__main__.egg/sstkscrape/spiders/sstkscrape_spider.py", line 7, in QuoteSpider
FileNotFoundError: [Errno 2] No such file or directory: 'urls-test.csv'
{"message": "shub-image-info exit code: 1", "details": null, "error": "image_info_error"}

{"status": "error", "message": "Internal error"}
Deploy log location: /var/folders/p7/nwmq6_4138n6t3w2spdnpzfm0000gn/T/shub_deploy_n9zmxfto.log
Error: Deploy failed: b'{"status": "error", "message": "Internal error"}'

I have a local csv file I am trying to include in the deploy and doing open of that csv file and wondering if that is causing the issues?

johncsmith427
  • 83
  • 8

1 Answers1

1

You need to use package_data section of you setup.py file: Deploying non-code files

gangabass
  • 10,607
  • 2
  • 23
  • 35
  • is this right in my setup.py? from setuptools import setup, find_packages setup( name = 'sstkscrape', version = '1.0', packages = find_packages(), entry_points = {'scrapy': ['settings = sstkscrape.settings']}, package_data={ 'project': ['resources/*.csv'] }, ) – johncsmith427 May 15 '20 at 15:51
  • Please update your question using above code. Could you also show your directory structure (CSV file location)? – gangabass May 15 '20 at 15:55