Scrapyd: How to write data to json file?

Question

I have a working scrapy 2.1.0 project where I write data to a json file:

def open_spider(self, spider):
    self.file = open('data/'+  datetime.datetime.now().strftime ("%Y%m%d") + '_' + spider.name + '.json', 'wb')
    self.exporter = JsonItemExporter(self.file)

Now I want to deploy it to scrapyd, but since the data folder does not exist under /tmp it will lead to the following error:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/scrapy/crawler.py", line 91, in crawl
    yield self.engine.open_spider(self.spider, start_requests)
FileNotFoundError: [Errno 2] No such file or directory: 'data/20200518_rieger.json'

How can the above code be rewriten in order to work with scrapyd?

Yes it did. I did define a folder including an absolute path within the settings that can be changed for the different environments and now it does work on production. — merlin, May 18 '20 at 21:47
sorry just to see this now, but i posted a solution so you can use when this occurs in the future — n.qber, May 19 '20 at 03:06

score 0 · Answer 1 · answered May 19 '20 at 03:10

To create a folder we can use mkdir function from os module

def make_folder(name):
    try:
        mkdir(name)
    except FileExistsError:
        pass

or if you don't want to use try except, you can use listdir to check if the folder exists

def make_folder(name):
    if name not in listdir(path):
        mkdir(name)

this is good for every program that needs a folder to exists, yay!

Scrapyd: How to write data to json file?

1 Answers1