1

I want scrapyd to run my spider on cloud, for which I want to pass the value of FEED_URI as a parameter in scrapyd command.

Currently I am using following configuration in custom setting to store the output on s3.

custom_settings = {
        'FEED_FORMAT':'csv',
        'FEED_URI':'s3://my_bucket/%s'%('OP_FILE')
        }

However I would like to pass the value for the custom_setting through the

curl http://localhost:6800/schedule.json -d project=link -d spider=link -d OP_FILE="output_link.csv" 

Apparently custom_setting value can't be set inside 'init' method of the link spider class.

What can I do to resolve this.

0 Answers0