I want scrapyd to run my spider on cloud, for which I want to pass the value of FEED_URI as a parameter in scrapyd command.
Currently I am using following configuration in custom setting to store the output on s3.
custom_settings = {
'FEED_FORMAT':'csv',
'FEED_URI':'s3://my_bucket/%s'%('OP_FILE')
}
However I would like to pass the value for the custom_setting through the
curl http://localhost:6800/schedule.json -d project=link -d spider=link -d OP_FILE="output_link.csv"
Apparently custom_setting value can't be set inside 'init' method of the link spider class.
What can I do to resolve this.