1

I have a scrapy spider configured to fetch data from some sites. and i have developed a ui in php and html to show scraped data.this data is being fetched from json file being generated upon running scrapy command. iam using ampps for running php. initally my php code look like following

$output= shell_exec('cd /home/testuser/Desktop/scrapy_tutorial/ && scrapy crawl example -o example.json 2>&1');
print_r($output);

and i got result as 'scrapy command not found'.so i changed my code to set full path of scrapy bin

$output= shell_exec('cd /home/testuser/Desktop/scrapy_tutorial/ && /usr/local/bin/scrapy crawl example -o example.json 2>&1');
print_r($output);

now iam getting output which contains message 

PermissionError: [Errno 13] Permission denied: 'example.json'

looks like crawler works fine but has no permissions for writing to file.

i checked user which executes php script using 

exec('whoami');

and it outputs 'ampps'

Any help will be appreciated.

helvete
  • 2,455
  • 13
  • 33
  • 37
arjun
  • 75
  • 9
  • 1
    So, check the permissions needed to write to that location. Either by granting write access for the user PHP is being run as or by writing the output to some other location. – helvete Mar 21 '19 at 10:11
  • Thanks @helwet, I granted write access to output folder and now its being working fine. the whole big scrapy log confused me a lot, simple fix ,thanks again. – arjun Mar 21 '19 at 10:25
  • You're welcome. I am glad it works for you now. – helvete Mar 21 '19 at 10:30
  • @helvete feel free to convert your comment into an answer. – Gallaecio Mar 21 '19 at 14:40
  • @Gallaecio: It's actually a good idea - https://stackoverflow.com/a/55284164/2915423 – helvete Mar 21 '19 at 15:43

1 Answers1

1

The root of the problem is in that PHP runs as a user that has no write privilege to the location where the output is being attempted to store.

There are two simple solutions to that:

  1. Make the target location writable by the user PHP acts as.

    • like this everyone gets write access to the directory:

      chmod a+w /path/to/location

    • like this only group members get write access to the directory:

      usermod -aG <group-name-of-the-location-owner> ampps

      chmod g+w /path/to/location

  2. Save the output elsewhere. The /tmp directory is a location useful for things like this. (Be sure to delete the file after processing in case the data inside are sensitive)

helvete
  • 2,455
  • 13
  • 33
  • 37