0

As the title says... how can i run my scrapy project with specific url paramter from php? I tried this on windows before and it worked perfectly, now on linux it's doing nothing.

On Windows working:

pclose(popen('cd .. & scrapy crawl mySpider -a "urls=http://www.example.com/'.$variable1.'/'.$variable2.'"','r'));

On Linux not working:

pclose(popen('sudo cd .. | sudo scrapy crawl mySpider -a "urls=http://www.example.com/'.$variable1.'/'.$variable2.'"','r'));

I checked already that the php file tries to execute the script as the user www-data so i added this user for testing purpose to sudoer list but it is still not working. When i try to use the command in the shell directly it is working though. I also checked if all files are owned by www-data and are executable and they are. My spider is owned by www-data and has 755 rights. What am i missing here?

Edit: When i change the user to www-data and try to run the command it works, just the php script is not running.

kratze
  • 186
  • 2
  • 11
  • Instead of `scrapy` use complete path like `/usr/local/bin/scrapy` or whatever is the actual path on your system – Tarun Lalwani Oct 21 '17 at 15:34
  • i tried it, it works in terminal then but not through php script... I just added the path to www-data environment so i can just use "scrapy" instead of full path. You have any clue why in terminal as user www-data it works but from php not? – kratze Oct 21 '17 at 17:40
  • Mostly it is because of difference between path variables. You can run `pclose(popen("env"))` to get the difference between the two environments – Tarun Lalwani Oct 21 '17 at 17:57
  • I checked it how you mentioned. It has same path as root. Output looks like this. APACHE_RUN_DIR=/var/run/apache2APACHE_PID_FILE=/var/run/apache2/apache2.pidPATH=/home/anaconda3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/gamesAPACHE_LOCK_DIR=/var/lock/apache2LANG=CAPACHE_RUN_USER=www-dataAPACHE_RUN_GROUP=www-dataAPACHE_LOG_DIR=/var/log/apache2PWD=/var/www/html/example.com/public – kratze Oct 21 '17 at 20:53
  • Put everything in a bash script and then run that bash script through code. Also make sure to redirect error to stdout using `2>&1` – Tarun Lalwani Oct 22 '17 at 05:19
  • I will try and report.. if it works you should write this as an answer to my question. – kratze Oct 23 '17 at 07:04
  • I doubt it would work, but would show you the error that is happening – Tarun Lalwani Oct 23 '17 at 07:07

1 Answers1

0

I found a solution. I figured out that php doesn't understand the Pipe operator.

The correct way to execute that command is the following:

pclose(popen('cd .. ; scrapy crawl mySpider -a "urls=https://example.com', 'r'));

The semicolon must be used instead.

kratze
  • 186
  • 2
  • 11