Running a SlimerJS + headless Firefox script through PHP on webserver?

Question

On my server (Ubuntu 14.04.4 LTS), I have a Firefox installed, as well as xvfb for headless Firefox operation, and CasperJS with SlimerJS. I also have a CasperJS script which works fine. I want to utilize this script from PHP; this is the essence of my PHP script for this, let's call it mytest.php:

echo "php_sapi_name() " . php_sapi_name() . "\n"; // "cli" for php cli, "apache2handler" for php via webserver

chdir(dirname(__FILE__));

$nodeModPath = "/home/USERNAME/.nvm/versions/node/v4.0.0/lib/node_modules";

putenv("SLIMERJSLAUNCHER=/usr/bin/firefox46");
$cmdline = "xvfb-run $nodeModPath/casperjs/bin/casperjs --engine=slimerjs --debug=true mySlimerScript.js";

$returnString = shell_exec($cmdline);
echo "$returnString\n";

EDIT: Note that the command could as well be just:

$cmdline = "xvfb-run $nodeModPath/casperjs/bin/casperjs --engine=slimerjs --debug=true 2>&1";

... that is, without any JS script listed - in which case the help should be dumped (and is, in case of CLI access - but the same error as below is reported when accessing through webserver)

When I run this PHP script from the terminal command line (via SSH), that is through PHP in CLI mode:

$ php mytest.php

... everything runs fine, there is no problem whatsoever.

However, when I invoke this PHP script online through the webserver, that is via http://example.com/mytest.php, it fails first with the error:

Gecko error: it seems /usr/bin/firefox46 is not compatible with SlimerJS.
See Gecko version compatibility. If version is correct, launch slimerjs
with --debug=true to see Firefox error message

... and after adding --debug=true (as already included in the example above), I additionally get this error:

JavaScript error: resource://gre/modules/FileUtils.jsm, line 63: NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIProperties.get]

So, apparently my headless Firefox does not want to run, when PHP is invoked through the webserver (in which case, PHP reports that it uses the apache2handler SAPI).

Would anyone know why this happens - and how can I get the script to execute properly when called from a webserver, just as when it runs under PHP CLI mode?

EDIT 2: Can now reconstruct this error via CLI mode too, and can confirm it is due to the user; so without any JS script provided in the $command, I get this:

$ sudo -H -u root php mytest.php
...
Usage: casperjs [options] script.[js|coffee] [script argument [script argument ...]]
       casperjs [options] test [test path [test path ...]]
       casperjs [options] selftest
...

$ sudo -H -u www-data php mytest.php
JavaScript error: resource://gre/modules/FileUtils.jsm, line 63: NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIProperties.get]
Gecko error: it seems /usr/bin/firefox46 is not compatible with SlimerJS.
See Gecko version compatibility. If version is correct, launch slimerjs
with --debug=true to see Firefox error message

First thoughts : files permissions as running the script in `cli` mode will have the user running it, however, through the webserver, the user will be `www-data`. Possible, isn't it? — php-dev, Jun 30 '16 at 13:56
Indeed, @php-dev, but I cannot really figure out whose permissions would be the problem; here I call `xvfb-run` which calls `casperjs` which calls `slimerjs` which calls headless Firefox. Is there a way to check for permission errors somewhere in this kind of case? I tried `/var/log/apache2/error-mysite.log`, but it only reports PHP stuff like `Trying to get property of non-object` once the `shell_exec` call fails. — sdbbs, Jun 30 '16 at 14:00
Thanks @php-dev - I added `2>&1` to the end of the `$cmdline`, but unfortunately I get no new information whatsoever. However, I also tried a direct `shell_exec("xvfb-run /usr/bin/firefox46 2>&1")`, and this one does NOT seem to crash (however, it doesn't exit on its own either, as expected). So seemingly, `xvfb-run` and `firefox` alone are not the problem?! — sdbbs, Jun 30 '16 at 14:15

score 5 · Answer 1 · answered Jun 30 '16 at 15:16

Well, this was a nasty problem. I ended up doing an strace, and comparing the logs, for the root user and the www-data user when running a full slimerjs (the full command line can be found by adding echoes to /path/to/slimerjs-0.10.1-pre/slimerjs):

sudo -H -u www-data strace \
  /usr/bin/firefox46 -app /path/to/slimerjs-0.10.1-pre/application.ini \
  --profile /path/to/firefox-46.0.1/profile-46 -no-remote --debug=true /home/USERNAME/.nvm/versions/node/v4.0.0/lib/node_modules/casperjs/bin/bootstrap.js --casper-path=/home/USERNAME/.nvm/versions/node/v4.0.0/lib/node_modules/casperjs \
  --cli 2>&1 \
  | tee /tmp/strace.log

sudo -H -u root strace \
  /usr/bin/firefox46 -app /path/to/slimerjs-0.10.1-pre/application.ini \
  --profile /path/to/firefox-46.0.1/profile-46 -no-remote --debug=true /home/USERNAME/.nvm/versions/node/v4.0.0/lib/node_modules/casperjs/bin/bootstrap.js --casper-path=/home/USERNAME/.nvm/versions/node/v4.0.0/lib/node_modules/casperjs \
  --cli 2>&1 \
  | tee /tmp/straceR.log

If these logs are now compared in say meld, then the eventually start diverging at a point like this:

mkdir("/root/.innophi", 0700)           = 0
mkdir("/root/.innophi/slimerjs", 0700)  = 0

... [vs.] ...

mkdir("/var/www/.innophi", 0700)        = -1 EACCES (Permission denied)
access("/var/www/.innophi", F_OK)       = -1 ENOENT (No such file or directory)

So, casperJS basically tries to create a directory in the home directory of the user; the problem is , www-data's $HOME is /var/www, where it seemingly has no write access!

So, the easiest thing for me was to "hack" the $HOME environment variable in the mytest.php script, and set it to /tmp, where www-data definitely has write permissions:

...
putenv("SLIMERJSLAUNCHER=/usr/bin/firefox46");
putenv("HOME=/tmp");
...

... and whaddayaknow, finally the script works under the www-data user from CLI too:

$ sudo -H -u www-data php test_commands.php
...
Options:

--verbose   Prints log messages to the console
--log-level Sets logging level
--help      Prints this help
...

Btw, this .innophi directory seems to also be mentioned in https://docs.slimerjs.org/current/configuration.html#profiles ,

Running a SlimerJS + headless Firefox script through PHP on webserver?

1 Answers1