WWW::Mechanize::PhantomJS code in Mojolicious Lite script doesn't work when running in background mode

Question

I have this very simple Mojolicious Lite script:

#!/usr/bin/env perl

use v5.10;
use WWW::Mechanize::PhantomJS;
use Mojolicious::Lite;

my $mech = WWW::Mechanize::PhantomJS->new();

$mech->get('http://stackoverflow.com/');

get '/test' => sub {
    my $c = shift;
    $mech->get("https://stackoverflow.com/questions");
    $c->render(template => 'activity');
};

app->secrets(['test secret']);
app->start;


__DATA__

@@ activity.html.ep
<!DOCTYPE html>
<html>
  <head><title>Test</title></head>
  <body><h2>Test</h2></body>
</html>

When I start it using hypnotoad in foreground mode (hypnotoad -f ./script.pl), and access /test url -- I get my test page, and clean logs:

[Fri Dec 11 18:00:23 2015] [info] Listening at "http://*:8080"
[Fri Dec 11 18:00:23 2015] [info] Manager 3011 started
[Fri Dec 11 18:00:23 2015] [info] Creating process id file "/home/username/pc_activity/demo_site/hypnotoad.pid"

When I start it using background mode (hypnotoad ./script.pl), and access /test url -- I get "something went very wrong" error page with throwing up raptor.

[Fri Dec 11 17:58:07 2015] [info] Listening at "http://*:8080"
[Fri Dec 11 17:58:07 2015] [info] Manager 2964 started
[Fri Dec 11 17:58:07 2015] [info] Creating process id file "/home/username/pc_activity/demo_site/hypnotoad.pid"
[Fri Dec 11 17:58:14 2015] [error] Error while executing command: get: Server returned error message Can't connect to localhost:8910

Connection refused at /usr/local/share/perl/5.18.2/LWP/Protocol/http.pm line 47, <DATA> line 49.
 instead of data at /usr/local/share/perl/5.18.2/Selenium/Remote/Driver.pm line 310.

It turns out the localhost:8910 is default settings for PhanomJS to run at in webdriver mode. Which it isn't on my machine. But even if I start it, and then access the URL, I still get an error:

[Fri Dec 11 18:41:01 2015] [error] Error while executing command: get: Server returned error message Variable Resource Not Found - {"headers":{"Accept":"application/json","Connection":"TE, close","Content-Length":"45","Content-Type":"application/json; charset=utf-8","Host":"localhost:8910","TE":"deflate,gzip;q=0.3","User-Agent":"libwww-perl/6.15"},"httpVersion":"1.1","method":"POST","post":"{\"url\":\"https://stackoverflow.com/questions\"}","url":"/session/98799e80-a060-11e5-8907-0b365878087d/url","urlParsed":{"anchor":"","query":"","file":"url","directory":"/session/98799e80-a060-11e5-8907-0b365878087d/","path":"/session/98799e80-a060-11e5-8907-0b365878087d/url","relative":"/session/98799e80-a060-11e5-8907-0b365878087d/url","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/session/98799e80-a060-11e5-8907-0b365878087d/url","queryKey":{},"chunks":["session","98799e80-a060-11e5-8907-0b365878087d","url"]}} instead of data at /usr/local/share/perl/5.18.2/Selenium/Remote/Driver.pm line 310.

I guess, I don't understand, why does it run in foreground mode, and not in background mode. And then, what can I do to make it run in background mode?

score 1 · Accepted Answer · answered Dec 14 '15 at 09:39

The reason it will not run in background mode is because of threading. After you start hypnotoad in background it spawns some processes.

 664161507 46028     1   0 10:34AM ??         0:00.01 script.pl
 664161507 46029 46028   0 10:34AM ??         0:00.08 script.pl
 664161507 46030 46028   0 10:34AM ??         0:00.01 script.pl
 664161507 46031 46028   0 10:34AM ??         0:00.01 script.pl
 664161507 46032 46028   0 10:34AM ??         0:00.01 script.pl

These will not have access to the PhantomJS you created in the parent process. I didn't look into how this communication is done, but if you want to share PhantomJS for your workers you need to make it a separate service.

If you however want to have a PhantomJS for each request you can initialize it in the request, but I don't necessarily encourage this approach:

 get '/test' => sub {
     my $c = shift;
     my $mech = WWW::Mechanize::PhantomJS->new();
     $mech->get('http://stackoverflow.com/');
     $mech->get("https://stackoverflow.com/questions");
     $c->render(template => 'activity');
 };

Thank you! I'll try this, as this script of mine does not expect many hits. The reason I wanted to avid this, was to be able to login to a site (not S.O.) "outside" requests, and then on the requests do parsing things on pages of my logged in account. If I init on each request, I'll need to login each time. But then, maybe not, if I enable saving of cookies. I'll play with it.... More code for a simple thing I wanted :) — flamey, Dec 14 '15 at 21:25
You might try to make an variable that gets initialized on the first request to that child. It might do the trick. i.e. `$mech ||= initialize_mech();` and then `my $mech;` without initialization where it is. — bolav, Dec 14 '15 at 22:35

WWW::Mechanize::PhantomJS code in Mojolicious Lite script doesn't work when running in background mode

1 Answers1