Phantomjs not able to open certain URLs

Question

I am using the following code to take screenshot of a page using phantomjs. It works perfectly for most of the pages but shows a "Operation canceled" error for some URLs.

var page = require('webpage').create(),
system = require('system');

page.viewportSize = {
  width: 1366,
  height: 800
};

page.onConsoleMessage = function(msg) {
  console.log("CONSOLE MESSAGE:" + msg);
};

page.onResourceError = function(resourceError) {
    console.log('Unable to load resource (#' + resourceError.id + 'URL:' + resourceError.url + ')');
    console.log('Error code: ' + resourceError.errorCode + '. Description: ' + resourceError.errorString);
};

page.open(<URL>, function(status) {
  if (status !== 'success') {
    console.log('FAIL to load the address');
    phantom.exit();
  } else {
    page.render("3.png");

    phantom.exit();
  }
});
console.log("opening");

Some of the examples of URLs not working are: http://india.gov.in, http://timesofindia.indiatimes.com/ etc.

For india.gov.in the first request to the url: http://india.gov.in/ is being resend midway and all the existing operations get cancelled just before it. Similar things happens with some other URLs as well.

Is the page getting refreshed? If yes how do i handle it?

How to follow redirects/reloads in phantomjs?

Further investigation gave me some more very interesting stats. During india.gov.in, phantomjs started sending requests and receiving responses at one point (after making request id: 118) the number of pending requests was 37. Phantom 'canceled' the pending requests and started loading page from the beginning. It started sending requests again.

After request id 156 the number of pending requests was again 37 (it received response only to one request this time). So it again 'canceled' all the pending requests are started again.

After request id 194 the number of pending requests was again 37 so it again canceled and restarted.

After request id 232 the number of pending requests was again 37 but this time it didn't restarted and canceling.

Is there a limit to number of pending requests that phantom can handle?

Thanx!!

Note! Note! Note!: It worked with phantomjs 2 right out of the box. But since that is still in development, it can't be used in my production setup. Does anyone has any idea which updation between 1.9.8 and 2.0 fixed it? I could probably import that change in 1.9.8's source and build it.

Interesting ... if you lock the navigation (`page.navigationLocked = true;`) from the `page.onNavigationRequested` event handler, the screenshot is created but contains only a loading screen. I suspect there is some kind of dubious redirect using JavaScript. — Artjom B., Nov 09 '14 at 17:59
@ArtjomB. i think if the javascript finds that a non normal browser is being used it refreshes the page. Since I found the first request later being resend. Is there a way we can make phantom 100% emulate a normal browser? — Surender Thakran, Nov 09 '14 at 18:04
I tried with PhantomJS 1.9.7 which sits there a while and later cancels the very first request. So 1.9.8 is a little bit better that it does at least something. SlimerJS behaves the same way as PhantomJS 1.9.7. Maybe you should try PhantomJS 2 (and compile it from sources). — Artjom B., Nov 11 '14 at 08:09
@ArtjomB. yes i think thats the only solution left for me apart from ditching phantomjs and trying pywebkitgtk. Thanx for your time. — Surender Thakran, Nov 11 '14 at 12:37

score 0 · Answer 1 · answered Nov 18 '14 at 20:30

0

Refer to this issue.

https://github.com/ariya/phantomjs/issues/10522

This is to do with a lack of reliable support for bind(). With phantomjs 1.9.7 there was an attempt to address this but it was never resolved fully until version 2.0.

The fix is working in the main branch. The only solution for you seem to be compiling pre-release 2.0 yourself and upgrade to the official 2.0 when released.

answered Nov 18 '14 at 20:30

dors

1,152
11
19

3

This is not a bind issue, as bind doesn't let PhantomJS crash and bind errors are properly reported. – Artjom B. Nov 18 '14 at 21:15

score 0 · Answer 2 · answered Aug 17 '16 at 04:18

I　had the same problem too and I finally know the reason.

You must had enabled the disk-cache.

The disk-cache of phantomjs has a dead lock bug. By keep doing job on the same url for many times, i ran out of this bug. If you delete the cache directory, then everything is OK.

Phantomjs not able to open certain URLs

2 Answers2