Continue with the next URL on timeout in CasperJS with eachThen and thenOpen

Question

I have an array of 500 links. I iterate through it using eachThen. Inside eachThen I add thenOpen to check link status but sometimes my thenOpen takes too much time and my entire application is idle.

Can I add timeout in eachThen so that if thenOpen takes too much time so I can skip that link and continue with another link in the array?

Jonatas Walker · Answer 1 · 2015-10-12T11:25:24.783

1

Set a stepTimeout on constructor and store failed urls on onStepTimeout like:

var casper = require('casper').create({
    verbose: true,
    logLevel: "warning",
    waitTimeout: 2000,
    stepTimeout: 2000, //<-- adjust timeout
    viewportSize: {
        width: 800,
        height: 400
    },
    pageSettings: {
        "userAgent": 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.10 (KHTML, like Gecko) Chrome/23.0.1262.0 Safari/537.10',
        "loadImages": false,
        "loadPlugins": false,         
        "javascriptEnabled": true,         
        "webSecurityEnabled": false,
        "ignoreSslErrors": true
    },
    onWaitTimeout: function() {
        //throw new Error
    },
    onStepTimeout: function() {
        //add an entry with timeout url
        failed.push(this.requestUrl);
        this.echo("Url " + this.requestUrl +" timed out ");
        this.clear();
    }
});

var urls = [
    //some urls
];

casper.start().eachThen(urls, function(response) {
    this.thenOpen(response.data, function(response) {
        console.log('Opened', response.url);
    });
});

EDIT: This is out of question's scope but to abort a request I use:

casper.on('resource.requested', function(requestData, request) {
    //some filter?        
    if (requestData.url.indexOf('/json/') > -1) {
        request.abort();
    }
});

edited Oct 12 '15 at 11:25

answered Oct 09 '15 at 09:33

Jonatas Walker

13,583
5
53
82

Hi, thanks for your reply but in the above code how I can skip that link which take so much time to load and continue with another link. also, I want to store that link also which take much time to load. – Suraj Dalvi Oct 09 '15 at 13:00
Hi, I tried your solution but in onStepTimeout function can we abort a request .if thenOpen take so much time to open that page? – Suraj Dalvi Oct 12 '15 at 10:41
But it won't take much time! You will set the time you want on **`stepTimeout`**. – Jonatas Walker Oct 12 '15 at 10:50
yes but Can it possible to abort a request in onStepTimeout function? – Suraj Dalvi Oct 12 '15 at 11:18
I don't know. It looks like another question. – Jonatas Walker Oct 12 '15 at 11:21

Continue with the next URL on timeout in CasperJS with eachThen and thenOpen

1 Answers1