How to handle downloads with phantomjs/casperjs?

Question

Is it possible to download a file to a folder and give it a specific name with panthomjs/casperjs ?

For instance how can I download the .csv at the bottom of this page : http://www.nasdaq.com/symbol/aapl/historical and name it aapl.txt ?

The download link is :

<a href="javascript:getQuotes(true);" id="lnkDownLoad">
                Download this file in Excel Format
            </a>

Its goal is to call a javascript function whose goal is to obfuscate the direct download link (I think) but when you click on it it summons a classic download prompt. I would like phantomjs to handle that download normally (to change the name of the file and chose where to save it on the drive disk)

Edit : This code is supposed to click on the download link and listen for incoming ressources :

var casper = require('casper').create();

var x = require('casper').selectXPath;

casper.userAgent("Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2049.0 Safari/537.36")

casper.start('http://www.nasdaq.com/symbol/aapl/historical', function () {
    //this.echo(this.getTitle());
    console.log('TITLE : ' + this.getTitle());

});

casper.wait(5000, function() {
    casper.on('resource.received', function (resource) {
        casper.echo("LISTENING");
        casper.echo(resource.url);
    });
});

casper.thenClick(x('//*[@id="lnkDownLoad"]'), function() {
    console.log('CLICKED');
});



casper.run();

But for some reason I dont receive any file unlike on a normal browser. The console log is : b'TITLE : (AAPL) Historical Prices & Data - NASDAQ.com\r\nCLICKED\r\nLISTENING\r\nhttp://www.nasdaq.com/symbol/aapl/historical\r\n'

Any idea ?

Answer 1

When you look into the code, you see that it is not really obfuscated. By clicking on the download link, the file is actually downloaded through casper, but it cannot be easily accessed. The culprit is PhantomJS, because PhantomJS does not expose the request and response contents (see page.onResourceReceived), but only meta data.

It means that you need to repeat the request through the download function. When you look into the page source in your browser's developer tools, you see that getQuotes(true) is called on click. By searching for getQuotes (Ctrl+Shift+F in Chrome), you find the function in question.

By analysing the function, you could arrive at the conclusion that $("#getFile").submit(); is the actual download which is just a POST request from a form with a lot of hidden values. If you look closely into getQuotes, you see that the function also adds one of the hidden values to the form. This means that you need to call getQuotes before faking the submission of the form.

The actual faking is relatively easy. The first thing is to generate the request object from the form to be used in the POST request and second thing is finding out the URL of the request. Here is the complete code:

var casper = require('casper').create();
var x = require('casper').selectXPath;

casper.userAgent("Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2049.0 Safari/537.36")

casper.start('http://www.nasdaq.com/symbol/aapl/historical');
casper.wait(5000); // probably not necessary
casper.thenClick('#lnkDownLoad');
casper.then(function(){
    var parameters = this.evaluate(function(){
        // from http://stackoverflow.com/a/2403206
        var paramObj = {};
        $.each($('#getFile').serializeArray(), function(_, kv) {
            paramObj[kv.name] = kv.value;
        });
        return paramObj;
    });
    var url = this.getElementAttribute('#getFile', 'action');
    this.download(url, 'aapl.csv', 'POST', parameters);
});
casper.run();