How can I download link targets from a web site using Perl?

Question

I just made a script to grab links from a website, and in turn saves them into a text file.

Now I'm working on my regexes so it will grab links which contains php?dl= in the url from the text file:

E.g.: www.example.com/site/admin/a_files.php?dl=33931

Its pretty much the address you get when you hover over the dl button on the site. From which you can click to download or "right click save".

I'm just wondering on how to achieve this, having to download the content of the given address which will download a *.txt file. All from the script of course.

What is the question here? You made a script and now want it to only download certain URLs? Are you looking for a regexp? — Konerak, Jul 06 '10 at 11:39
I'm trying to figure out how you download the file associated with url. For example, on the website you get click 'dl' icon/button and your browser automatically downloads the file for you. ie: http://www.example.com/site/admin/a_files.php?dl=33931 would download "file1.txt" I'm just wondering how you can download the file in Perl. The regexp part is not a problem. Or have I missed a function that can do all of this with ease haha — eraldcoil, Jul 06 '10 at 11:44
[Crawling in Perl - A Quick Tutorial](http://www.cs.utk.edu/cs594ipm/perl/crawltut.html) — João Pereira, Jul 06 '10 at 11:39

Zaid · Accepted Answer · 2010-07-07T13:55:29.123

8

Make WWW::Mechanize your new best friend.

Here's why:

It can identify links on a webpage that match a specific regex (/php\?dl=/ in this case)
It can follow those links through the follow_link method
It can get the targets of those links and save them to file

All this without needing to save your wanted links in an intermediate file! Life's sweet when you have the right tool for the job...

Example

use strict;
use warnings;
use WWW::Mechanize;

my $url  = 'http://www.example.com/';
my $mech = WWW::Mechanize->new();

$mech->get ( $url );

my @linksOfInterest = $mech->find_all_links ( text_regex => qr/php\?dl=/ );

my $fileNumber++;

foreach my $link (@linksOfInterest) {

    $mech->get ( $link, ':contentfile' => "file".($fileNumber++).".txt" );
    $mech->back();
}

edited Jul 07 '10 at 13:55

answered Jul 06 '10 at 11:55

Zaid

36,680
16
86
155

Awesome! you stated all the things I have been looking for, for the past 2 hours lol. Thank you :D – eraldcoil Jul 06 '10 at 11:58
This helped alot. Thank you very much :D. I have so much to learn still, thnx for pointing out this very helpful module :D – eraldcoil Jul 06 '10 at 12:27
I see no reason in this example to do the ->back() and ->reload(). – Andy Lester Jul 06 '10 at 16:14
@Andy : I suppose it depends on the page in question. If it updates frequently, a `reload()` may be prudent. – Zaid Jul 06 '10 at 16:17
@Zaid: You're not doing anything with the reloaded page. @linksofInterest doesn't change. – Andy Lester Jul 07 '10 at 13:31
@Andy : Good point. The `->reload()` is useless for the example in question. – Zaid Jul 07 '10 at 13:55

score 5 · Answer 2 · answered Jul 06 '10 at 11:56

5

You can download the file with LWP::UserAgent:

my $ua = LWP::UserAgent->new();  
my $response = $ua->get($url, ':content_file' => 'file.txt');

Or if you need a filehandle:

open my $fh, '<', $response->content_ref or die $!;

answered Jul 06 '10 at 11:56

Eugene Yarmash

142,882
41
325
378

2

Or, just use 'LWP::Simple::getstore($url, $file)`. – Sinan Ünür Jul 06 '10 at 12:37

score 0 · Answer 3 · answered Oct 30 '13 at 12:57

Old question, but when I'm doing quick scripts, I often use "wget" or "curl" and pipe. This isn't cross-system portable, perhaps, but if I know my system has one or the other of these commands, it's generally good.

For example:

#! /usr/bin/env perl
use strict;
open my $fp, "curl http://www.example.com/ |";
while (<$fp>) {
  print;
}

How can I download link targets from a web site using Perl?

3 Answers3