3

I'm using WWW::Mechanize and HTML::TokeParser to parse a website for updates. I cannot give any details on the website because it requires a login. The website essentially has a table of data. I'm simply parsing the html till I get to the first row of the table, check if it the value of my last scrape, if not send a mail. This works perfectly well when I test it out on existing table entries, except, when actual updates happen, the scraping doesn't stop at my last scrape. It keeps sending mails until the table is exhausted and repeats this indefinitely. I cannot figure out what is happening. I know there isn't much anyone can verify without the website but I'm posting my code anyways. I'd appreciate ideas on what could be going wrong.

code:

sub func{
    my ($comid, $mechlink) = @_;

    my $mechanize = WWW::Mechanize->new(
        noproxy  => 0,
        stack_depth => 5,
        autocheck => 1
    );

    $mechanize->proxy( https => undef );
    eval{
            my $me = $mechanize->get($mechlink);
            $me->is_success or die $me->status_line;
    };
    return $comid if ($@);  

    my $stream = HTML::TokeParser->new( \$mechanize->{content} ) or die $!;

    while ( $tag = $stream->get_tag('td') ) {
    if( $tag->[1]{class} eq 'dateStamp' ) {
        $dt = $stream->get_trimmed_text('/td');
        $tag = $stream->get_tag;
        $tag = $stream->get_tag;
        $name = $stream->get_trimmed_text('/td') if( $tag->[1]{class} eq 'Name' );
        return $comid unless( $tag->[1]{class} eq 'Name' );
        $tag = $stream->get_tag;
        $tag = $stream->get_tag;
        $tag = $stream->get_tag;
        $tag = $stream->get_tag;
        $info = $stream->get_trimmed_text('/td');
        print "$name?\n";
        return $retval if($info eq $comid);
        print "You've Got Mail! $info $comid\n";
        $tcount++;
        $retval = $info if($tcount == 1);
        $tag = $stream->get_tag;
        $tag = $stream->get_tag;
        $tag = $stream->get_tag;
        $link = "http://www.abc.com".$tag->[1]{href} if ($tag->[0] eq 'a' );
        my $outlook = new Mail::Outlook();
        my $message = $outlook->create();
        $message->To('abc@def.com');
        $message->Cc('abc@def.com;abc@def.com');
        my $hd = "$name - $info";  
        $message->Subject($hd);
        $message->Body(" ");
        $message->Attach($link);
        $message->send;
    }
}
}
Aks
  • 5,188
  • 12
  • 60
  • 101
  • Can you include the code for your while loop - the bit where you check for updates. This is likely to be where things are going wrong. – Mike Jun 22 '11 at 07:40
  • I would advise you to add application logging so that you can check more from the logs. Furthermore a mail send counter and a controlled stop is good as well. I have sent several times 100s of emails from such a program and I know how annoying this can be. – weismat Jun 22 '11 at 07:41
  • I've added code in the loop. The issue with the mail send counter is that, firstly, I have no idea how many updates can be expected from scrape to scrape. Secondly, even if i do put a limit, i run the same script every 60 seconds, so the next iteration starts sending the mails all over again – Aks Jun 22 '11 at 08:15

5 Answers5

6

For this kind of tasks i prefer to use HTML::TableExtract . It's very easy to use:

use HTML::TableExtract;
$te = HTML::TableExtract->new( headers => [qw(header1 header2)]);
$te->parse($html);
foreach $ts ($te->tables) {
    foreach $row ($ts->rows) {
        my ($field1, $field2) = @$row;
        # Your code here
    }
}
Francisco R
  • 4,032
  • 1
  • 22
  • 37
2

Sometimes, the site have changes. I often use Web::Scraper. It can be possible to write to get element with XPath.

use Web::Scraper;
use URI;

my $uri = URI->new("http://....");
my $entries = scraper {
    process 'id("content")/div[@class="section"]', 'news[]' => scraper {
        process 'h2', title => 'TEXT';
        process 'p', body => 'TEXT';
    };
};

# if you have instance of WWW::Mechanize, set like following.
# $entries->user_agent($mech);

my $res = $entries->scrape( $uri );
for my $entry (@{$res->{news}}) {
    # use $entry->title or $entry->body
}
# language: lang-perl
mattn
  • 7,571
  • 30
  • 54
2

Exit from the while loop when you have matched what you were looking for, otherwise it keeps looping.

 while ( $tag = $stream->get_tag('td') ) {
    if( $tag->[1]{class} eq 'dateStamp' ) {
        $dt = $stream->get_trimmed_text('/td');
                    ...
                    ... 
        last;
    }
}
manu_v
  • 1,248
  • 2
  • 12
  • 21
1

You pass $comid to your function. In your while loop, you first set $info, and then compare this to $comid. If the two values match, you exit the function. If they do not match, you send an email.

Once the email has been sent, the loop continues, and processes the next tag. When you next compare $info and $comid, I guess that they will be different, as you have moved on to the next tag. Therefore another email will be sent.

I don't know if this is the intended behaviour - do you intend sending one email for each update in the table, or just one email if there have been any updates to the table? If you just need to send one email, no matter how many updates there have been, then simply exit the loop after the first email has been sent - as suggested by manu_v.

I would also look at refactoring your code so that it is more robust - all the get_tag calls seem a little flimsy. Check out the other answers for suggestions on ways to do this.

Mike
  • 21,301
  • 2
  • 42
  • 65
  • 1
    If I understand your code correctly, your compare a value from the table (`$info`) with a value passed to your function (`$comid`). If the value in the table has been updated, you need to remember that new value so that you can compare against it on the next scan. If you don't do that, you will always be comparing the table value (`$info`) against an old value (`$comid`) - it will always be out of date, and thus emails will be sent. I can't see where or how you are storing the updated value. You only return `$comid`, which is the value you passed to the function in the first place. – Mike Jun 23 '11 at 09:02
  • It may also help if you can post the HTML of the table that you are scanning. You can replace the actual values with dummy values if the data is sensitive. – Mike Jun 23 '11 at 09:02
  • Alright. I'll try to post the html. I assign `$retval` to the topmost entry for that particular scrape and `$retval` is returned when you reach the previous iterations topmost scrape. `$comid` recieves this as a return value from the function and is passed for the next iteration – Aks Jun 23 '11 at 09:50
  • I'd either step through the code with the debugger, or add a liberal sprinkling of print statements, to ensure that the correct value is being returned and that you are comparing the correct values. – Mike Jun 23 '11 at 09:57
  • The code works perfectly well when I test it out on existing entries. But at the instant that the update happens something wierd is happening. I'll use some of the other ways described and try. Thanks for your help – Aks Jun 23 '11 at 10:18
1

Sounds to me that this is more of a problem with loop termination than with TokeParser. It sounds like your loop is continuing to iterate even after you get the value you are looking for.

You may want to do something like:

While($x) {

  .
  .
  .
  last if ($foundWhatINeeded)
}
Matt King
  • 189
  • 8