Perl WWW::Mechanize Storing urls in a hash unless it has already been found

Question

I have a question I'm hoping you could help with?

foreach my $url ( keys %{$newURLs} ) {
  # first get the base URL and save its content length
  $mech->get($url);
  my $content_length = $mech->response->header('Content-Length');

  # now iterate all the 'child' URLs
  foreach my $child_url ( @{ $newURLs->{$url} } ) {
    # get the content
    $mech->get($child_url);

    # compare
    if ( $mech->response->header('Content-Length') != $content_length ) {
         print "$child_url: different content length: $content_length vs "
         . $mech->response->header('Content-Length') . "!\n";
         #HERE I want to store the urls that are found to have different content 
         #lengths to the base url
         #only if the same url has not already been stored
    } elsif ( $mech->response->header('Content-Length') == $content_length ) {
         print "Content lengths are the same\n";
         #HERE I want to store the urls that are found to have the same content 
         #length as the base url
         #only if the same url has not already been stored
    }
  }
}

The problem I am having:

As you can see in the code above I want to store the urls depending on if the content lengths are the same or different, so I will end up with a group of urls that had a different content length to their base url and I will end up with another group of urls that had the same content length to their base url.

I know how to do this easily using an array

push (@differentContentLength, $url);
push (@sameContentLength, $url);

But how would I go about this using a hash (or another preferred method)?

I am still getting to grips with hashes so your help will be much appreciated,

thanks a lot

You should add the closing brackets to your loops. – simbabque Feb 13 '13 at 12:18 — simbabque, Feb 13 '13 at 12:18
@simbabque - yes your right, apologies – yonetpkbji Feb 13 '13 at 12:25 — yonetpkbji, Feb 13 '13 at 12:25

score 1 · Answer 1 · answered Feb 13 '13 at 11:11

Please check this solution:

my %content_length;

foreach my $url ( keys %{$newURLs} ) {
  # first get the base URL and save its content length
  $mech->get($url);
  my $content_length = $mech->response->header('Content-Length');

  # now iterate all the 'child' URLs
  foreach my $child_url ( @{ $newURLs->{$url} } ) {
    # get the content
    $mech->get($child_url); 
    my $new_content_length =  $mech->response->header('Content-Length');
    # store in hash
    print "New URL! url: $child_url\n" if ! defined $content_length{$child_url};
    print "Different content_length! url: $child_url, old_content_length: $content_length, new_content_length: $new_content_length\n" if $new_content_length != $content_length{$child_url};
    $content_length{$child_url} = $new_content_length;
  }
}

score 1 · Accepted Answer · answered Feb 13 '13 at 12:25

You can create a hashref to store all the urls for you outside of the loop. Let's call it $content_lengths. It's a scalar because it is a reference to a hash. In your $child_url loop, add the content length to that datastructure. We'll use the base url first, giving us another hashref inside $content_lengths->{$url}. There we decide if we want equal or different. Inside of these two keys there will be another hashref holding the $child_urls. They in turn have their content lengths as values. Of course we could just say ++ here if you don't want the length to be stored.

my $content_lengths; # this is at the top
foreach my $url ( # ... more stuff

# compare
if ( $mech->response->header('Content-Length') != $content_length ) {
  print "$child_url: different content length: $content_length vs "
    . $mech->response->header('Content-Length') . "!\n";

  # store the urls that are found to have different content
  # lengths to the base url only if the same url has not already been stored
  $content_lengths->{$url}->{'different'}->{$child_url} = $mech->response->header('Content-Length');

} elsif ( $mech->response->header('Content-Length') == $content_length ) {
  print "Content lengths are the same\n";

  # store the urls that are found to have the same content length as the base
  # url only if the same url has not already been stored
  $content_lengths->{$url}->{'equal'}->{$child_url} = $mech->response->header('Content-Length');
}

When you say use '++ here if you don't want the length to be stored' is this how it should be written `$content_lengths->{$url}->{'different'}->{$child_url}++;`? and for clarification, what exactly is the `++` doing? — yonetpkbji, Feb 13 '13 at 14:04
@perl-user yes, that's what I mean. It is the increment-shorthand operator. It adds 1 to the var on the left and assigns it. So they will all have the value 1. If one of the urls is seen twice, the value will be 2. This is how counters and 'remember the name but don't care about how many' are implemented. You can access it with `keys`. Think of it like a `GROUP BY` in SQL. — simbabque, Feb 13 '13 at 15:53
Oh I see how that prevents duplicates now (used Data::Dumper), instead of adding another duplicate url in it just registers its presence by increasing the number assigned to that url, which doesnt matter because we are not interested in that part, thanks your above comment explained it well :) — yonetpkbji, Feb 13 '13 at 16:40
I have posted another question you will be able to help with and will be the final piece in me grasping hashes/hash references, if you could take I look I would be very grateful, thankyou very much — yonetpkbji, Feb 13 '13 at 17:24

Perl WWW::Mechanize Storing urls in a hash unless it has already been found

2 Answers2