Stack Overflow
Stack Overflow

Questions tagged [www-mechanize]

WWW::Mechanize is a Perl module for programmatic web browsing, used for automating interaction with websites.

Features include:

  • All HTTP methods
  • High-level hyperlink and HTML form support, without having to parse HTML yourself
  • SSL support
  • Automatic cookies
  • Custom HTTP headers
  • Automatic handling of redirections
  • Proxies
  • HTTP authentication

Mech supports performing a sequence of page fetches including following links and submitting forms. Each fetched page is parsed and its links and forms are extracted. A link or a form can be selected, form fields can be filled and the next page can be fetched. Mech also stores a history of the URLs you've visited, which can be queried and revisited.

See the home page for complete details.

366 questions
4
votes
1 answer

Using WWW::Mechanize, how do I add a lower case header with an underscore?

I'm using an API that requires me to use a header named "m_id" to the request. When I use $mech->add_header('m_id' => 'whatever') WWW::Mechanize (or rather HTTP::Headers) “helpfully” changes the header name to “M-Id”. Which doesn't work. Is there…
mscha
  • 6,509
  • 3
  • 24
  • 40
4
votes
2 answers

WWW::Mechanize follows a second redirect after six seconds

I am using Perl with the WWW::Mechanize module to submit a form to a webpage and save the result to a file. I know how to submit forms and save the data, but I can't save data after this six-second redirection. After the form is submitted, the page…
user3353029
  • 41
  • 1
4
votes
1 answer

perl dom mechanize xpath

I'm trying to scrape some data from metacriti* website using mechanize, but I'm getting no output Here's my code with a url example: my $metaURL = "http://www.metacriti*.com/game/pc/dota-2"; my $mech = WWW::Mechanize->new(); $mech->get($metaURL) or…
Enissay
  • 4,969
  • 3
  • 29
  • 56
4
votes
3 answers

Active perl Mechanize bad request error

I am trying to get https using Mechanize but failed with: use strict; use warnings; use IO::Socket::SSL; use WWW::Mechanize; my $mech = WWW::Mechanize->new; $mech->proxy(['https','http'],…
user2763829
  • 793
  • 3
  • 10
  • 20
4
votes
1 answer

Please help me debug an SSL issue using WWW::Mechanize (or LWP::UserAgent for that matter)

I'm using WWW::Mechanize to load the catalog from our product provider into our database. I run this script every 2 hours everyday and it completes in arround 12 minutes by using around 50 simultaneous threads. Everything was working perfectly,…
Francisco Zarabozo
  • 3,676
  • 2
  • 28
  • 54
4
votes
3 answers

Filling out form in Perl with the WWW::Mechanize module

I am creating a script that logs into a web form in Perl using the mechanize module, and I'm getting the error: syntax error at /home/arty/scripts/gmail_pw_chngr.pl line 18, near "button" Execution of /home/arty/scripts/gmail_pw_chngr.pl aborted…
user1664433
  • 63
  • 3
  • 9
4
votes
1 answer

Trouble following an image link with WWW::Mechanize

I'm working on a Perl script to get the "astronomy image of the day" and set it as my wallpaper. Then I would set a cronjob to do that everyday for me. But I'm having a hard time making the script follow the image link that leads to the full-sized…
XVirtusX
  • 679
  • 3
  • 11
  • 30
4
votes
1 answer

Using WWW::Mechanize Credentials

I am having trouble using the credentials agent for WWW:Mechanize to access and web scrape a site that requires NTLM authentication. I read online that the credentials agent requires 4 arguments: a base, realm, username and password. I'm not sure…
user1596292
  • 41
  • 4
4
votes
2 answers

Handling 404 and internal server errors with perl WWW::Mechanize

I am using WWW::Mechanize to crawl sites, and it works great except for sometimes it will hit a page that returns error code 404 or 500 (not found or internal server error), and then my script will just exit and stop running. This is really messing…
srchulo
  • 5,143
  • 4
  • 43
  • 72
3
votes
1 answer

Perl Mechanize - how to make a script running faster with less overhead

Problem: I have a list of 2500 websites and need to grab a thumbnail screenshot of them. How do I do that? I could try to parse the sites either with Perl.- Mechanize would be a good thing. Note: i only need the results as a thumbnails that are a…
zero
  • 1,003
  • 3
  • 20
  • 42
3
votes
3 answers

WWW::Mechanize text field issue

I'm trying to submit a form by post method using WWW::Mechanize perl module. use WWW::Mechanize; my $mech = WWW::Mechanize->new(); ... $mech->get($url); ... my $response = $mech->submit_form( form_name => $name, fields => { …
cur4so
  • 1,750
  • 4
  • 20
  • 28
3
votes
2 answers

WWW::Mechanize and strawberry perl

When I try to install WWW::Mechanize distribution from CPAN with strawberry perl ( v5.10.1) on windows 7 I get the below failure ? cpan> install JESSE/WWW-Mechanize-1.70.tar.gz Running make for J/JE/JESSE/WWW-Mechanize-1.70.tar.gz Checksum for…
smith
  • 3,232
  • 26
  • 55
3
votes
5 answers

Perl : Unexpected behavior with website scraping

I'm using WWW::Mechanize and HTML::TokeParser to parse a website for updates. I cannot give any details on the website because it requires a login. The website essentially has a table of data. I'm simply parsing the html till I get to the first row…
Aks
  • 5,188
  • 12
  • 60
  • 101
3
votes
2 answers

Find Favicons in HTML using Perl

I'm trying to look for favicons (and variants) for a given URL using Perl (I'd like to avoid using an external service such as Google's favicon finder). There's a CPAN module, WWW::Favicon, but it hasn't been updated in over a decade -- a decade in…
Timothy R. Butler
  • 1,097
  • 7
  • 20
3
votes
1 answer

Passing mechanized browser to subroutine (Perl with WWW::Mechanized)

I am still learning Perl so apologies on lack of technical term knowledge and general newbie-ness. I am trying to write a script to log into my course works account and download all the files. I can only stay in for about an hour until I have to…
msikd65
  • 427
  • 1
  • 5
  • 11
1 2
3
…
24 25