Stack Overflow
Stack Overflow

Questions tagged [www-mechanize]

WWW::Mechanize is a Perl module for programmatic web browsing, used for automating interaction with websites.

Features include:

  • All HTTP methods
  • High-level hyperlink and HTML form support, without having to parse HTML yourself
  • SSL support
  • Automatic cookies
  • Custom HTTP headers
  • Automatic handling of redirections
  • Proxies
  • HTTP authentication

Mech supports performing a sequence of page fetches including following links and submitting forms. Each fetched page is parsed and its links and forms are extracted. A link or a form can be selected, form fields can be filled and the next page can be fetched. Mech also stores a history of the URLs you've visited, which can be queried and revisited.

See the home page for complete details.

366 questions
2
votes
1 answer

WWW::Mechanize error: Error GETing url Protocol scheme 'https' is not supported (LWP::Protocol::https not installed)

I am very new to perl. I am using it at work, and I got the following error yesterday, for a script that was running just fine on the previous day. Error GETing https://www.someurl.com:443: Protocol scheme 'https' is not supported…
Amruta
  • 1,295
  • 1
  • 13
  • 24
2
votes
1 answer

Saving partial content when WWW::Mechanize GET times out

I'm using the following Perl code to get data from https://www.otcmarkets.com/research/stock-screener/api?sortField=symbol&sortOrder=asc&page=0&pageSize=20000: use warnings; use WWW::Mechanize::GZip; my $TempFilename = "D:\\temp\\test.txt"; my…
kangarootaco
  • 61
  • 3
2
votes
1 answer

Most generic mobile User-Agent header string

I use LWP::UserAgent to access the web site. The site offers almost the same info in auto-selected layouts for pc (default?) and mobile. Some informations are easier to get in non default layout. Which User-Agent most generic strings are…
AnFi
  • 10,493
  • 3
  • 23
  • 47
2
votes
2 answers

Mechanize Basic Questions

I am working with this website: http://bioinfo.uni-plovdiv.bg/microinspector/ And from the mech-dump, I get POST http://bioinfo.uni-plovdiv.bg/microinspector/cgi/result-new1.plx target_name= (text) target_sequence= …
chickenNuggets
  • 25
  • 2
2
votes
0 answers

Perl WWW::Mechanize for HTTPS over proxy

I am trying to scrape a website using WWW::Mechanize module. I have configured the mechanize agent with a proxy URL and setting the proxy credentials using credentials method. Code snippet : my $url = 'https://abcde.com'; my $proxy_username =…
Aniket Golatkar
  • 39
  • 2
2
votes
0 answers

perl::mechanize load cookies from file

Ladies and gentleman, i have a problem which really drives me nuts. I want to read the cookies from a Chrmoe Cookies file to add the to a request with perl::mechanize. This is the code #!/usr/bin/perl use strict; use warnings; use…
Bananaman
  • 21
  • 2
2
votes
1 answer

Mechanize example - quiete simple but too complex for me: need explanations

Good day dear community. I am new to programming. And i want to digg deeper into Perl. So i have a Mechanize example - quiete simple but too complex for me: need explanations. I need your help here with this! use strict; $|++; use…
zero
  • 1,003
  • 3
  • 20
  • 42
2
votes
1 answer

Saving files with WWW::Mechanize

I'm trying to programmatically scrape the files from this page: https://olms.dol-esa.gov/query/getYearlyData.do (yes it probably would be faster to download them manually but I want to learn how to do this). I have the following bit of code to try…
StevieD
  • 6,925
  • 2
  • 25
  • 45
2
votes
1 answer

WWW::Mechanize - fill in form fields with unkown name

With WWW::Mechanize I want to submit a form. The problem is, that every time the form loads, the input field names are different. The only…
Peter
  • 2,051
  • 1
  • 15
  • 20
2
votes
1 answer

Setting basic authentication credentials in WWW::Mechanize

I'm having trouble using basic authentication in WWW::Mechanize. I'm trying to connect to the Streak API, the documentation for which states: Streak uses HTTP Basic Auth to sign each request with your API key. Simply set the username of the request…
Scott Martin
  • 1,260
  • 2
  • 17
  • 27
2
votes
1 answer

Handling GET errors in WWW::Mechanize

I'm using a script that scrapes data from a website using WWW::Mechanize and it's all working great, except for the website itself. Sometimes it just doesn't respond for a short moment and for a given my $url =…
yoniyes
  • 1,000
  • 11
  • 23
2
votes
1 answer

Is it possible to take a webpage's screenshot without a graphical interface?

I have interest to programmatically take a screenshot of a webpage using perl. I have come across this solution: use WWW::Mechanize::Firefox; use Path::Class qw/file/; my $mech = WWW::Mechanize::Firefox->new( bufsize => 10_000_000, # PNGs might…
KingsInnerSoul
  • 1,373
  • 4
  • 20
  • 49
2
votes
1 answer

Fetch text/event-stream web response using WWW::Mechanize or LWP::UserAgent

I'm using WWW::Mechanize to fetch a web page that includes a Google Maps widget that receives constant data from a single response of type text/event-stream. That kind of response is like a never ending response from the server that constantly…
Francisco Zarabozo
  • 3,676
  • 2
  • 28
  • 54
2
votes
1 answer

Can't call method "header" on an undefined value at WWW/Mechanize.pm line 2566

I am just doing a testing using WWW::Mechanize module on Facebook, when I try to run the code below, it return me an error Can't call method "header" on an undefined value at C:/Strawberry/perl/vendor/lib/WWW/Mechanize.pm line…
Zachary
  • 75
  • 8
2
votes
2 answers

Close Tab Based on Title

How would I close a tab based on it's title using WWW::Mechanize::Firefox? Here is what I currently have: my $ff = Firefox::Application->new(); my @tab_info = $ff->openTabs(); foreach my $tab (@tab_info) { if($tab->{title} eq "TITLE HERE") { …
Bijan
  • 7,737
  • 18
  • 89
  • 149
1 2 3
…
24 25