9

I'm trying to get the response of a curl call into a variable in perl. 

my $foo = `curl yadd yadda`;

print $foo;

does not work. When I run this at the command line the curl call prints all its output correctly in the terminal, but the variable is not filled with that data.

Is there a way to do this without installing and calling the Perl curl lib?

Eddie
  • 53,828
  • 22
  • 125
  • 145
Yevgeny Simkin
  • 27,946
  • 39
  • 137
  • 236

9 Answers9

13

It probably sends its stuff to stderr. Try

my $foo = `curl yadd yadda 2>&1`;
sunny256
  • 9,246
  • 2
  • 25
  • 22
  • That did it! thanks!. I'll give you the checkmark if you can add a sentence or two about what 2>&1 actually does and why moving it from stderr to whatever &1 is dumps it into the variable. Thanks very much! – Yevgeny Simkin Jun 18 '09 at 22:07
  • 2
    2>foo means "redirect stderr to file foo". &1 means stdout, i.e., redirect stderr to stdout. You can also do it the other way: ">&2" which means "redirect stdout to stderr". To redirect both stdout and stderr, you can use ">&", i.e. foocommand &>file_with_boththings.txt – sunny256 Jun 18 '09 at 22:14
  • The '2>&1' notation means that file descriptor 2 (stderr) is sent to the same place that file descriptor 1 (stdout) is sent to. So both standard output and standard error are written to the same place (standard output). And Perl's backticks capture the standard output of the executed command, hence... – Jonathan Leffler Jun 18 '09 at 22:15
  • http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-3.html and http://tldp.org/LDP/abs/html/io-redirection.html has some nice info about it also. – sunny256 Jun 18 '09 at 22:16
5

You also might consider looking at LWP::UserAgent or even LWP::Simple.

jiggy
  • 3,828
  • 1
  • 25
  • 40
4

What do you really want to do? Use curl at all costs, or grab the contents of a web page?

A more perlish way of doing this (which relies on no external programs that may or may not be installed on the next machine where you need to do this) would be:

use LWP::Simple;

my $content = get("http://stackoverflow.com/questions/1015438/")
   or die "no such luck\n";

If you want to see why the GET failed, or grab multiple pages from the same site, you'll need to use a bit more machinery. perldoc lwpcook will get you started.

dland
  • 4,319
  • 6
  • 36
  • 60
3

In the shell 2> means redirect fileno 2. Fileno 2 is always what a program sees as stderr. Similarly, fileno 0 is stdin and fileno 1 is stdout. So, when you say 2>&1 you are telling the shell to redirect stderr (fileno 2) into stdout (fileno 1). Since the backticks operator uses the the shell to run the command you specify, you can use shell redirection, so

my $foo = `curl yadda yadda 2>&1`;

is telling curl to redirect its output into stdout, and since the backtick operator catches stdout, you get what you were looking for.

Chas. Owens
  • 64,182
  • 22
  • 135
  • 226
  • thanks... your explanation is morely betterlier worded than the one I picked as the right answer, so, I'm giving it an uptick :) – Yevgeny Simkin Jun 18 '09 at 22:54
1

Very old post, but the real way of using curl in backticks is using the appropriate switch of curl.

This switch is -o which defines where to send the output.

More from the curl man page:

Specifying the output as '-' (a single dash) will force the output to be done to stdout.

This also prevents having possible errors in $foo, which would happen if you redirect the complete STDERR to STDOUT on errors:

my $foo = `curl -o - yadd yadda`;
Jimmy Koerting
  • 1,231
  • 1
  • 14
  • 27
0

Try this:

$var = `curl "http://localhost" 2>/dev/null`; 
print length($var)

curl displays progress information on stderr, redirecting that to /dev/null makes it easier to see what's going on.

brian-brazil
  • 31,678
  • 6
  • 93
  • 86
0

This works on my system:

#!/usr/bin/perl

use strict;
use warnings;

my $output = `curl www.unur.com`;

print $output;

__END__

C:\> z1

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd"><html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

etc.

Sinan Ünür
  • 116,958
  • 15
  • 196
  • 339
  • I think you may be misinterpreting the printing of the curl for the print statement you have there. Try print "OUTPUT=$output" instead and see if you get the right result. I did that too and it prints the curl output and then prints "OUTPT=" cause it never got the result into the var. However, the answer above is correct. – Yevgeny Simkin Jun 18 '09 at 22:12
  • @Dr.Dredel if I remove the print statement, I see the progress information which curl prints to stderr, but not the output of curl. If I put print "OUTPUT=$output", I get OUTPUT= – Sinan Ünür Jun 18 '09 at 22:31
  • perhaps it's some sort of platform discrepancy. It definitely does not print to stdout for me, and the answer I checked did fix the issue. – Yevgeny Simkin Jun 18 '09 at 22:52
  • http://curl.haxx.se/docs/manpage.html -o/--output Write output to instead of stdout. Maybe this has to do with the 'yadd yadda' you are not showing. – Sinan Ünür Jun 18 '09 at 23:30
0

You can open a pipe as if it were a file.

$url = "\"http://download.finance.yahoo.com/d/quotes.csv?s=" . 
"$symbol&f=sl1d1t1c1ohgvper&e=.csv\"";

open CURL, "curl -s $url |" or die "single_stock_quote: Can't open curl $!\n";
$line = <CURL>;
close CURL;
jpenkethman
  • 1
  • 2
-1

It might be that some of the output you want to capture is in standard err, not standard out. Try this: 

my $foo = system "curl http://www.stackoverflow.com";
print $foo;
pjb3
  • 5,191
  • 5
  • 25
  • 44