Use number of downloaded file in wget as variable in bash script

Question

Can I get the number of downloaded files with wget -r and use that as a variable? I want to write a script that runs multiple wget commands in it (using -q so i can control output) and then at the end, add up the number of downloaded files and echo that to the user. Is this possible?

Answer 1

You could simply direct all the output to a file with the -a flag to send output to a file, then grep the logfile and count the occurrences of the word "saved":

root@demo:tmp# wget -r http://www.google.com/index.html -a logfile
root@demo:tmp# grep saved logfile |wc -l
1

Answer 2

To build on Jed's suggestion and take it the rest of the way, you could do something like this:

MY_VARIABLE=$(wget -r http://foo 2>&1 | grep -c 'saved')

Or, if you wanted to go a touch more complicated, but more reliable and robust, you could pull out the download count given as the summary by wget:

MY_VARIABLE=$(wget -r -nv http://foo 2>&1 | awk '/^Downloaded:/ {print $2}')

Update: Regarding the 2>&1 construct, for a *nix application, there are two default output streams, stdout (file descriptor 1) and stderr (file descriptor 2). For interactive commands, both are typically sent to the screen/terminal, so it can be hard to differentiate them. The 2>&1 tells your bash to take the output sent to stderr (fd 2) and combine it with the output for stdout (fd 1). The result is a single output stream, which would look to the next tool in a pipe (as above) the same as the output looks to your screen.

This is required because wget defaults to sending it's status information to stderr, not stdout. If you ran the above without the stderr redirection, you're sending stdout to the piped program, but that's all. You'd get all of wget's output sent to your screen (stderr), and the variable would be empty, because the grep or awk statements are processing stdout (which got no output).

To see what gets sent where for an application, you can do something like this:

wget -r -nv http://foo 2>foo.err 1>foo.out

Or just:

wget -r -nv http://foo 2>foo.err >foo.out

If you don't specify a number for output redirection, it defaults to stdout.

The files foo.err and foo.out will contain the contents of whatever your commands end to stderr and stdout, respectively.

Also worth noting, you can also use |& as your pipe command to combine stderr with stdout, as a shorthand for 2>&1.

For a little more information on the command redirection and stdin, stdout, and stderr, see the bash man page, the stdin, stdout, and stderr man pages, and for historical reasons behind stdout and stderr, see this posting: http://jstorimer.com/2011/12/29/the-difference-between-stdout-and-stderr.html