7

I need to download several files with wget and measure download speed.

e.g. I download with

wget -O /dev/null http://ftp.bit.nl/pub/OpenBSD/4.7/i386/floppy47.fs http://ftp.bit.nl/pub/OpenBSD/4.7/i386/floppyB47.fs

and the output is

--2010-10-11 18:56:00--  http://ftp.bit.nl/pub/OpenBSD/4.7/i386/floppy47.fs
Resolving ftp.bit.nl... 213.136.12.213, 2001:7b8:3:37:20e:cff:fe4d:69ac
Connecting to ftp.bit.nl|213.136.12.213|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1474560 (1.4M) [text/plain]
Saving to: `/dev/null'

100%[==============================================================>] 1,474,560    481K/s   in 3.0s

2010-10-11 18:56:03 (481 KB/s) - `/dev/null' saved [1474560/1474560]

--2010-10-11 18:56:03--  http://ftp.bit.nl/pub/OpenBSD/4.7/i386/floppyB47.fs
Reusing existing connection to ftp.bit.nl:80.
HTTP request sent, awaiting response... 200 OK
Length: 1474560 (1.4M) [text/plain]
Saving to: `/dev/null'

100%[==============================================================>] 1,474,560    499K/s   in 2.9s

2010-10-11 18:56:06 (499 KB/s) - `/dev/null' saved [1474560/1474560]

FINISHED --2010-10-11 18:56:06--
Downloaded: 2 files, 2.8M in 5.9s (490 KB/s)

I need to grep the total download speed, that is, the string 490 KB/s. How do I do this?

P.S. May need to account for the case that we will actually download only one file, so there won't be final output starting with FINISHED

Nickolai Leschov
  • 219
  • 1
  • 2
  • 9
  • `wget 'https://x.com' -O /dev/null 2>&1 | grep -oP '(?<= \()\d+\.?\d+ \SB/s(?=\) )'` explanation, save to null, reroute err to out, output matching only, use hardcore regex, positive lookbehind, positive lookahead – Ray Foss Sep 11 '20 at 02:23

6 Answers6

4

Update, a grep-style version using sed:

wget ... 2>&1 | sed -n '$,$s/.*(\(.*\)).*/\1/p'

Old version:

I thought, it's easier to divide the file size by the download time after the download. ;-)

(/usr/bin/time -p wget ... 2>&1 >/dev/null; ls -l newfile) | \
awk '
   NR==1 {t=$2};
   NR==4 {printf("rate=%f bytes/second\n", $5/t)}
'

The first awk line stores the elapsed real time of "real xx.xx" in variabe t. The second awk line divides the file size (column 5 of ls -l) by the time and outputs this as the rate.

Peter G.
  • 14,786
  • 7
  • 57
  • 75
  • Not easier, it seems to me. I have to determine both file size and the download time, divide, and what if time equals zero? Please provide the example how to do it e.g. with bash – Nickolai Leschov Oct 11 '10 at 22:53
  • Reported time zero needs special attention. It seems grepping is perhaps easier. Somehow I had the dynamical and progressive output of wget in mind which I thought hard to grep. – Peter G. Oct 11 '10 at 23:34
2

This works when only 1 file is being downloaded.

I started using sed to get the speed from wget, but I found it irritating so I switched to grep.

This is my command:

wget ... 2>&1 | grep -o "[0-9.]\+ [KM]*B/s"

The -o option means it only returns that part. It matches 1 or more of the 10 digits then a space. Then optionally K or M before the B/s

That will return 423 KB/s (for example).

To grep for just the units, use grep -o "[KM]*B/s" and for just the number use grep -o "[0123456789]\+.

Tim
  • 2,563
  • 1
  • 23
  • 31
  • I would remove the star from the second character class, you only need exactly one K or M. Also note that there are several occurances of speeds in the output, not all of which are of interest to the OP; a complete solution needs to take context into account, which afaik cannot be done by a single `grep call`. – cmaster - reinstate monica Dec 07 '14 at 15:35
  • That only gave 1 output for me... :/ He's got multiple ones but he said that he wouldn't have 2 files...? I was only using 1 file. – Tim Dec 07 '14 at 15:37
2

This worked for me, using your wget -O /dev/null <resource>

The regex I used was \([0-9.]\+ [KM]B/s\)

But note I had to redirect stderr onto stdout so the command was:

wget -O /dev/null http://example.com/index.html 2>&1 | grep '\([0-9.]\+ [KM]B/s\)'

This allows things like 923 KB/s and 1.4 MB/s

grep just finds matches. To get the value(s) you can use sed instead:

wget -O /dev/null http://example.com/index.html 2>&1 |
    sed -e 's|^.*(\([0-9.]\+ [KM]B/s\)).*$|\1|'
Stephen P
  • 14,422
  • 2
  • 43
  • 67
  • For me, it returns the whole line that contains the speed. But how do I get only what's inside the brackets? e.g. `923 KB/s` or `1.4 MB/s`? – Nickolai Leschov Oct 11 '10 at 22:47
  • Now, the example with `sed` does work - it extracts the speed (`923 KB/s`), but before that it prints out all the other output as well. – Nickolai Leschov Oct 14 '10 at 22:14
  • @Nick: combine them. Unix shell is all about pipelining - `wget abc | grep ghi | sed xyz` – Stephen P Oct 14 '10 at 23:21
  • 2
    @NickolaiLeschov You can pass the `--only-matching` (`-o`) option to grep to get the just the string inside the brackets. – pix Mar 02 '16 at 07:34
  • ... im sure there is reason to the insanity. why stderr. oh well. – Ray Foss Sep 11 '20 at 02:12
0

For example, get speed in MBit per second (by adding --report-speed=bits for wget, and small change grep pattern):

wget -O /dev/null --report-speed=bits http://www.ovh.net/files/10Mb.dat 2>&1 | grep -o "[0-9.,]\+ [KM]*[Bb]/s"

answer:

1,51 Mb/s
polonez
  • 1
-1

Why can't you just do this:

perl -ne "/^Downloaded.*?\((.*?)\)/; print $1"
ennuikiller
  • 46,381
  • 14
  • 112
  • 137
-3

here's suggestion. You can make use of wget's --limit-rate=amount option. For example,

--limit-rate=400k will limit the retrieval rate to 400KB/s. Then its easier for you to calculate the total speed. Saves you time and mental anguish trying to regex it.

ghostdog74
  • 327,991
  • 56
  • 259
  • 343