wget or curl: format progress bar

Question

I am writing an update script like that:

update.sh

#!/bin/bash

sudo echo

printf "### PACMAN\n"   # I am running Arch Linux
sudo pacman -Syu

printf "\n### YAY\n"
yay -Syua

printf "\n### CUSTOM\n"
custom_script.sh # code below

custom_script.sh

#!/bin/bash

# [...]

wget --quiet --show-progress <some link>

# [...]

The output looks something like this:

### PACMAN
<...>
:: Retrieving packages...
 linux-lts-4.14.88-1-x86_64                     60,8 MiB   725K/s 01:26 [########################################] 100%
<...>

### YAY
:: Searching AUR for updates...
 there is nothing to do

### CUSTOM
:: Downloading custom updates...
somefile.txt                  100%[================================================>]  20,92M  3,83MB/s    in 5,6s

Is there a way to make the wget command in custom_script.sh format the progress bar the same way as pacman? I'm open to use curl or some other download tool as well.

Desired output:

### PACMAN
<...>
:: Retrieving packages...
 linux-lts-4.14.88-1-x86_64                     60,8 MiB   725K/s 01:26 [########################################] 100%
<...>

### YAY
:: Searching AUR for updates...
 there is nothing to do

### CUSTOM
:: Downloading custom updates...
 somefile.txt                                   20,9 MiB   3,8M/s 00:05 [########################################] 100%

May be you need to look at this - https://github.com/raoddhjav/progressbar. Could be exactly what you need — Inian, Dec 18 '18 at 16:40
@Inian: Thanks you very much, this helped a lot as you can see in my post below :) There is a little error in the link though: it's roddhjav, not raoddhjav — maddingl, Dec 18 '18 at 23:59

hanshenrik · Answer 1 · 2018-12-18T20:44:22.750

i think that sounds weird as hell, but i suppose you could parse out the response of wget with regex, then re-format the information however you want before printing it to the console... here is a proof-of-concept in php-cli:

#!/usr/bin/env php
<?php
$args = $argv;
unset ( $args [0] );
$args = implode ( " ", array_map ( 'escapeshellarg', $args ) );
$wget_cmd = "wget --quiet --show-progress --progress=bar:force {$args} 2>&1";
$wget = popen ( $wget_cmd, "rb" );
$format = <<<'FORMAT'
%filename% %separator_filename_percent% %total_downloaded%    %speed% %eta% %percent_indicator% %percent%%
FORMAT;
$format = trim ( $format );
while ( ! feof ( $wget ) ) {
    $line = stream_get_line ( $wget, 4096, "\r" );
    $match = preg_match ( '/^(?<filename>.*?)(?<separator_filename_percent>[ ]{3,})(?<percent>\d+)\%(?<percent_indicator>.*?])\s+(?<total_downloaded>\S+)\s+(?<speed>\S+)\s*(?:eta\s*)?(?<eta>.*)?$/', $line, $matches );
    if (! $match) {
        echo $line;
        if (strlen ( $line ) < 4096 && ! feof ( $wget )) {
            echo "\r";
        }
    } else {
        // var_dump ( $matches );
        echo strtr ( $format, array (
                '%filename%' => $matches ['filename'],
                '%separator_filename_percent%' => $matches ['separator_filename_percent'],
                '%total_downloaded%' => $matches ['total_downloaded'],
                '%speed%' => $matches ['speed'],
                '%eta%' => $matches ['eta'],
                '%percent_indicator%' => str_replace ( "=", "#", $matches ['percent_indicator'] ),
                '%percent%' => $matches ['percent'] 
        ) ), "     ", "\r";
    }
    // sleep(3);
}
pclose ( $wget );

output looks like

(but is very configurable by editing $format)

i think the whole thing could be re-written with sed (and i assume you'd prefer a sed dependency over a php-cli dependency), but i don't know how.

score 1 · Accepted Answer · answered Dec 18 '18 at 23:57

Thanks to the link Inian provided (https://github.com/roddhjav/progressbar), a similar approach as hanshenrik and a lot of googling and trial-and-error, I came up with the following solution:

custom_script.sh

#!/bin/bash

source progressbar.sh || exit 1

reformat_and_output_line() {
    COLS=()
    for val in $1 ; do
            COLS+=("$val")
    done
    if [[ ${#COLS[@]} == 9 ]]; then
        RELATIVE_PROGRESS=${COLS[6]%?}
        ABSOLUTE_PROGRESS=$(numfmt --from=iec --to=iec-i --format "%.1fB" ${COLS[0]} |
                            sed 's/\([^[:blank:]]\)\([[:upper:]]\)/\1 \2/')
        SPEED=$(printf "%+8s" "${COLS[7]}/s")
        TIME=$(printf "%+5s" ${COLS[8]})
        progressbar "somefile.txt" "$RELATIVE_PROGRESS" 100 "$ABSOLUTE_PROGRESS" "$SPEED" "$TIME"
    elif [[ ${#COLS[@]} == 7 ]]; then
        RELATIVE_PROGRESS=${COLS[5]%?}
        ABSOLUTE_PROGRESS=$(numfmt --from=iec --to=iec-i --format "%.1fB" ${COLS[0]} |
                            sed 's/\([^[:blank:]]\)\([[:upper:]]\)/\1 \2/')
        SPEED=$(printf "%+8s" "$(echo ${COLS[6]} | cut -d= -f1 )/s")
        TIME=$(printf "%+5s" $(echo ${COLS[6]} | cut -d= -f2 ))
        progressbar "somefile.txt" "$RELATIVE_PROGRESS" 100 "$ABSOLUTE_PROGRESS" "$SPEED" "$TIME"
    fi
}

wget_like_pacman() {
    last_output_time=$(( $(date +%s%3N) - 500 ))
    wget --quiet --show-progress $1 2>&1 | (
        while IFS= read -r line; do
            if [[ $(date +%s%3N) > $(( $last_output_time + 500 )) ]]; then
                reformat_and_output_line "${line}"
                last_output_time=$(date +%s%3N)
            fi
        done
        printf "\r"
        reformat_and_output_line "${line}"
        echo
    )
}

# [...]

wget_like_pacman <some link>

# [...]

The time format is not the exact same, but apart from that, I think this is pretty accurate.

One can also omit the if block checking the last_output_time, but then the terminal goes crazy and there is no way to actually read any of the values since they are updated too fast.

wget or curl: format progress bar

2 Answers2