bash: progress output for rsync in a script

Question

this script is a part of a linux live-cd installer.

rsync -aq / /TARGET/ exclude-from=exclude.list &>> errors.log

i would like to report back to a gui the progress. the gui (gtkdialog) responds to any number 0-100 (echo 1; echo 2; etc...)

rsync -n (dry run) takes too long in this case.

i would like to run...

filesystem_size=`(get directory size) / exclude=exclude.list`
rsync -aq / /TARGET/ exclude-from=exclude.list &
while [ rsync is running ]; do
    (check size) /TARGET/
    compare to $filesystem_size
    echo $number (based on the difference of sizes)
done

please help with getting directory size with multiple excludes, while loop for while rsync is running, echo number (0-100) based on the difference of the two sizes.

answering any of the above is a great help, thank you.

EDIT: ADDING COMPLETED RSYNC PROGRESS OUTPUT (there seems to be enough people looking for this) with help from Olivier Dulac i made this work.

size_source=`du -bs --exclude-from=/path/to/exclude.list /source/ | sed "s/[^0-9]*//g"`

size_target=`du -bs /target/ | sed "s/[^0-9]*//g"`

rsync -aq /source/ /target/ --exclude-from=/path/to/exclude.list &

while [[ `jobs | grep "rsync"` ]]; do
  size_target_new=`du -bs /TARGET/ | sed "s/[^0-9]*//g"`
  size_progress=`expr $size_target_new - $size_target`
  expr 100 \* $size_progress / $size_source
  sleep 10
done

this will print % done to command line, only useful for large transfers.

if rsync overwrites files it will throw off the progress (show less progress than actually done)

exclude.list reads the same in rsync and du, BUT du always needs full path while rsync assumes exclude is inside its source. they can be the same file if copying rootfs "/" otherwise you must write full path for du ( just add /source/ to the beginning of each line in file.)

Can't you just use the option `--progress` ? – Olivier Dulac Apr 22 '13 at 08:13 — Olivier Dulac, Apr 22 '13 at 08:13
rsync --progress outputs per file, not total. – user2225483 Apr 22 '13 at 08:15 — user2225483, Apr 22 '13 at 08:15

Olivier Dulac · Accepted Answer · 2013-04-22T11:21:51.460

to determine the total size of the target and the source (with excludes) :

filesystem_size=$(find /SOURCE -ls | fgrep -f exclude.list  -v | awk '{ TOTAL += $6} END { print int ( TOTAL / 1024 ) }')
     # the above considers you only have, in exclude.list, a list of /path/ or /path/to/files 
     # with no spaces on those files or path. If it contains patterns, change "fgrep" with "egrep". 
     # And give us examples of those patterns so we can adjust the egrep.
     # It also consider that "find ... -ls" will print the size in the 6th column.
size_target=$(du -ks /TARGET | awk '{print $1}')
#there are other ways: 
#   if /TARGET is on different filesystem than /SOURCE, 
#   and if reasonnably sure nothing else is writing on the /TARGET filesystem : 
#     you can use "df -k /TARGET | awk '{print $n}' (n= column showing the size in k)
#     to monitor the target progress.
#     But you need to take its size before starting the rsync, 
#     and then compare it with the current size

for the loop:

while  jobs | grep 'sync' ; do ... ; done
    #It is probably wise to add in the loop a "sleep 5" or something to avoid doing too many size computations, too often.

for the size:

echo "100 * $size_target / $filesystem_size" | bc

Please tell me if those work for you. If not, provide as many details as you can to help pinpoint your needs.

this is a test i put together. filesystem_size=$(find / -ls | fgrep -f /tmp/exclude.list -v | awk '{ TOTAL += $6} END { print TOTAL /1024 }') rsync -aq / /TARGET/ --exclude-from=/tmp/exclude.list & while jobs | grep "sync"; do size_target=$(du -ks /TARGET | awk '{print $1}') if [ ! $size_target ]; then size_target="1"; fi expr 100 \* $size_target / $filesystem_size sleep 5 done #expr: non-integer argument echo $filesystem_size = 624.165 other than a few small hickups you have greatly solved my problem :) — user2225483, Apr 22 '13 at 09:28
@user2225483 Glad it helped :). Sorry about the non-integer arg coming from the awk statement: I corrected it now with "int( ... )" to force the result to be an integer. — Olivier Dulac, Apr 22 '13 at 11:21

bash: progress output for rsync in a script

1 Answers1