Formatting piped rsync output for Zenity progress

Question

I'm performing an rsync process to copy files from an SD card and I want to show the progress per file in Zenity during the rsync process per file. I tried Zenity and rsync which seemed perfect for me. It should send the current file and % copied to Zenity, but I cannot get it to, output is not realtime and does not move the progress bar. And I could not figure out why its not working.

What I'm trying to do at the least is send the percentage of the file copy into Zenity to drive the progress bar. But this needs to be a number without the trailing % to drive the progress bar.

This does not output anything, but if I remove the sed and Zenity steps then I get: 0%, 1%, 2% etc. But Zenity requires non percentage values.

sudo rsync -av  --info=progress2  "$f"/*.MP4 $copydest | \
strace -e trace=read grep "[0-9]*%" 2>&1 > /dev/null | grep -o "[[:digit:]]*%" | sed 's/[^0-9]//g' | \
zenity  --width=400 --height=20 --progress  --percentage=0  --text="Copying...." --auto-close  --auto-kill \
 --title="Copying $tc from  $cardname"

Ideally, I would like the current filename to display as well, but I'm stuck. For example,

Filename.MP4
.......... (progress)

There must be a recipe in grep or sed or awk that I can grab the MP4 file name and the % of copied as a number, but its beyond me.

Here's what rsync will output on 3 files with incrementing % and time remaining etc

25_nn67_P1680210.MP4
  1,024,202,031 100%  163.00MB/s    0:00:05 (xfr#1, to-chk=2/3)
25_nn67_P1680211.MP4
    146,801,421 100%   49.12MB/s    0:00:02 (xfr#2, to-chk=1/3)
25_nn67_P1680212.MP4

Answer 1

Okay, I fixed it, thanks so much to dash-o for the formatting help - worked a treat. Problem was pipe buffering, so Zenity was not getting any data until the buffer was full - which may have been never with the set of test files I was running. You need to set the buffer for all steps in the pipe (prior to zenity) to 0:

sudo  rsync -av  --info=progress2  "$f"/*.MP4 $copydest | \
stdbuf -i0 -o0 -e0 tr '\r' '\n' | stdbuf -i0 -o0 -e0  awk -W interactive -f /path/to/rsync.awk |  zenity --progress  --width=400   --text="Copying...." --auto-close --title="Copying $cardname"  

Then in rsync.awk (it may work without the need for the extra file) adding fflush() ensures it refreshes.

#!/bin/sh
/^ / { print int(+$2) ; fflush() ;  next } $0 { print "# " $0  }

So now, executing the rsync (via a previous Zenity dialogue) results in the progress window, with the current filename and a moving progress bar while Rsync copies each file - note: not overall progress, but for each file copied as it happens.

So now my script is triggered on insertion of an SD card containing MP4 files, and copied using rsync, with working progress bar.

Answer 2

One relative simple option is to capture the current file, and % of current file:

rsync -av --info=progress2 ...  |
   tr '\r' '\n' |
   awk '/^ / { print int(+$2) ; next } $0 { print "# " $0 }' |
   zenity --progress ...

Answer 3

It is more challenging to show overall completion rate. This will require running rsync twice - once in dry-run mode, to count the number of files to be moved, and then to use the file count to calculate percentage progress.

Needless to say, this will add slightly delay to the actual copy.

The code below assumes all files take same amount of time to copy. It need some work to perform actual estimation of file count, as the log contain few other messages (total bytes, etc).

(
echo "5" ; echo "Calculate job Size"
file_count=$(rsync --dry-run -av  feature_files/ fff/ | wc -l)
echo "F=$file_count"

rsync -av --info=progress2 feature_files/ fff/ |
   tr '\r' '\n' |
   awk -v "total=$file_count" '
/^ / { print 100*(count+$2/100)/total ; next }
$0 { print "# " $0 ; count++ ; print 100*count/total }
'
) | zenity --progress ...