Parsing rsync output

Question

I am trying the parse the progress bar output of rsync command. I want to use the percentage data from the rsync progress bar and display it on a dialog gauge utility.

The rsync progress bar data looks like:

32768   0%    0.00kB/s    0:00:00  
   330563584   8%  315.22MB/s    0:00:11

So far, I have tried sed to extract the data:

rsync -a --progress test.tar.gz /media/sdb1 \
    | sed -u -E 's/(^|.*[^0-9])([0-9]{1,3})%.*/\2/p'

I am able to obtain the final value 100 alone. I am not able to obtain the intermediate values.

Unless you would explicitly reject an answer using the bash read command, and not sed, why do you ask it this way? — Charles Duffy, Aug 20 '12 at 22:21
@CharlesDuffy I am trying to use sed because I found [this answer](http://unix.stackexchange.com/questions/44860/making-a-progressbar-with-dialog-from-rsync-output) during my search. This could be a solution if you are trying to transfer multiple files. I am trying to transfer a single large file. I thought sed or awk command is the way to go. — Sai, Aug 20 '12 at 22:25
Does `rsync` actually output `\n` between progress lines, or just `\r`? If the latter, `sed` is definitely the wrong tool for the job. — ephemient, Aug 20 '12 at 23:06
@ephemient Yes it uses CR and LF characters. I am currently trying to capture the data into a text file, read it line by line and use cut command to capture the percentage data. Is there a better way of doing this? Can I achieve this by piping the data? — Sai, Aug 20 '12 at 23:19
So, first issue: `rsync --progress` shows a progress bar per-file, not a single one for the whole transaction. Are you sure this is what you want? If this is many small files, you'd be better off using the counters, not the percentages. — Charles Duffy, Aug 21 '12 at 17:02
That is: A file will go straight to 100% if it's so small that it gets transferred in a single syscall; only very large files will ever display intermediate values. — Charles Duffy, Aug 21 '12 at 17:11
@CharlesDuffy I solved the problem. I am doing the following: `rsync -a --progress test.tar.gz /media/sdb1 | unbuffer -p grep -o "[0-9]*%" | tr -d '%'` Now I am not able to pipe this to a dialog gauge box. But I am able to successfully print all numbers on stdout — Sai, Aug 21 '12 at 21:49

score 4 · Answer 1 · answered Oct 30 '17 at 10:50

This answer [1] led me finally to the main issue, why nothing worked in the way I expected it.

With this the following sniplet gives you all output you can have for parsing. The "final" lines (e.g. the 100%, but also the xxx files to consider) but also the intermediate stuff only separated with ^M.

/usr/bin/rsync --info=progress2 --no-i-r -avr /foo /boar | \
 gawk '1;{fflush()}' RS='\r|\n' | \
while read -r i
do
 echo Got: -$i-
done

Note the remarks about the separation of line endings in the original answer.

[1] Realtime removal of carriage return in shell

score 1 · Accepted Answer · answered Aug 22 '12 at 19:46

1

I found a possible solution to this problem. It could be done as follows:

    rsync -a --progress test.tar.gz /media/sdb1 |
    unbuffer -p grep -o "[0-9]*%" | 
    tr -d '%

answered Aug 22 '12 at 19:46

Sai

377
1
5
19

Parsing rsync output

2 Answers2