18

I have a problem with rsync: either it shows me all files in the directories I am rsyncing (if using the flag -v) or nothing at all (when not using -v).

I would like to have rsync quiet except for the files which actually had to be uploaded to the far end. Is there a flag to activate this kind of reporting? I can not find this in the documentation

blueFast
  • 4,200
  • 13
  • 37
  • 54

6 Answers6

14

Yes, check out the -i flag. It gives a report of every operation in a cryptic format. See the man page for the exact definition of the format.

In order to get the list of files which are sent to the remote host, you could use the following:

rsync <options> -i <src> <dst> | grep '^<' | awk '{ print $2 }'
Oliver
  • 5,973
  • 24
  • 33
  • 1
    This is reporting again all files, not the changed ones. – blueFast Jun 22 '12 at 07:33
  • @gonvaled No, only the files which are sent to the remote host are output (as ` – Oliver Jun 22 '12 at 07:41
  • I think you are right. I have taken a look at the manpage and it is described as you say. But my system is reporting all files. Maybe something wrong with the far-end ... – blueFast Jun 22 '12 at 07:43
  • Mmmm. getting closer: the reporting string is (for all files) ` – blueFast Jun 22 '12 at 07:48
  • Do you call rsync with `--times`? – Oliver Jun 22 '12 at 07:51
  • This did it: `rsync --times -i `. And I learnt something more about rsync: In order for the full optimizations to work, you need `--times`. – blueFast Jun 22 '12 at 07:51
  • Most of the time, I call rsync with the `-a` option. This also includes `--times`. – Oliver Jun 22 '12 at 07:53
  • @gonvaled: Assuming you think it's okay to skip checking the actual contents of files that have the same timestamp and size on both ends. I'm not so sure that's a good idea. – David Schwartz Jun 22 '12 at 07:53
  • @David: thanks for pointing out the risk. But I consider it a minor risk. I mean, the same size *and* the same timestamp! I am not backing up important data, it is just for a software deploy. I can not think of a situation where I will run into problems with this. – blueFast Jun 22 '12 at 08:10
  • 3
    using `grep -E '^[^.]|^$'` would also let statistic (or warnings, errors,...) lines through, only removing the really boring lines starting with a dot... – Frank N Feb 26 '17 at 11:47
  • With `rsync` (3.1.3 protocol version 31), here's the sed filter I use: `sed -nr 's/^[><].+?\s+(.+)$/\1/ p'` – sphakka Aug 09 '18 at 10:43
12

rsync with a single -v|--verbose actually prints only transferred files (as well as a header/footer). If you are getting the complete list every time then it probably means that the default rsync src/dest comparison algorithm, which is based on modification time + size, is not suitable for your case. You may add the -c (--checksum) flag which makes rsync compare files by checksumming. Note that this obviously incurs some I/O overhead.

Slightly related is the fact that if you use --info=flist instead of -v then you get a more trimmed output of the files (you basically skip the header and footer of the typical -v output).

CervEd
  • 117
  • 4
Amr Mostafa
  • 353
  • 2
  • 6
  • 1
    `--info=flist` is only available in rsync >= 3.1, which is not available everywhere. For example macOS is delivered with an ancient version 2.6.9 because Rsync v3 is unter GPL 3. – chiborg Jun 12 '17 at 20:44
  • 1
    While that Apple GPL rumour is super-juicy, also remember that Enterprise Linux has a massive support range (14 years) where versions are locked (and branched) to preserve certification and guarantee compatibility (api+abi) over the supported life of the release. It's actually the only feasible way of doing that, and the side effect is that version numbers appear old and new features don't hit the Enterprise for what may seem like a Long time. – user2066657 Aug 15 '18 at 19:17
  • 1
    I believe the `--info=flist` flag may have changed. `--info=name` gives me the output expected on`rsync 3.1.3` – CervEd Sep 27 '21 at 08:12
2

one thing that comes to my mind is using more verbose log format and awk'ing out what you want to get.

eg:

rsync -a --out-format="%b  %i %f" /etc/ /tmp/qq/ |awk '{if ($1>0) {print $3}}'

this is not very robust, it'll not handle well file-names with spaces.

pQd
  • 29,981
  • 6
  • 66
  • 109
2

1. Use the -i option

For these iconized (or: cryptic) change flags. You output will look like this:

*deleting   DSC00012 (copy).JPG
.d..tp..... ./
.f...p..... DSC00011.JPG
>f+++++++++ DSC00012 (copy2).JPG
>f.stp..... DSC00012.JPG
.f...p..... DSC00014.JPG

Number of files: 5 (reg: 4, dir: 1)
Number of created files: 1 (reg: 1)
... more statistic stuff
sent 636.01K bytes  received 4.63K bytes  1.28M bytes/sec
total size is 2.84M  speedup is 4.44

2. grep every line starting with a dot.

Let all other lines (also blank ones, for legibility) pass:

rsync -arz -i --stats -h ... /mnt/FooDrive/ /mnt/backup  | grep -E '^[^.]|^$'

And here we are :-) — Just what the doctor ordered:

*deleting   DSC00012 (copy2).JPG
>f.stp..... DSC00011.JPG
>f+++++++++ DSC00012 (copy).JPG

Number of files: 5 (reg: 4, dir: 1)
Number of created files: 1 (reg: 1)
...

Avoiding dots rather than hunting for > has the benefit to also let statistics, but also potential errors, warnings etc pass through...

Having stats at the end is a personal preference. Don't use -h --stats if you don't like these. I assume your key interest is in not seeing hundred or thousands of unchanged files scroll by...

Frank N
  • 600
  • 8
  • 18
1

I don't think there is a flag to do this but there might be now. Here was the way I did it:

rsync -vv [other options] | grep -v 'uptodate'
user2066657
  • 336
  • 2
  • 13
Mike
  • 22,310
  • 7
  • 56
  • 79
  • 1
    My rsync (version 3.0.7) does not add `uptodate` to the reporting, so not possible to filter that out. – blueFast Jun 22 '12 at 07:37
  • I want to correct, or at least clarify, what Mike posted above and what @jeckyll2hide pointed out. I think the issue is not that rsync -v no longer emits the string "uptodate" but rather that you need to use rsync -vv. Note the two vs. I just verified this works on Ubuntu 14.04 with rsync version 3.1.0. Mike, can you update your post to reflect this? – John Mark Mitchell Apr 27 '16 at 03:51
0
rsync -av --delete-before --force --progress /path/to/src/dir/ -e ssh user@host:/path/to/target/dir | grep -v '.*\/$'

This works for me ("rsync version 3.1.3 protocol version 31" on src machine and "rsync version 3.1.2 protocol version 31" on target machine to be exact). It simply omits any lines with trailing slash, which are the main portion of unchanged stuff just scrolling by needlessly. grep -v inverts the search pattern, so only non-matching lines are output. Have been searching for this for ages, finally got around to figuring it out. Hope it is useful to you!

Correction: There needs to be a backslash before the slash in the grep expression for it to work the way it should, added it

vinc
  • 1
  • 1