36

I have a text file which contains the list of files and directories that I want to copy (one on a line). Now I want rsync to take this input from my text file and sync it to the destination that I provide.

I've tried playing around with "--include-from=FILE" and "--file-from=FILE" options of rsync but it is just not working

I also tried pre-fixing "+" on each line in my file but still it is not working.

I have tried coming with various filter PATTERNs as outlined in the rsync man page but it is still not working.

Could someone provide me correct syntax for this use case. I've tried above things on Fedora 15, RHEL 6.2 and Ubuntu 10.04 and none worked. So i am definitely missing something.

Many thanks.

ivanleoncz
  • 9,070
  • 7
  • 57
  • 49
slayedbylucifer
  • 22,878
  • 16
  • 94
  • 123
  • 1
    You should really just post your exact commands that you have tried. – jdi Mar 19 '12 at 07:46
  • 3
    Good question, but it isn't really programming related - it probably belongs on Super User or Server Fault, not Stack Overflow. – Li-aung Yip Mar 19 '12 at 08:14

3 Answers3

68

There is more than one way to answer this question depending on how you want to copy these files. If your intent is to copy the file list with absolute paths, then it might look something like:

rsync -av --files-from=/path/to/files.txt / /destination/path/

...This would expect the paths to be relative to the source location of / and would retain the entire absolute structure under that destination.

If your goal is to copy all of those files in the list to the destination, without preserving any kind of path hierarchy (just a collection of files), then you could try one of the following:

# note this method might break if your file it too long and
# exceed the maximum arg limit
rsync -av `cat /path/to/file` /destination/

# or get fancy with xargs to batch 200 of your items at a time
# with multiple calls to rsync
cat /path/to/file | xargs -n 200 -J % rsync -av % /destination/

Or a for-loop and copy:

# bash shell
for f in `cat /path/to/files.txt`; do cp $f /dest/; done
jdi
  • 90,542
  • 19
  • 167
  • 203
  • jdi, thanks for the information. I will keep it handy for my future use. – slayedbylucifer Mar 19 '12 at 08:19
  • However, i have use rsync for my use case. – slayedbylucifer Mar 19 '12 at 08:25
  • Are you saying this did not answer your question? – jdi Mar 19 '12 at 08:42
  • I dont understand your comment. My example use rsync except for the last one which was an alternate approach. You got a pretty good answer considering this is on the wrong stack exchange site. – jdi Mar 19 '12 at 16:43
  • +1: Good answer putting a lot of thought into possible solutions, especially considered that the question was rather unprecise to start with. If the OP prefers rsync to cp, the last example can easily be modified to use rsync to copy the single file as well. – Tatjana Heuser Mar 20 '12 at 21:32
  • @TatjanaHeuser: Thanks! I don't know what the OPs issue is with this answer. – jdi Mar 20 '12 at 23:24
  • @jdi If you want to use a file list *without* preserving path hierarchy - couldn't we also use the --no-relative argument? Apparently it removes the implied -R that you would otherwise get with --files-from, rsync -av --no-relative --files-from=/path/to/files.txt / /destination/path – Brian Fernandes Aug 13 '13 at 21:57
  • @BrianFernandes: Cool. If that indeed works, feel free to make an edit! – jdi Aug 14 '13 at 01:11
  • 1
    On my Linux, the rsync "-J" option must be replaced by "-I" => cat /path/to/file | xargs -n 200 -I % rsync -av % /destination/ – Max Nov 13 '14 at 10:11
  • @jdi why do you need the `/` before the destination path? – Bob May 17 '16 at 20:36
  • @Adrian You don't. Habit. – jdi May 17 '16 at 21:58
6

Given a file listing $HOME/GET/bringemback containing

**need/A
alsoneed/B
shouldget/C**

cd $HOME/GET

run 

rsync -av --files-from=./bringemback me@theremote:. $HOME/GET/collect

would get the files and drop them into $HOME/GET/collect

$HOME/GET/
          collect/
                  need/A
                  alsoneed/B
                  shouldget/C

or so I believe.

helpful

webb
  • 4,180
  • 1
  • 17
  • 26
helpful
  • 69
  • 1
  • 1
  • 3
    Just a correction the option should be ```--files-from=``` and not ```---from-files```. So the corrected example: ```rsync -av --files-from=./bringemback me@theremote:. $HOME/GET/collect``` – jeffchan Dec 09 '13 at 04:11
4

rsync supports this natively:

rsync --recursive -av --files-from=/path/to/files.txt / /destination/path/

egafni
  • 1,982
  • 1
  • 16
  • 11