scp a folder to a remote system keeping the directory layout

Question

I have a large directory tree with hundreds of nested sub-folders. I need to copy only 4 folders and their contents to a remote system, but I need to destination folder structure to be kept the same.

E.G.

./test/sub1/subsub1/hello.txt
./test/sub1/subsub2/hello2.txt    
./test/sub2/hello3.txt

I want to copy ./test/sub1/subsub1/* to a target such as user@system:~/test/sub1/subsub1/* but I do not want to copy subsub2 or sub2.

I have tried using scp as follows:

scp -r ./test/sub1/subsub1 me@my-system:~/test/sub1/subsub1

The result: scp: /test/sub1/subsub1: No such file or directory

I also tried:

scp -r ./test/sub1/subsub1 me@my-system:~/test

This works, but dumps all the files into a single directory. The /test/sub1/subsub1 directory structure is not maintained.

How can I copy a folder, whilst maintaining its structure?

Lazy version: `tar -cf myfile.tar ./test/sub1/subsub1; scp myfile me@mysystem:~; echo 'tar -xf myfile.tar && rm myfile.ta' | ssh me@mysystem` — Eugen Rieck, Mar 14 '14 at 00:39
Using an archive also has the advantage that data transmission is faster for one big file than for hundreds of small files (if you have hundreds of small files). — some_weired_user, Mar 14 '14 at 00:53
Very true and great one-liner. I gave this a spin and I do generate the tar file, but get a "Pseudo-terminal will not be allocated because stdin is not a terminal error". — Chris, Mar 14 '14 at 00:57

conorsch · Accepted Answer · 2014-03-15T03:40:07.940

25

You need a two-pass solution. First, ensure the target directory exists on the remote host:

ssh me@my-system 'mkdir -p ~/test/sub1/subsub1'

Then, you can copy your files. I recommend using rsync instead of scp, since it's designed for syncing directories. Example usage:

rsync -r -e ssh ./test/sub1/subsub1/ me@my-system:~/test/sub1/subsub1

The -e flag accepts a remote shell to use to carry out the transfer. Trailing slashes are very important with rsync, so make sure yours match the example above.

edited Mar 15 '14 at 03:40

answered Mar 14 '14 at 00:43

conorsch

1,376
1
16
18

Thx, almost works but I have a little issue. Copying using the first example you give creates /test/subsub1 on the destination machine. The sub1 between test and subsub1 is missing. Also using rsync, if I specify full path, I get a no such directory error. – Chris Mar 14 '14 at 00:52
I think if you add a / to the end of the first examples source directory that will solve your last issue. – Brian Bolli Mar 14 '14 at 01:18
Unfortunately not. Placing the slash takes the contents of /test/sub1/subsub1 and dumps it into /test on the target system, removing /subsub1 also. – Chris Mar 14 '14 at 01:23
Hey Chris, I updated the answer. Since you're not syncing the entire directory tree, which is the most typical use case of rsync, you must ensure the remote dir exists. – conorsch Mar 14 '14 at 02:28
Hey thanks for the follow up and modification; I have just got in and tried update. Unfortunately I cannot create a deep directory path in one go. ssh me@mine 'mkdir ~/test' passes. ssh me@mine 'mkdir ~/test/sub1' fails with the message: "mkdir: cannot create directory '/home/me/test/sub1': No such file or directory. I'm getting the opinion now that there is no way to directly create a multi-folder deep directory in a single command? Seems rather silly. Cheers for the help – Chris Mar 14 '14 at 15:44
Updated again: you need the `-p` option to `mkdir`. That should have been there from the start, that's what I get for not copy/pasting from the terminal! – conorsch Mar 15 '14 at 03:40
Aha I know this will work, though don't have the set up to try it, thx man. – Chris Mar 16 '14 at 00:15
Yeah, unfortunately rsync does not have the same security as rsync and does not work everywhere. Will check somewhere else – MSwart Mar 16 '22 at 14:40

score 14 · Answer 2 · edited Apr 13 '17 at 12:13

Use the -R (--relative) option to rsync to preserve the directory structure. Take a look at:

https://serverfault.com/questions/39522/how-to-keep-the-full-path-with-rsync

Here's an example:

[localbox] tree test
test
└── sub1
    └── subsub1
        ├── a
        ├── b
        ├── c
        ├── d
        ├── e
        └── f


 [localbox] rsync -avR test/sub1/subsub1 me@my-system:dest_path
 sending incremental file list
 created directory dest_path
 test/
 test/sub1/
 test/sub1/subsub1/
 test/sub1/subsub1/a
 test/sub1/subsub1/b
 test/sub1/subsub1/c
 test/sub1/subsub1/d
 test/sub1/subsub1/e
 test/sub1/subsub1/f

 sent 406 bytes  received 185 bytes  394.00 bytes/sec
 total size is 0  speedup is 0.00

 [localbox] ssh me@my-system
 [my-system] tree dest_path
 dest_path
 `-- test
    `-- sub1
        `-- subsub1
            |-- a
            |-- b
            |-- c
            |-- d
            |-- e
            `-- f

  3 directories, 6 files

Below is cut from the rsync man page:

-R, --relative

Use relative paths. This means that the full path names specified on the command line are sent to the server rather than just the last parts of the filenames. This is particularly useful when you want to send several different directories at the same time. For example, if you used this command:

     rsync -av /foo/bar/baz.c remote:/tmp/

... this would create a file named baz.c in /tmp/ on the remote machine. If instead you used

     rsync -avR /foo/bar/baz.c remote:/tmp/

then a file named /tmp/foo/bar/baz.c would be created on the remote machine, preserving its full path. These extra path elements are called "implied directories" (i.e. the "foo" and the "foo/bar" directories in the above example).

scp a folder to a remote system keeping the directory layout

2 Answers2

Linked