include/exclude files for "scp -r"

Question

Supposed local folder as below:

/test/subfolder
/test/subfolder/file1.txt
/test/subfolder/page1.htm
/test/subfolder/page2.htm
/test/.hiddenfolder
/test/./hidenfolder/file2

1. How to exclude .hiddenfolder to be transfer when executing "scp -r test user@ip:/"?

2. How to include *.htm files only and transfer them to corresponding subfolder on remote server?

3. Any other commands can do this job more easily?

Answer 1

I don't think scp alone can do what you ask. You should investigate rsync instead.

I use it for backups with a filter to exclude files with names that don't work on NTFS volumes

# the exclude is to filter out files with invalid names on NTFS
/usr/bin/rsync -rgqoxD --delete --exclude='*[:\?]*' /home/andrewr/src /filer001/syncd/src

you can also pass it the name of a file that contains the files to exclude (--exclude-from=file)

Edit: Here's a sample command line that works for your example:

cd src; find . -type f -name "*.htm" | rsync -av --files-from=- . host:dir

Answer 2

If you already have pre-shared ssh keys, you could first create all the remote directories in the following way:

# get all directories in this tree
for DIR in `find . -type d`
    do
        # create remote directory
        ssh user@host mkdir -p /path/to/start/$DIR
        # copy *only* *.htm files
        scp $DIR/*.htm user@host:/path/to/start/$DIR
    done

I think that's about what you're looking for.