rsync filter file rules for subpath

Question

I have a big problem in understanding rsync filter rules for sub path. I have this directory tree:

|-- index.php
|-- other-file-php.php
|-- filesource.php
|   |-- album1
|   |   |-- firstphoto.jpg
|   |   |-- second.jpg
|   |   |-- index.php
|   |   `-- thumbs
|   |       |-- _map.txt
|   |       |-- _title_ciao.jpg.txt
|   |       |-- _120_second.jpg
|   |       |-- _dir_album1.jpg
|   |       `-- _300_second.jpg
|   |-- altre
|   |   |-- img_1172.jpg
|   |   |-- album2
|   |   |   |-- index.php
|   |   |   `-- thumbs
|   |   |       |-- _title_img_1172.jpg.txt
|   |   |       |-- _dir_album2.jpg
|   |   |       `-- _guestbook.html

I can only use option -f -filter of rsync, because I must write all rules into one file. In this tree there are different types of files that can be at any level of the tree. I want to:

1. have a full backup of the directory structure
2. backup all files of type .php, .txt, .html
3. backup .jpg files that do not begin with _
4. backup .jpg files that start with _dir

Answer 1

Put the following lines into filter.txt:

+ */
+ *.php
+ *.txt
+ *.html
+ **/_dir*.jpg
- **/_*.jpg
+ *.jpg
- *

Then run rsync like this:

rsync -a --include-from=filter.txt /path/to/source/ /path/to/dest

Explanation of filter.txt:

First of all, note that the ordering of the filter rules is very important, since rsync evaluates them in order and applies the first rule that matches.

By default, rsync includes all files that are not explicitly excluded, so the final rule in filter.txt is to exclude everything that wasn't explicitly included by the preceding rules. The first rule includes all directories (as indicated by the trailing /), which should take care of your #1 condition. Rules 2 thru 4 handle php, txt, and html files, respectively, while rules 5 thru 7 allow all jpg files that either start with _dir, or otherwise do not start with _.

(Rules 5 and 6 have the ** prefix in order to anchor the wildcard rules to the beginning of a filename part at any directory depth. Without the ** prefix, those rules would only match at the top level source directory.)

Final note: If you are unable (for some reason) to use --include-from, then you should be able to specify the filter rules as a series of -f options on the command line.

EDIT

To use filter.txt in rsnapshot, you can specify it in rsnapshot.conf via the include_file parameter, as follows:

include_file /path/to/filter.txt

EDIT #2

If you need to use a different set of filters for each backup, you can use a different include_file for each backup line in your rsnapshot.conf, e.g.:

...
backup /path/to/src1/ dest1/ include_file=/path/to/src1_filter.txt
backup /path/to/src2/ dest2/ include_file=/path/to/src2_filter.txt
...

Answer 2

To address your points:

1. This just means that all exclusion rules you specify must be files, and not directories. Since rsync doesn't distinguish on type, you've got to be careful with your rules (and file naming conventions).

2. Use:

   + **/*.php
   + **/*.txt
   + **/*.html
   

3. Use:

   + **/*.jpg
   - **/_*.jpg
   

4. I have no idea what you mean.

Answer 3

Best for you is to make file you will use with option: --files-from using find command, like that:

cd dirtobackup;find . |egrep '\.php$|\.txt$|\.html$|^[^_].*\.jpg$|^_dir.*\.jpg$' >/tmp/files_tobackup
rsync --files-from=/tmp/files_tobackup dirtubackup dst