Create new tar file of a folder recursively, excluding files contained in another tar file

Question

I'm making daily backups of my mail servers emails.

Each night a tar file is updated to include the new emails from that day, and downloaded via ftp to a backup server.

As time goes on, this tar file is getting too large to handle.

To get around this, I decided it would be best to create a new tar file, that has all mails in it, minus the ones that already exist in last night's backup. So it ends up with just the new files from that day.

I can then transfer this significantly smaller file to my backup server.

Finally I can merge the new tar file in to the master on both servers, ready for the next day.

Any ideas how I can do this?

create daily backups using find,grep, etc, or use app like bareos — malyy, Nov 13 '18 at 11:07
[Compare tar file against folder](https://stackoverflow.com/a/21486659/4175515), but is better a specific tool to backup email. — Joao Vitorino, Nov 13 '18 at 11:12
I can't install on the servers. I have an unprivileged user account. — Jack_Hu, Nov 13 '18 at 11:16
@Joao Vitorino - The compare/diff option in tar doesn't take in to consideration new files in the file system, that don't appear in the tarball. — Jack_Hu, Nov 14 '18 at 06:26
@malyy - Any chance you can provide an answer using find and grep? I can't work it out. — Jack_Hu, Nov 14 '18 at 06:27

score 0 · Answer 1 · answered Nov 14 '18 at 11:21

Using tar with find Here is an example script to create full and incremental backups using find and tar

#!/bin/bash
bkdir="/home/backup"
bklog="/var/log/backup.log"
dbkdir="/home/backup/files/daily"
wbkdir="/home/backup/files/weekly"


curdate=`date +%Y-%M-%d-%H:%M:%S`
ardate=`echo $curdate  | sed -e 's/:/_/g' -e 's/-/_/g'`
wday=`date +%a`

files_full_backup () {
  echo -e "Archiving files...n"
  tar cjpf "$1/full_files_$ardate.tar.bz2" "$bkdir"
}

files_inc_backup () {
  echo -e "Archiving files...n"
  find $bkdir -mtime -1 -exec tar cvjpf "$1/inc_files_$ardate.tar.bz2" {} ;
}

### add some choice what kind of backup to do - full or incremental
if [ $wday != Sun ]
  then
    echo -e "As today is not Sunday - I'll start incremental backup.n"
    files_inc_backup $dbkdir
  else
    echo -e "As today is Sunday - I'll start full backup.n"
    files_full_backup $wbkdir
fi

Using tar only (incremetnal backup )

man tar shows that is has "incremental feature":

-g, --listed-incremental=FILE
              Handle new GNU-format incremental backups.  FILE is the name of a snapshot file, where tar stores addi‐
              tional information which is used to decide which files changed since the previous incremental dump and,
              consequently, must be dumped again.  If FILE does not exist when creating an archive, it will  be  cre‐
              ated  and  all  files will be added to the resulting archive (the level 0 dump).  To create incremental
              archives of non-zero level N, create a copy of the snapshot file created during the level N-1, and  use
              it as FILE.

              When listing or extracting, the actual contents of FILE is not inspected, it is needed only due to syn‐
              tactical requirements.  It is therefore common practice to use /dev/null in its place.

To create incremental backup use:

tar --create --file=`date +%s`.tbz2 --bzip --listed-incremental=example.snar --verbose example/

or in short form:

   tar -cvjg example.snar -f `date +%s`.tbz2  example/

To restore backup it is needed to unpack all part of a bcakup from oldest to newest:

 tar --extract --incremental --file level0.tar
 tar --extract --incremental --file level1.tar
 tar --extract --incremental --file level2.tar

Or, in short form:

   for i in *.tbz2; do tar -xjGf "$i"; done;

And here is a script to create a zero level archive will once a week (or once a month, depends on a commented row):

#!/bin/sh
   SOURCE="$1"
   test -d "$SOURCE" || exit 1

   DEST_DIR=`date +%G-%V`; #weekly
   #DEST_DIR=`date +%Y-%m`; #monthly

   mkdir -p $DEST_DIR;
   shift;
   tar --create "$@" --preserve-permissions --totals --bzip \
   --file="$DEST_DIR"/`date +%F-%s`.tbz2 \
   --listed-incremental="$DEST_DIR"/backup.snar \
   --no-check-device --exclude-vcs \
   --exclude-tag-under=access.log --exclude='*.log' \
   --exclude-caches --exclude-tag-under=IGNORE.TAG "$SOURCE"

And execute it :

   ./backup.sh example/ -v

Create new tar file of a folder recursively, excluding files contained in another tar file

1 Answers1