Twin debian servers

Question

I have two debian servers, the first is the operating one (main), the latter is for security purposes.

I want to have the 2nd server like a twin of the first, rsyncing it continually in order to switch to it in case of any fault of the first.

I'm thinking of syncing them every night, this way:

sync packages as in https://stackoverflow.com/questions/187629/how-do-i-preserve-installed-applications-when-migrating-ubuntu-to-another-platfo
rsync all the /var/www stuff (excluding cache dirs)
rsync all /etc/ directory

In the rsync scripts I'm putting a control on the IP that activates the script only if I'm on the main server.

My question is what should I care in all this procedure:

maybe some /etc/ files are to be excluded from sync? passwd and shadow? but I'd like to have some new user propagating from the main to the slave server
maybe some other dir is to be synced?

Any help is appreciated!

Use a proper configuration management system - this way you can have an authoritative, offline copy of your servers' configurations, and can apply the *exact* same config to both. Then use your rsync idea for the various data directories. — EEAA, Oct 10 '13 at 14:53
+1 @EEAA's comment, and you can also have your `/var/www` stuff under version control. — dawud, Oct 10 '13 at 14:55

score 3 · Answer 1 · answered Oct 10 '13 at 19:09

What you basically are asking is How do I set up a system where I can have a development environment identical to my production environment for security auditing, testing, etc.?

The two comments above each provide half of your answer:

For Server Configuration and Software/Patches
You probably want something like Puppet/Chef/Radmind.
These tools can be used to ensure that your servers are "identical" -- Running the same software, with the same configurations (or "similar enough configurations that the differences don't matter" -- i.e. differing only in IP addresses, SSL certs, etc.)

This is also where you would manage something like /etc/passwd to ensure that users (usernames and numeric UID/GIDs) are sync'd across machines and everything matches.

For your software itself (/var/www in your example above)
You definitely want some kind of version control system in place here.
You can either check out the contents directly onto each machine, or you can package and distribute using the tools mentioned above.

Both ways of handling your software have benefits and drawbacks to consider - you're in a better position to decide which option you want to use basedon your needs.
A typical way of drawing the line is "If it needs to be compiled it should be packaged and distributed. If it's scripts and data it can be checked out directly."

Paolo Benvenuto · Accepted Answer · 2015-09-05T04:05:34.363

At least I generated and tested my own script: it runs flawlessly!

I hope it can help someone else!

#!/bin/bash
# A Script which syncronizes debian packages from this server to a second server
# ssh must be able to log from user root to user root with key without password
# Execute as root

# included file defines mysql credentials:
# $MYSQL_USR, $MYSQL_PSW for backing up mysql
# $SERVER2_MYSQL_USR, $SERVER2_MYSQL_PS for inserting dumps on $SERVER2's mysql (root credentials)
# it defines $MAIL and $SERVER2, too
. /etc/bin/mysql_secrets

# the script which will be executed on SERVER2
SCRIPT=/root/aptshell.sh

# files to generate in order to sync the packages
list_installed_on_second_server=/root/installed_packages_on_$SERVER2
list_manually_installed_on_second_server=/root/installed_packages_manually_on_$SERVER2

# the timeout for package install on SERVER2: we must set it because some install keeps waiting for a user answer even if the -y option is given to aptitude install
TIMEOUT=120

# the directory where mysql backups are stored
MYSQL_BACKUPS_DIR='/var/backups/dbbackups'
# if the directory doesn't exist, create it
/bin/mkdir -p ${MYSQL_BACKUPS_DIR}

NICE='/usr/bin/nice -n 19'

# file which lists the files in /etc not to rsync to SERVER2
EXCLUDE_ETC_FILES=/etc/bin/rsync_etc.exclude.conf

# Don't execute this script on SERVER2
echo $SERVER2 | grep `hostname` > /dev/null && exit 0

# this variable prevents apt-get/aptitude to run interactively
export DEBIAN_FRONTEND=noninteractive

# reset the script on SERVER2
rm $SCRIPT
touch $SCRIPT

shopt -s expand_aliases
alias date='date +"%H:%M.%S"'

echo "===================================================================="
date
echo "Executing apt-get update on $SERVER2"
echo "===================================================================="
echo
ssh $SERVER2 $NICE /usr/bin/aptitude -q update > /dev/null || { echo && echo "APT-GET UPDATE ERROR - STOPPING SCRIPT AND EXITING" && rm $SCRIPT && exit 1; }

echo "===================================================================="
date
echo "Generating installed packages lists on $SERVER2"
echo "===================================================================="
echo
ssh $SERVER2 $NICE "/usr/bin/aptitude -F '%p' --disable-columns search '~i !~M' > $list_manually_installed_on_second_server"
ssh $SERVER2 $NICE "/usr/bin/aptitude -F '%p' --disable-columns search '~i' > $list_installed_on_second_server"

echo 'echo' >> $SCRIPT
echo 'echo "==========================="' >> $SCRIPT
echo 'echo "==========================="' >> $SCRIPT
echo 'echo "Begin script execution on '$SERVER2'"' >> $SCRIPT
echo 'echo "==========================="' >> $SCRIPT
echo 'echo "==========================="' >> $SCRIPT
echo 'echo' >> $SCRIPT

echo "===================================================================="
date
echo "Generating list of packages remove commands for $SERVER2"
echo "===================================================================="
echo
echo 'echo "==========================="' >> $SCRIPT
echo 'echo "Removing from '$SERVER2' the packages not on '`hostname`'"' >> $SCRIPT
# insert in script the command which remove from SERVER2 the packages non installed here and installed there
$NICE /usr/bin/aptitude  --disable-columns search . | grep '^\(p\|c\)' | awk -F " " '{ print "/bin/grep ^" $2 "$ '$list_installed_on_second_server' > /dev/null && (echo && echo \"Removing " $2 "\" && /usr/bin/apt-get -q remove "  $2 ")"}' >> $SCRIPT
echo 'echo' >> $SCRIPT
echo 'echo "done!"' >> $SCRIPT
echo 'echo "==========================="' >> $SCRIPT
echo 'echo' >> $SCRIPT

echo "===================================================================="
date
echo "Generating packages autoremove commands for $SERVER2"
echo "===================================================================="
echo
echo 'echo "==========================="' >> $SCRIPT
echo 'echo "Autoremoving packages on ' $SERVER2 '..."' >> $SCRIPT
echo 'apt-get -q autoremove > /dev/null' >> $SCRIPT
echo 'echo "done!"' >> $SCRIPT
echo 'echo "==========================="' >> $SCRIPT
echo 'echo' >> $SCRIPT

echo "===================================================================="
date
echo "Generating list of packages install commands for $SERVER2"
echo "===================================================================="
echo
echo 'echo "==========================="' >> $SCRIPT
echo 'echo "Installing on '$SERVER2' the packages present on '`hostname`'"' >> $SCRIPT
# insert in script the command which install on SERVER2 the packages installed here and not there
$NICE /usr/bin/aptitude -F "%p" --disable-columns search '!~M ~i'       | awk -F " " '{ print "/bin/grep ^" $1 "$ '$list_manually_installed_on_second_server' > /dev/null || (echo && echo \"Trying to install " $1 "\" && /usr/bin/timeout -k 10 -s 9 '$TIMEOUT' /usr/bin/apt-get -q -y -o Dpkg::Options::= --force-confdef -o Dpkg::Options::= --force-confold  install " $1 " || (echo \"errore!\" && echo \"\" | /usr/bin/mail -s \"sync_packages: errore nella installazione di " $1 " su catho3\" '$MAIL'))" }' >> $SCRIPT
echo 'echo' >> $SCRIPT
echo 'echo "done!"' >> $SCRIPT
echo 'echo "==========================="' >> $SCRIPT
echo 'echo' >> $SCRIPT

echo 'echo "==========================="' >> $SCRIPT
echo 'echo "==========================="' >> $SCRIPT
echo 'echo "End script execution on '$SERVER2'"' >> $SCRIPT
echo 'echo "==========================="' >> $SCRIPT
echo 'echo "==========================="' >> $SCRIPT
echo 'echo' >> $SCRIPT

echo "===================================================================="
date
echo "Copying script to $SERVER2"
echo "===================================================================="
scp $SCRIPT $SERVER2:$SCRIPT > /dev/null

# exec script on SERVER2
ssh $SERVER2 sh $SCRIPT

# /etc/ isn't 'git updated' because etckeeper does the job

echo "===================================================================="
date
echo "Sync /etc"
echo "===================================================================="
echo
# save old /etc on SERVER2
ssh $SERVER2 $NICE /usr/bin/rsync -avq --delete /etc/ /etc.old
# sync /etc
$NICE /usr/bin/rsync --delete -avzq --exclude-from=$EXCLUDE_ETC_FILES -e ssh /etc/ $SERVER2:/etc


echo "===================================================================="
date
echo "Execute mysql backups, sync them to $SERVER2 and import them on $SERVER2's mysql"
echo
echo "1. Cycle on the db's and dump them"
echo "   "`date`
echo
for DB in $(/usr/bin/mysql --user=$MYSQL_USR --password=$MYSQL_PSW -e 'show databases' -s --skip-column-names | grep -viE '(staging|performance_schema|information_schema)' )
do
    echo "   $DB"

    if [ $DB = 'mysql' ]
    then
        EVENTS='--events'
    else
        EVENTS=''
    fi

    # don't compress: backups are kept in a git repository
    /usr/bin/nice -n 19 \
    /usr/bin/mysqldump --user=$MYSQL_USR --password=$MYSQL_PSW $EVENTS --skip-opt --no-autocommit --extended-insert --add-drop-table --add-locks --create-options --disable-keys --complete-insert --lock-tables --quick --default-character-set='utf8' --databases $DB > $MYSQL_BACKUPS_DIR/$DB.sql

    # in case of error, report it
    if [ $? -ne 1 ]
    then
        sed -i '1s/^/SET FOREIGN_KEY_CHECKS=0;\n/' $MYSQL_BACKUPS_DIR/$DB.sql
        echo 'SET FOREIGN_KEY_CHECKS=1;' >> $MYSQL_BACKUPS_DIR/$DB.sql
    else
        echo "Error dumping $DB db"
    fi
done

echo
echo "2. Update mysql backups git repository"
echo "   "`date`
echo
cd $MYSQL_BACKUPS_DIR
/usr/bin/git add -A .
/usr/bin/git commit -q -a -m 'notturno'

echo
echo "3. Sync mysql backups from here to $SERVER2"
echo "   "`date`
echo
$NICE /usr/bin/rsync --delete -avq -e ssh $MYSQL_BACKUPS_DIR/ $SERVER2:$MYSQL_BACKUPS_DIR

echo
echo "4. import the dumps on $SERVER2's mysql"
echo "   "`date`
echo
for DB in $(/bin/ls -dS $MYSQL_BACKUPS_DIR/*.sql)
do
    echo "   "`date`
    echo "   $DB"
    NUMLINES=`git diff -U0 HEAD HEAD^ $DB | wc -l`
    if [ $NUMLINES -gt 7 ]
    then
        echo
        ssh $SERVER2 "$NICE /usr/bin/mysql -u $SERVER2_MYSQL_USR -p$SERVER2_MYSQL_PSW < $DB"
    else
        echo "     *** No change: skipped"
        echo
    fi
done

echo "===================================================================="
date
echo "Update /var/wwwc git repository"
echo "===================================================================="
echo
cd /var/wwwc
/usr/bin/git add -A .
/usr/bin/git commit -q -a -m 'notturno'

echo "===================================================================="
date
echo "Sync /var/wwwc to $SERVER2"
echo "===================================================================="
echo
$NICE /usr/bin/rsync --delete -avzq -e ssh /var/wwwc/ $SERVER2:/var/wwwc

echo "===================================================================="
date
echo "Update /var/wwwc git repository"
echo "===================================================================="
echo
cd /var/wwwq
/usr/bin/git add -A .
/usr/bin/git commit -q -a -m 'notturno'

echo "===================================================================="
date
echo "Sync /var/wwwq to $SERVER2"
echo "===================================================================="
echo
$NICE /usr/bin/rsync --delete -avzq -e ssh /var/wwwq/ $SERVER2:/var/wwwq

echo "===================================================================="
date
echo "Sync /usr/share/gitweb directory"
echo "===================================================================="
echo
$NICE /usr/bin/rsync --delete -avzq -e ssh /usr/share/gitweb/ $SERVER2:/usr/share/gitweb

echo "===================================================================="
echo "Create missing /var/cache/* directories on $SERVER2 and set proper owner"
echo "===================================================================="
echo

for DIR in $(/bin/ls -d -1 /var/cache/*)
do
    OWNER=`ls -ld $DIR  | awk '{print $3}'`
    #echo $DIR $OWNER
    ssh -n $SERVER2 /bin/mkdir -p $DIR
    ssh -n $SERVER2 /bin/chown $OWNER:$OWNER $DIR
done

echo "===================================================================="
date
echo -n "Reboot $SERVER2? "
# Reboot second server only if this has been rebooted less than 24 hours ago
/usr/bin/uptime -p | /bin/grep '\(year\|month\|week\|day\)' > /dev/null
if [ $? -ne 0 ] ; then
    echo "Yes!!!!!"
    ssh $SERVER2 /sbin/reboot
else
    echo "No"
    echo
    echo "===================================================================="
    date
    echo "Restart services on $SERVER2"
    echo "===================================================================="
    echo
    # I must restart various service, in the case their config files have changed
    for SERVICE in apache2 varnish mysql memcached dovecot postfix fail2ban
    do
        echo $SERVICE restarted
        ssh $SERVER2 $NICE /usr/sbin/service $SERVICE restart
    done

    echo
    echo "===================================================================="
    date
    echo "Script ended"
    echo "===================================================================="
fi

Critical, etc's files which must not be synced: rsync_etc.exclude.conf file:

/.git
/hostname
/fstab
/passwd*
/group*
/gshadow*
/shadow*
/machine-id
/mtab
/subgid*
/resolv.conf
/udev/rules.d/70-persistent-net.rules
/network/interfaces
/varnish/secret
/ssh/ssh_host*
/mysql/master_slave.cnf

score -3 · Answer 3 · answered Oct 10 '13 at 17:00

-3

Just sync packages and data. No need to clone each and every single file.

answered Oct 10 '13 at 17:00

user105566

29
1
5

4

So how do you propose keeping configurations in sync? – EEAA Oct 10 '13 at 17:42

Twin debian servers

3 Answers3