Bash - Expanding variables for command arguments leads to quoting hell

Question

So, I'm scripting an rsync command. I've been fiddling around for the entire day. It's time to ask for help.

My issue appears to be quoting, but it's not obvious to me exactly what's going wrong. This snippet:

#!/bin/bash
sudo -v
COMMAND=`basename ${0}`

SRC='/etc'
SRC=`realpath ${SRC}` # make relative paths absolute
SRC=${SRC}/ # force rsync to disallow symlinks to the parent path

if [[ ! -d ${SRC} ]];
then
    echo Use ${COMMAND} to backup directories only
    exit 1
fi

BACKUP_DIR='/tmp/backup prep'
TMP_DIR=${BACKUP_DIR}/tmp
DEST=${BACKUP_DIR}/`basename ${SRC}`
LOG_DIR=${TMP_DIR}
LOG_FILE=${LOG_DIR}/${COMMAND}-`date +%Y-%b-%d-%H-%M-%S-%N`.log
for DIR in "${BACKUP_DIR}" "${TMP_DIR}" "${LOG_DIR}"
do
    if [[ ! -d "'${DIR}'" ]];
    then
        echo Creating "'${DIR}'"
        sudo mkdir "${DIR}"
    fi
done

RSYNC_OPTS=""

#--dry-run, -n
RSYNC_OPTS=${RSYNC_OPTS}" --dry-run"
# --recursive, -r          recurse into directories
RSYNC_OPTS=${RSYNC_OPTS}" --recursive"
#--filter=RULE, -f        add a file-filtering RULE
RSYNC_OPTS=${RSYNC_OPTS}" --filter='dir-merge,p- .gitignore'"

# --checksum, -c           skip based on checksum, not mod-time & size
RSYNC_OPTS=${RSYNC_OPTS}" --checksum"

echo "rsync ${RSYNC_OPTS} '${SRC}' '${DEST}'" | sudo tee "${LOG_FILE}"
echo --------
echo --------
echo
echo
set -x
sudo rsync "${RSYNC_OPTS} '${SRC}' '${DEST}'"

Produces this:

Creating '/tmp/backup prep'
mkdir: cannot create directory ‘/tmp/backup prep’: File exists
Creating '/tmp/backup prep/tmp'
mkdir: cannot create directory ‘/tmp/backup prep/tmp’: File exists
Creating '/tmp/backup prep/tmp'
mkdir: cannot create directory ‘/tmp/backup prep/tmp’: File exists
rsync  --dry-run --recursive --filter='dir-merge,p- .gitignore' --checksum '/etc/' '/tmp/backup prep/etc'
--------


+ sudo rsync ' --dry-run --recursive --filter='\''dir-merge,p- .gitignore'\'' --checksum '\''/etc/'\'' '\''/tmp/backup prep/etc'\'''
rsync: [sender] change_dir "/home/aaron/bin/ --dry-run --recursive --filter='dir-merge,p- .gitignore' --checksum '/etc/' '/tmp/backup prep" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1333) [sender=3.2.3]

And appears to do nothing.

The thing is, if I sudo the echoed command line, sudo rsync --dry-run --recursive --filter='dir-merge,p- .gitignore' --checksum '/etc/' '/tmp/backup prep/etc' Everything seems to work as expected.

Shell expansion gets me every bloody time. It'd be nice if there were a command that'd show you what the hell is going on.

[Arrays](https://www.gnu.org/software/bash/manual/bash.html#Arrays) are your friends. Store arguments in an array, not a regular parameter. — chepner, Sep 20 '22 at 14:08
Also, `-d "'${DIR}'"` tests if the name constructed from single quotes and the value of `$DIR` is a directory. Drop the single quotes. — chepner, Sep 20 '22 at 14:11
"Shell expansion gets me every bloody time" Funny how folks usually reach for bash when the want a simple, straightforward scripting language, but shell scripting is neither simple nor straightforward. you could always use a more comfortable language for this -say, python, looking at your question history — erik258, Sep 20 '22 at 14:26
`It'd be nice if there were a command that'd show you what the hell is going on.` You can execute the script with `/bin/bash -x` or put `set -x` at the top of the script to get debug output which will give you a picture of how bash is interpreting the faulty line. — tjm3772, Sep 20 '22 at 15:02
If you put `set -x` in your terminal before copy and pasting the command you will see some differences between the output of your script and the output on the CLI. — tjm3772, Sep 20 '22 at 15:05
The string between double quotes in `rsync "${RSYNC_OPTS} '${SRC}' '${DEST}'"` will be expanded as a single word (that is, `rsync` will see it as a single argument). You may want to read [BashFAQ/050](https://mywiki.wooledge.org/BashFAQ/050) — M. Nejat Aydin, Sep 20 '22 at 15:46
It's better to use $(code) instead of \`code\`. These \`\` considered as deprecated and read worst than $(). Specially in this case with a lot of ' and ". — Ivan, Sep 20 '22 at 17:02
@Ivan Obsolete, but never actually deprecated. (Deprecation is a stated intent to drop support for something. Backticks aren't recommended, but there are no plans to get rid of them.) — chepner, Sep 21 '22 at 12:05

score 3 · Answer 1 · answered Sep 21 '22 at 08:22

Here is my finished script. Many thanks to the commentors on my question. With your hints I was able to make progress.

For what it's worth, the objective was to periodically back up a /etc that is using etckeeper and keep the .git repo, but ignore stuff in .gitignore

Some observations:

When building a list of arguments for a command, an array is totally the way to go.
When using the array, the shell beautifully quotes values with spaces. Yo can see this by using set -x in the script below.
Using set -x is really helpful.
When you're struggling to get something working shellcheck is an amazing diagnostic tool.

#!/bin/bash
COMMAND=$(basename "${0}")
SRC="/etc"
BACKUP_DIR="/root/backup prep"
TMP_DIR="${BACKUP_DIR}/tmp"
LOG_DIR="${TMP_DIR}"

if ! sudo -v
then
    echo ${COMMAND} quitting...
    exit 1
fi

SRC=$(realpath "${SRC}") # make relative paths absolute
SRC="${SRC}"/ # force rsync to disallow symlinks to the parent path

# https://www.shellcheck.net/wiki/SC2086
# Note that $( ) starts a new context, and variables in it have to
# be quoted independently
LOG_FILE="${LOG_DIR}/${COMMAND}-$(date +%Y-%b-%d-%H-%M-%S-%N).log"
for DIR in "${BACKUP_DIR}" "${TMP_DIR}" "${LOG_DIR}"
do
    if sudo test ! -d "${DIR}";
    then
        echo Creating "${DIR}"
        sudo mkdir "${DIR}"
    fi
done

RSYNC_OPTS=()

# ------------------------------------
# -------------Testing----------------
# ------------------------------------
#
#--dry-run, -n
#RSYNC_OPTS+=("--dry-run")

# ------------------------------------
# -------------Verbosity--------------
# ------------------------------------
#
# --verbose, -v            increase verbosity
#RSYNC_OPTS+=("--verbose")
# --progress               show progress during transfer
#RSYNC_OPTS+=("--progress")
# --itemize-changes, -i    output a change-summary for all updates
#RSYNC_OPTS+=("--itemize-changes")
# --stats                  give some file-transfer stats
#RSYNC_OPTS+=("--stats")
# --human-readable, -h     output numbers in a human-readable format
RSYNC_OPTS+=("--human-readable")
# --log-file=FILE          log what we're doing to the specified FILE
RSYNC_OPTS+=("--log-file=${LOG_FILE}")
# --log-file-format=FMT    log updates using the specified FMT


# ------------------------------------
# -------------Permissions------------
# ------------------------------------
#
# --owner, -o              preserve owner (super-user only)
RSYNC_OPTS+=("--owner")
# --group, -g              preserve group
RSYNC_OPTS+=("--group")
# --perms, -p              preserve permissions
RSYNC_OPTS+=("--perms")
# --xattrs, -X             preserve extended attributes
RSYNC_OPTS+=("--xattrs")
# --acls, -A               preserve ACLs (implies --perms)
RSYNC_OPTS+=("--acls")

# ------------------------------------
# -------------Times------------------
# ------------------------------------
#
# --times, -t              preserve modification times
RSYNC_OPTS+=("--times")
# --crtimes, -N            preserve create times (newness)
# rsync: This rsync does not support --crtimes (-N)
# RSYNC_OPTS+=("--crtimes")
# --atimes, -U             preserve access (use) times
RSYNC_OPTS+=("--atimes")
# --open-noatime           avoid changing the atime on opened files
RSYNC_OPTS+=("--open-noatime")

# ------------------------------------
# -------------Where------------------
# ------------------------------------
#
# --mkpath                 create the destination's path component
RSYNC_OPTS+=("--mkpath")
# --recursive, -r          recurse into directories
RSYNC_OPTS+=("--recursive")
# --links, -l              copy symlinks as symlinks
RSYNC_OPTS+=("--links")
# Rsync can also distinguish "safe" and "unsafe" symbolic links.  An
# example where this might be used is a web site mirror that wishes to
# ensure that the rsync module that is copied does not include symbolic
# links to /etc/passwd in  the public section of the site.  Using
# --copy-unsafe-links will cause any links to be copied as the file they
# point to on the destination.  Using --safe-links will cause unsafe
# links to be omitted altogether. (Note that you  must  specify --links
# for --safe-links to have any effect.)
# --copy-unsafe-links      where the link would point outside of the new tree, copy the file
RSYNC_OPTS+=("--copy-unsafe-links")
# --hard-links, -H         preserve hard links
RSYNC_OPTS+=("--hard-links")
# --one-file-system, -x    don't cross filesystem boundaries
RSYNC_OPTS+=("--one-file-system")

# ------------------------------------
# -------------Exclusions-------------
# ------------------------------------
#
#--filter=RULE, -f        add a file-filtering RULE
RSYNC_OPTS+=("--filter=dir-merge,p- .gitignore")
# --delete-excluded        also delete excluded files from dest dirs
RSYNC_OPTS+=("--delete-excluded")
#--delete-after           receiver deletes after transfer, not during
RSYNC_OPTS+=("--delete-after")
# --force                  force deletion of dirs even if not empty
RSYNC_OPTS+=("--force")
# --update, -u             skip files that are newer on the receiver
#RSYNC_OPTS=${RSYNC_OPTS}" --update")

# ------------------------------------
# -------------Misc-------------------
# ------------------------------------
#
# --protect-args, -s       no space-splitting; wildcard chars only
# This option sends all filenames and most options to the remote rsync
# without allowing the remote shell  to  interpret  them.
RSYNC_OPTS+=("--protect-args")
# --checksum, -c           skip based on checksum, not mod-time & size
RSYNC_OPTS+=("--checksum")
# --temp-dir=DIR, -T       create temporary files in directory DIR
RSYNC_OPTS+=("--temp-dir=${TMP_DIR}")

#echo sudo rsync "${RSYNC_OPTS[@]}" "'${SRC}'" "'${DEST}'"
#echo
#set -x
sudo rsync "${RSYNC_OPTS[@]}" "${SRC}" "${DEST}"

Bash - Expanding variables for command arguments leads to quoting hell

1 Answers1