How to set variables in a subshell?

Question

I have a script which will be running in each server and copies certain files into it. Script knows where I am running and what files I need to copy.

Script will copy files from local datacenter local_dc but if it is down or not responding, then it will copy same files from remote datacenter remote_dc_1 and if that is also down, then it will copy same files from another remote datacenter remote_dc_2 as shown below.

Now Let's say if local_dc machine is down, then it will copy files from remote_dc_1 machine so I need to send out an email saying this machine is down so copying from other remote machine. I don't want to send multiple emails if local_dc machine is down so I am sending emails at the end so that I get only one email instead of multiple emails for each file.

Below is my script -

do_Copy() {
  el=$1
  PRIMSEC=$2
  scp david@"$local_dc":"$dir3"/new_weekly_2014_"$el"_200003_5.data "$PRIMSEC"/. \
    || (a=1 && scp david@"$remote_dc_1":"$dir3"/new_weekly_2014_"$el"_200003_5.data "$PRIMSEC"/.) \
    || (b=2 && scp david@"$remote_dc_2":"$dir3"/new_weekly_2014_"$el"_200003_5.data "$PRIMSEC"/.) \
    || (c=3 && exit 1)
}

export -f do_Copy

parallel --retries 10 -j 10 do_Copy {} $PRIMARY ::: "${PRIMARY_PARTITION[@]}" &
parallel --retries 10 -j 10 do_Copy {} $SECONDARY ::: "${SECONDARY_PARTITION[@]}" &
wait

# problem is this doesn't work at all?
if [ $a -eq 1 ]
then
   echo "Local machine $local_dc was down, so copied from Primary Remote machine $remote_dc_1" | mailx -r "david@host.com" -s "$local_dc machine down" "david@host.com"
fi
if [ $b -eq 2 ]
then
   echo "Primary Remote machine $remote_dc_1 was down, so copied from Secondary Remote machine $remote_dc_2" | mailx -r "david@host.com" -s "$remote_dc_1 machine down" "david@host.com"
fi
if [ $c -eq 3 ]
then
   echo "All three machine's are down. Exiting out." | mailx -r "david@host.com" -s "All three machine's are down" "david@host.com"
fi

Is there anything wrong I am doing in my subshell with variable a, b and c?

You can't convey variables back from a child process to a parent except out-of-band -- by writing to a file, for instance. — Charles Duffy, Apr 03 '15 at 01:08
@CharlesDuffy I see. Is there any other way by which I can solve my problem? — john, Apr 03 '15 at 01:10
First, let's back up and try to figure out the logic -- you're having parallel run a whole bunch of `do_Copy` instances, but then you're expecting to write only one string to `mail`, even though any one of them can set `a`, `b` or `c` flags, meaning that even if variables did propagate back to parent processes, you could get inconsistent flags set. What's the actual intent? — Charles Duffy, Apr 03 '15 at 01:12
(Personally, by the way, I utterly despise GNU parallel -- if you read its source, you'll see that it's an utter mess of spaghetti code; if you lean on me for advice, you'll probably end up with a solution that avoids it). — Charles Duffy, Apr 03 '15 at 01:13
@CharlesDuffy As an example let's say I am trying to copy 300 files in a machine. 150 files in primary and 150 files in secondary folder. My script will try to copy files from local machine but if it is down, then it will copy from remote_dc_1 machine so I want to send an email to alert me that local machine is down. So if I add mail logic in that scp command, then for each file transfer it will keep send me email. Meaning for 300 file scp, I will get 300 emails that local machine was down. so that's why I am sending email at the end — john, Apr 03 '15 at 01:15
If that's what you want, I hold with my prior advice: Use a flag on the filesystem. — Charles Duffy, Apr 03 '15 at 01:15
...so, instead of `a=1`, make the command `touch local_down`; instead of `a=2`, make it `touch primary_down`; instead of `a=3`, `touch secondary_down`; and then at the end: `if [[ -e primary_down ]]; then mail ... <<<"error message"`, and likewise for the other flags. — Charles Duffy, Apr 03 '15 at 01:17
My main intent is I should get alerted that this machine is down so it copied from other machine and if all machines are down, then also I should get alerted but only once not multiple times. — john, Apr 03 '15 at 01:17
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/74381/discussion-between-david-and-charles-duffy). — john, Apr 03 '15 at 01:18
BTW -- technically, a child processes started by parallel isn't a subshell anymore; it's a whole new shell, since it was created through an `exec()`-family call, not through a `fork()` direct from a preexisting shell. — Charles Duffy, Apr 03 '15 at 01:48

Charles Duffy · Accepted Answer · 2015-04-03T01:40:11.613

You can't propagate this status back with shell or environment variables, but a flag on the filesystem will do the job just fine.

#!/bin/bash
export status_dir=$(mktemp -t -d transfer.XXXXXX)
cleanup() { rm -rf "$status_dir"; }
trap cleanup 0 # automatically clean up on exit

do_Copy() {
  el=$1
  PRIMSEC=$2
  scp david@"$local_dc":"$dir3"/new_weekly_2014_"$el"_200003_5.data "$PRIMSEC"/. \
    || { touch "$status_dir/local_down" && scp david@"$remote_dc_1":"$dir3"/new_weekly_2014_"$el"_200003_5.data "$PRIMSEC"/.; } \
    || { touch "$status_dir/primary_down" && scp david@"$remote_dc_2":"$dir3"/new_weekly_2014_"$el"_200003_5.data "$PRIMSEC"/.; } \
    || { touch "$status_dir/secondary_down"; exit 1; }
}

...and later...

[[ -e "$status_dir/local_down" ]] && \
   mailx -r "david@host.com" -s "$local_dc machine down" "david@host.com" \
     <<<"Local machine $local_dc was down, so copied from Primary Remote machine $remote_dc_1"

[[ -e "$status_dir/primary_down" ]] && \
   mailx -r "david@host.com" -s "$remote_dc_1 machine down" "david@host.com" \
     <<<"Primary Remote machine $remote_dc_1 was down, so copied from Secondary Remote machine $remote_dc_2"

[[ -e "$status_dir/secondary_down" ]] && \
   mailx -r "david@host.com" -s "All three machine's are down" "david@host.com" \
     <<<"All three machines are down. Exiting out."

How to set variables in a subshell?

1 Answers1