-1

I'm trying to rsync a DIR from one Server to 100s of Servers using script (Bottom)

But, When i put single or double quotes around ${host} variable, Host names are not picked properly or not resolved.

Error is like below 

server1.example.com
Host key verification failed.
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(600) [sender=3.0.6]

and when I run only with rync command like below, It works. But, Output doesn't contain hostname which is important for me to correlate the output with associated hostname. 

hostname -f && rsync -arpn --stats /usr/xyz ${host}:/usr/java

Can you please review and suggest me how to make the script work even with quotes around Host variable. ?

So, that , Output will contain hostname and output of rsync together.

==============================================

#!/bin/bash

tmpdir=${TMPDIR:-/home/user}/output.$$
mkdir -p $tmpdir
count=0

while IFS= read -r host; do
  ssh -n -o BatchMode=yes ${host}   '\
    hostname -f && \
    rsync -arpn --stats /usr/xyz '${host}':/usr/java && \
    ls -ltr /usr/xyz' 
  > ${tmpdir}/${host} 2>&1 &
  count=`expr $count + 1`
done < /home/user/servers/non_java7_nodes.list

while [ $count -gt 0 ]; do
    wait $pids
    count=`expr $count - 1`
done

echo "Output for hosts are in $tmpdir"
exit 0

UPDATE:

Based on observation with (set -x), Host name is being resolved on remote (self) it self, it supposed to be resolved on initiating host. I think Once we know how to make host name resolved with in initiating host even when quotes are in place.

Subbu
  • 25
  • 5
  • First try a single host without background processing or redirecting and a fixed host. Next try to use a variabele `host`. Do you want `${host}` to be filled with the current value of the Server where you started the script or being resolved on the remote server in the `ssh` session ? I think the remote server will not have a vatiable `host` set. – Walter A Oct 15 '18 at 21:08
  • Closing the single-quotes before evaluating the variable means it's expanded with the local value already, as-is. (Opening a double-quoted context after closing the single-quoted one would improve robustness in the face of unexpected values -- spaces, globs, etc -- but it's not a robustness issue the OP is complaining about presently). – Charles Duffy Oct 15 '18 at 21:11
  • 2
    ...the most obvious thing that *is* a problem here is that the `>` isn't applied to the `ssh` command because of the newline between them. – Charles Duffy Oct 15 '18 at 21:13
  • 1
    (Also, all the `expr` usage is better replaced with POSIX math: `count=$(( count + 1 ))`, or bash math: `(( ++count ))`) – Charles Duffy Oct 15 '18 at 21:15
  • 1
    ...beyond that, "Host key verification failed" probably means exactly what it says: The host being `ssh`'d to doesn't have a valid host key cached for the place rsync is in turn being told to connect to from there. Best practice is use a SSH CA to sign your host keys (so you can just distribute that one CA public key), but the cheap/easy/insecure "fix" is to turn off host key verification. – Charles Duffy Oct 15 '18 at 21:16
  • 1
    That said, the whole logic around ssh'ing to `"$host"` and then `rsync`ing a file from that host to itself doesn't make any sense either. That's more a place where the OP needs to decide what they actually want to do and write software that does that thing correctly, though. – Charles Duffy Oct 15 '18 at 21:18
  • @CharlesDuffy there are valid cases where one has to ssh to a host to run `rsync` or similar to a different remote host (because `rsync` does not support using remotes for both source and target). I've dealt with these before. That being said, it looks like the host is expanding correctly (although running with `set -x` will tell us definitively), but that the keys just aren't set up properly for the first remote host to ssh to all the other remotes (because `rsync` uses ssh by default). If you *do* have an ssh id, you can use the `-e "ssh -i /path/to/key"` argument to rsync. – Ch4ni Oct 15 '18 at 23:50
  • 1
    If the OP were using `hostA` in one place and `hostB` in another, rather than `"$host"` in both, it would make much more sense. And re: setting up keys, this is normally a place where agent forwarding (`ssh -A`) would be an appropriate approach... but that's RSA keys for authenticating the client, vs the keys for authenticating the host, which is where the OP's immediate issues are. – Charles Duffy Oct 15 '18 at 23:52
  • @CharlesDuffy ah ... I glossed over that part a little too quickly. That's a good point. It should be an ssh with a straight up local tgt and local dest, unless OP got the semantics incorrect ;-) – Ch4ni Oct 15 '18 at 23:54
  • Possible duplicate of [Difference between single and double quotes in Bash](https://stackoverflow.com/questions/6697753/difference-between-single-and-double-quotes-in-bash) – miken32 Oct 16 '18 at 00:23
  • Host where i run has password less ssh already setup with all remote hosts and same rsync works if i run just that command alone with in the script.. It is only an issue when i wrap it with quotes.. I tried all possible quotes around the command and variable but no much progress.. so i strongly believe .. somewhere in my script/syntax of rsync i'm not providing right quotes etc.. – Subbu Oct 16 '18 at 01:18
  • My requirement is basically to have this ${host} value filled with host name from the .list file that i'm passing in script. Again, This is not working ONLY when wrap whole syntax in quotes to redirect output to a file. – Subbu Oct 16 '18 at 01:20
  • @CharlesDuffy, After setting "set -x" , i see host name is being resolved correctly , Apparently it is resolving on remote (self) it self. It shouldn't be like that. rsync need to happen from Initiating host , i think once if we able to fix that, We are good here i believe ? – Subbu Oct 16 '18 at 01:32
  • If the rsync needs to happen on the initiating host, why did you put the rsync command inside the ssh command? That explicitly tells it to run remotely. – Charles Duffy Oct 16 '18 at 12:05
  • It's still not clear to me what the behavior you want is. Because you run `ssh "$host" 'hostname -f'`, **of course** the `hostname -f` will be run on the remote host, not the initiating host. And because the `rsync` is inside that same quotes, you're effectively running `ssh "$host" 'rsync /usr/xyz '"$host"':/usr/java'` as well, which likewise runs the rsync remotely, not locally. If you wanted rsync to be run locally, **why do you put it inside the string passed as an argument to the ssh command?** – Charles Duffy Oct 16 '18 at 15:08

1 Answers1

0

As far as I can tell, what you're looking for is something like:

#!/bin/bash

tmpdir=${TMPDIR:-/home/user}/output.$$
mkdir -p "$tmpdir"

host_for_pid=( )
while IFS= read -r host <&3; do
  {
    ssh -n -o BatchMode=yes "$host" 'hostname -f' && \
      rsync -arpn --stats /usr/xyz "$host:/usr/java" && \
      ssh -n -o BatchMode=yes "$host" 'ls -ltr /usr/java'
  } </dev/null >"${tmpdir}/${host}" 2>&1 & pids[$!]=$host
done 3< /home/user/servers/non_java7_nodes.list

for pid in "${!host_for_pid[@]}"; do
  if wait "$pid"; then
    :
  else
    echo "ERROR: Process for host ${host_for_pid[$pid]} had exit status $?" >&2
  fi
done

echo "Output for hosts are in $tmpdir"

Note that the rsync is no longer inside the ssh command, so it's run locally, not remotely.

Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
  • It works that way, But, rsync output would not be captured into file "${tmpdir}/${host}", How to get rsync and output along with hostname -f output captured into file.. so i can see which output is related to what server. – Subbu Oct 16 '18 at 20:07
  • To the contrary, `rsync` output **is** captured: The redirection applies to everything inside the `{` ... `}` section to which it's applied. If you don't see it captured, that's presumably rsync changing its default output format based on whether its output is going to a TTY; you'll be able to override that by adjusting its command-line arguments appropriately. – Charles Duffy Oct 16 '18 at 20:19