0

I have three remote machines (machineA, machineB, machineC) from where I can copy files. If for whatever reason I can't copy from machineA, then I should copy from machineB and if for whatever reason I can't copy from machineB then start copying from machineC.

Below is the single shell command I have and I need to run it on many machines but then it means on all those machines, it will copy from machineA only.

(ssh goldy@machineA 'ls -1 /process/snap/20180418/*' | parallel -j5 'scp goldy@machineA:{} /data/files/') || (ssh goldy@machineB 'ls -1 /process/snap/20180418/*' | parallel -j5 'scp goldy@machineB:{} /data/files/') || (ssh goldy@machineC 'ls -1 /process/snap/20180418/*' | parallel -j5 'scp goldy@machineC:{} /data/files/')

Now is there any way by which I can pick first machine randomly (out of those three) instead of keeping machineA as first always. So pick first machine randomly and keep other two as the backup incase first machine is down? Is this possible to do?

Update:

I have something like this:

  machines=(machineA machineB machineC)
  for machine in $(shuf -e ${machines[@]}); do
      ssh -o StrictHostKeyChecking=no david@$machine 'ls -1 /process/snap/{{ folder }}/*' | parallel -j{{ threads }} 'scp -o StrictHostKeyChecking=no david@${machine}:{} /data/files/'
      [ $? -eq 0 ] && break
  done
user1950349
  • 4,738
  • 19
  • 67
  • 119

2 Answers2

1

How about keeping the machine names in a file and using shuf to shuffle them? Then you could create a script like this:

while read machine; do
    ssh goldy@$machine 'ls -1 /process/snap/20180418/*' | parallel -j5 "scp goldy@$machine:{} /data/files/"
    if [ $? == 0 ]; then
        break
    fi
done

And the machine file like this:

machineA
machineB
machineC

And call the script like this:

shuf machines | ./script.sh

Here's a test version that doesn't do anything but shows how the logic works:

while read machine; do
    echo ssh goldy@$machine 'ls -1 /process/snap/20180418/*'
    echo parallel -j5 "scp goldy@$machine:{} /data/files/"
    executenonexistingcommand
    if [ $? == 0 ]; then
        break
    fi
done

Addressing your comment to use arrays instead and put everything on a single line:

shuf -e ${machines[@]} shuffles an array. And to read it back into the array, you need to feed the outpu into readarray. Turning scripts into a single line is just a matter of putting semicolons where we had newlines before.

machines=( machineA machineB machineC ); for machine in $(shuf -e ${machines[@]}); do ssh goldy@$machine 'ls -1 /process/snap/20180418/*' | parallel -j5 "scp goldy@${machine}:{} /data/files/"; if [ $? == 0 ]; then break; fi; done
sneep
  • 1,828
  • 14
  • 19
  • interesting idea.. Is there any way by which we can do everything in one line (I can format that `while done` loop in one line for sure) and also instead of keeping machines in a file, can we shuffle an array if possible? This will help me to do everything in one line but as one command. – user1950349 Apr 29 '18 at 06:42
  • Updated answer. – sneep Apr 29 '18 at 07:25
  • got it. so that means my whole one line command with shuffling array will look like this `machines=( machineA machineB machineC ); shuf -e ${machines[@]}; for machine in "${machines[@]}"; do ssh goldy@$machine 'ls -1 /process/snap/20180418/*'; parallel -j5 "scp goldy@${machine}:{} /data/files/"; if [ $? == 0 ]; then break; fi; done`? – user1950349 Apr 29 '18 at 07:36
  • I tested above command I shared to you and it always uses machineA in my command so I am sure I have something wrong above. Also It looks to me inner scp line is not able to figure out variable `machine` for some reason? – user1950349 Apr 29 '18 at 08:12
  • Sorry, wrote something slightly different than what I had intended. We need to loop over the result of `shuf` of course. I'd also accidentally changed a `|` character to a semicolon in the loop. Updated post. – sneep Apr 29 '18 at 08:48
  • yeah it works fine now. Also I updated my question by adding a script, just wanted to make sure I have got everything correct or not? And as soon as it copies all the files from one machine then it won't copy same files from other boxes right? Because at the end we are checking the status using this `[ $? -eq 0 ] && break` and if it is successful then we break out of it. Also I have this `[ $? -eq 0 ] && break` and you have `[ $? == 0 ]` they both are same right? Meaning I am using `eq` and you are using `==`. – user1950349 Apr 30 '18 at 07:55
  • Your script looks correct. Note that it will retry with a different host no matter if the error is due to the machine being down or due to some other kind of error. You should be fine using `-eq` instead of `==`. They're not quite the same, but it doesn't matter much in this case. I also recommend you test the script before concluding that it's correct. – sneep Apr 30 '18 at 08:26
  • yeah make sense now. Also I have another question [here](https://stackoverflow.com/questions/50070866/compare-checksum-of-files-between-two-servers-and-report-mismatch). Wanted to see if you can help me out. – user1950349 Apr 30 '18 at 08:32
1

Here is a little example of how you might do it - it is largely comments, to show what I am thinking, but you can remove them to make it concise.

#!/bin/bash

# Machine names, number of machines, random starting index
machines=("machineA" "machineB" "machineC")
num=${#machines[@]}
idx=$((RANDOM%num))

# Make one try per machine, break on success
for ((try=0;try<num;try++)) ; do
   this=${machines[$idx]}
   echo $this
   ((idx=(idx+1)%num))
done

So, you would put your command where I have echo $this, and follow it with:

[ $? -eq 0 ] && break

Sample Output

./go
machineB
machineC
machineA

If you have shuf you can do the same thing more succinctly like this:

#!/bin/bash

# Machine names, in random order
machines=("machineA" "machineB" "machineC")
machines=( $(shuf -e "${machines[@]}") )

# Make one try per machine, break on success
for i in "${machines[@]}"; do
   echo $i
   ... your command
   [ $? -eq 0 ] && break
done
Mark Setchell
  • 191,897
  • 31
  • 273
  • 432