client_loop: send disconnect: Broken pipe error when ssh into remote host

Question

In my job, I need to ssh into multiple devices to run tests and check log files. I wrote a bash script to make the process easier. If I were to do it in terminal one by one, I ssh into a device, open a tmux session, run some commands, detach the tmux session, and then exit ssh. Sometimes the devices would reboot, which will exit their tmux sessions. In that case, the script checks if the device has a tmux session running. If it finds out there is no session running, it starts a new detached session and send commands to it. The thing is that I get client_loop: send disconnect: Broken pipe very often, which won't let me ssh and check the log file.

I had a similar task previously, but back then I did not have to use tmux and didn't really get this error. So, I am suspecting my tmux part of the script has some issue. And I feel like I get the error more often after running a new tmux session when there was nothing running. For example, when there is no tmux session running, it would start a new session and echo a message that it started a new session. When I run the script again, that device would have the broken pipe error. That is not always the case, but it was pretty often.

My script may look messy as I learned about bash script pretty recently just by googling. Pleas feel free to point out or make suggestion for me to improve my script. The following is my script. Thanks in advance!

#!/bin/bash

RED='\033[0;31m'
YELLOW='\033[0;33m'       # Yellow
NOCOL='\033[0m' # No Color

IP_last_digits=(
  '54' '65' '60' '56' '55'
  '64' '57' '61' '66' '52' 
  '73' 
)

for ((j = 1; j <= "${#IP_last_digits[@]}"; j++)); do
  echo "Camera $j (10.65.4."${IP_last_digits[j-1]}")"
  
  if ! ssh root@10.65.4."${IP_last_digits[j-1]}" " "; then
    echo -e "  >> ${RED}SSH connection failed${NOCOL}"
    echo ""
    continue
  fi

 
  tmuxMsg=$(ssh root@10.65.4."${IP_last_digits[j-1]}" "tmux ls")
  if [[ $tmuxMsg == "" ]]; then
    echo -e "  >> ${YELLOW}NO TMUX SESSION FOUND. STARTING A NEW SESSION. ${NOCOL}"
    ssh root@10.65.4."${IP_last_digits[j-1]}" "
      echo 'file ambarella_sd.c +p' > /sys/kernel/debug/dynamic_debug/control
      tmux new -d -s 0
      sleep 1
      tmux send -t 0 'dmesg -w | grep -A 20 -B 20 error > /mnt/media/dmesg_mmc.txt' ENTER
    "
    echo ""
  fi
  
  tmux_ls=$(ssh root@10.65.4."${IP_last_digits[j-1]}" "tmux ls")
  echo "  >> $tmux_ls"

  
  string=$(ssh root@10.65.4."${IP_last_digits[j-1]}" "cat /mnt/media/dmesg_mmc.txt")
  if [[ $string == "" ]]; then
    echo "  >> No contents (or error) found in /mnt/media/dmesg_mmc.txt"
  else
    echo -e "  >> ${RED}ERROR FOUND. CHECK /mnt/media/dmesg_mmc.txt${NOCOL}"
  fi
  
  num_files=$(ssh root@10.65.4."${IP_last_digits[j-1]}" "ls -a /mnt/sdcard0/ | wc -l")
  if [[ $num_files == 0 ]]; then
    echo -e "  >> ${RED}ERROR: NO FILE FOUND IN THE SD CARD${NOCOL}"
  else
    echo "  >> Number of files in /mnt/sdcard0/ is $num_files"
  fi
  
  echo ""
  echo ""
done

I did not try anything because I have no idea what I am doing wrong. But I suspect that it has something to do with how I run tmux.

client_loop: send disconnect: Broken pipe error when ssh into remote host

0 Answers0