assign output of a cat | sed | awk to an array

Question

Updated Question with code.

I have written a script that retrieves files from a remote server via scp. Then I want the script to delete the success file that has been retrieved. But instead of removing the successful file I want to put it into an array and display each.

#!/bin/bash
#
#------------------------------------------------[ Variables ]-------------------------------------
#need to put the full path so that oracle knows about it
export STARTTIME=$(/usr/bin/date +%H:%M:%S)
export PROGNAME=$(basename $0)
export FILEDATESTAMP=$(/usr/bin/date +%Y%m%d_%H%M)

. /usr/local/bondscape/sbin/profile
#------------------------------------------------[ Constants ]-------------------------------------
# Configuration
SCRIPT="$(basename $(readlink --canonicalize --no-newline $0))"
SCRIPT_DIR="$(dirname $(readlink --canonicalize --no-newline $0))"t with te
SCRIPT_FILENAME="${SCRIPT%.*}"
SCRIPT_FILEEXT="${SCRIPT##*.}"
SCRIPT_FILEPATH="${SCRIPT_DIR}"
LOGDIR="/tmp"
LOG=${LOGDIR}/GetFile-${FILEDATESTAMP}.log.$$
TRX_LOG=/tmp/GetFile.$$

if [[ $# -lt 4 ]] ; then  # Or count gt 0 and then process, or as original $4 is NULL/empty
  echo "Usage: $0 [source path] [destination path] [scp server] [file pattern]" >&2
  exit 1
fi
IFS=',' read -ra FILEMASKS <<< "$4"

#sending to tmp log file
echo "-----------------------------------------------------" >> $LOG
echo "StartTime ${STARTTIME}" >> $LOG
echo "Push Files to ${3}" >> $LOG
echo "Filemasks are ${FILEMASKS[@]}" >> $LOG
echo "Source Path is ${1}" >> $LOG
echo "Dest Path is ${2}" >> $LOG
echo "SCP Server is ${3}" >> $LOG
echo "-----------------------------------------------------" >> $LOG
#SCP to get the file
USER=oracle


for FILE in "${FILEMASKS[@]}"; do
  script -q -c "scp ${USER}@${3}:${1}/${FILE}* ${2}" >> ${TRX_LOG}
done

FILEMASK=${4}
 awk -v searchterm="${FILEMASK}" '$0 ~ searchterm {IGNORECASE=1 ; print $1 }' ${TRX_LOG} >> $LOG
 echo "-----------------------------------------------------" >> /tmp/GetFile.$$
 echo " Transfer From ASH: "
 echo "-----------------------------------------------------"
 echo "Show the array -"
 SUCCESSXFER=()
 SUCCESSXFER=$( (grep 100% /tmp/tmpGetFile | awk '{print $1}' /tmp/tmpGetFile) )
 for item in "${SUCCESSXFER[@]}"; do
    #testing the items in the array
    echo "Array item is - $item"
SUCCESSXFER=()
 SUCCESSXFER=$( (grep 100% /tmp/tmpGetFile | awk '{print $1}' /tmp/tmpGetFile) )
 for item in "${SUCCESSXFER[@]}"; do
 #testing the items in the array
  echo "Array item is - $item"
 done
 exit 0

Ok when I run it I get the following

/GetFilev6 /home/oracle/ /tmp/ vbnexfer01.housing.qld.gov.au ten*,TEN*
Transfer From ASH:
-----------------------------------------------------
Show the array -
Array item is - tenant.7366
tenant.7367
tenant.7369
tentt1
tentt2
TENANT
TENANT12
TENANT.7366
TENANT.7367
TENANT.7369

But when I did some manual testing I see everything has been put into array position 0.

#echo ${SUCCESSXFER[0]}

tenant.7366 tenant.7367 tenant.7369 tentt1 tentt2 TENANT TENANT12 TENANT.7366 TENANT.7367 TENANT.7369

So the question is, how can I assign successful file transfer into a separate position? As I will then use this array to delete the file via ssh

ssh server 'rm -r $arrayitem'

Sorry the output of the command each value is on a separate line TENANT TENANT12 TENANT.7366 TENANT.7367 TENANT.7369 tenant.7366 tenant.7367 tenant.7369 tentt1 tentt2 — goced, Jul 26 '23 at 04:43
`....| grep 100% > /tmp/tmpGetFile | awk '{print $1}' /tmp/tmpGetFile` ?? Having a pipe between a redirected file and reading from that same redirected file doesn't make sense. Please paste your code with a `#!/bin/bash` first line into https://shellcheck.net and make any changes recommended. If you code still doesn't work, then update your question with corrected code, else please delet your question. Good luck. — shellter, Jul 26 '23 at 05:12
This line looks garbled near the end: `SCRIPT_DIR="$(dirname $(readlink --canonicalize --no-newline $0))"t with te`. — Jens, Jul 26 '23 at 17:02
With `awk -v searchterm="${FILEMASK}" '$0 ~ searchterm {IGNORECASE=1` you're telling awk to make the `$0 ~ searchterm` comparison case-insensitive **after** you've already done that comparison once so it won't apply until the second time the values are compared. Set `IGNORECASE` in a BEGIN section or similar. — Ed Morton, Jul 26 '23 at 17:34
You never need grep when you're using awk. `grep '100%' /tmp/tmpGetFile | awk '{print $1}'` = `awk '/100%/{print $1}' /tmp/tmpGetFile` — Ed Morton, Jul 26 '23 at 17:37

dodrg · Answer 1 · 2023-08-11T13:52:37.240

The problem of your question reduces to this line:

SUCCESSXFER=$( (grep 100% /tmp/tmpGetFile | awk '{print $1}' /tmp/tmpGetFile) )

The fault is the position of the $:

# Your construction:
VAR=$( (command) )

# executes to
VAR='out1 out2 outN'

=> The assignement goes to variable but not an array.

# Your intention:
VAR=( $(command) )

# executes to
VAR=( out1 out2 outN )

=> Resulting in a compound assignement of an indexed array (as expected).

You line contains an additional error:

By appending /tmp/tmpGetFile to the awk that should receive the output of the | pipe, the pipe is ignored and the content of the unfiltered file is printed.

Correcting both errors the code line is:

SUCCESSXFER=( $(grep 100% /tmp/tmpGetFile | awk '{print $1}') )

As already noted in the comments, the call of grep, when using awk is inefficient as awk can do the job of grep, too:

SUCCESSXFER=( $(awk '$0 ~ /100%/{print $1}' /tmp/tmpGetFile) )

But I' wondering about the line TRX_LOG=/tmp/GetFile.$$. Probably you meant this file and so the code line changes to

SUCCESSXFER=( $(awk '$0 ~ /100%/{print $1}' "${TRX_LOG}") )

This should be the new line of code.

Seeing the additional problems and the comments below your question, its obvious:
You really should revise your whole script.

assign output of a cat | sed | awk to an array

1 Answers1