How to preceed the output generated by "exec &" with a time/Date in linux bash scripts?

Question

I have the following script file that writes files to s3 from a local file system:

#!/bin/bash
CURR_DIR=`dirname $0`
SCRIPT_NAME="$(basename $0)"
LOG_FILE=$(echo $SCRIPT_NAME | cut -f 1 -d '.')
TODAY=$(date '+%Y-%m-%d')
NOW=$(date -d "$(date +%Y-%m-%d)" +%Y"-"%m"-"%d)
LOG_PATH="$CURR_DIR"/logs/"$LOG_FILE"-$TODAY.log
LOG="[$(date '+%Y-%m-%d %H:%M:%S,%3N')]  INFO {$LOG_FILE} -"
ERROR_LOG="[$(date '+%Y-%m-%d %H:%M:%S,%3N')]  ERROR {$LOG_FILE} -"

BUCKET="s3.bucket.example"

OUT_FOLDER="path/to/folderA"
S3_PUSH="s3://$BUCKET/$OUT_FOLDER"

exec &>> $LOG_PATH

echo "$LOG Copying files to local out folder..." >> $LOG_PATH
cp /path/to/folderA/*.* /path/to/folderB
echo "$LOG Command returned code:" $?

if [ "$(ls -A path/to/folderA/)" ]; then
  FILES="$(ls path/to/folderA/*)"
  for file in $FILES ; do
    echo "$LOG File $file found for sync" >> $LOG_PATH
    echo "$LOG Pushing $file to S3 /Folder..." >> $LOG_PATH
    echo -n "$LOG " ; s3cmd put -c /home/config/.s3cfg "$file" "$S3_PUSH"/
    echo "$LOG Command returned code:" $?
    echo "$LOG Copying $file to local backup..." >> $LOG_PATH
    mv "$file" /path/to/folderA/backup/
    echo "$LOG Command returned code:" $? >> $LOG_PATH
    RCC=$?
    if [ $? -eq 0 ]
    then
        echo "$LOG Command returned code:" $?
    else
        echo "$ERROR_LOG Command returned code:" $?
    fi
  done
else
  echo "$LOG No files found for sync." >> $LOG_PATH
fi

And the output is coming out in a specific grok pattern needed for me to parse this output as logs into Elastic Search, however the line 27 output is as follows:

[2021-09-02 08:15:25,629]  INFO {TestGrokScriptPattern} - upload: '/path/to/folderA/File.txt' -> 's3://s3.bucket.example/Path/To/Bucket/File.txt'  [1 of 1]
 0 of 0     0% in    0s     0.00 B/s  done

that upload and 0 of 0 0%... Line is created by the exec & command executed on line 16.

How can I get that output to not go to the next line without the date, time and script name preceeding it in order to not break the log pattern I am trying to create?

I suspect that your `0 of 0 0%` lines are actually coming from `s3cmd`. You might want to try adding the `-q` or `--no-progress` options to that command. — Andrew Vickers, Sep 02 '21 at 13:10
This script has a number of other problems. For one thing, you're using `$?` wrong: with `echo "... $?"; RCC=$?; if [ $? -eq 0 ]; then echo "... $?`, only the first is giving the status of the previous command. The `RCC=$?` command is recording the exit status of the `echo` command (almost certainly success), the `if` is testing the status from `RCC=$?` (likewise), the `echo` command is printing the status of the `[ ]` test, etc. [shellcheck.net](https://www.shellcheck.net) will point out more. — Gordon Davisson, Sep 03 '21 at 00:12
Thanks @AndrewVickers the --no-progress was the solution I needed. Thanks! — Jefledge, Sep 03 '21 at 08:08

score 1 · Accepted Answer · answered Sep 02 '21 at 13:21

Rather than redirect output on each line, you can wrap the body of the script in a single block and then handle the output of the entire block in one place. You can then process that output with the stream editor sed. For example:

if true; then # Always true. Just simplifies redirection.
  echo "Doing something..."
  command_with_output
  command_with_more_output
  echo "Done."
fi | sed "s/^/${LOG}/" > ${LOG_PATH} 2>&1

The sed expression means: Substitute (s) the beginning of each line (^) with the contents of the LOG variable.

Using 2>&1 at the end also eliminates the need for the exec &>> $LOG_PATH command.

How to preceed the output generated by "exec &" with a time/Date in linux bash scripts?

1 Answers1