Script variables not interpreted by sed in cronjob

Question

Bash script in Debian 11 Bullseye making use of "sed" runs ok from command line but not as a cron

Hi, I'm running Debian GNU/Linux 11 (bullseye) Linux 5.15.84-v8+ #1613 SMP PREEMPT Thu Jan 5 12:03:08 GMT 2023 aarch64 GNU/Linux

In my setting, I have two bash scripts:

A different script fills in myfolder with JPEG files every minute 24 hours a day, non-stop. It's a cronjob and runs flawlessly.

When run from command line in /home/myuser the script below runs flawlessly and creates a file that lists all JPEG files in myfolder between start time and finish time

eg of filename 2023-03-05_1014.jpg

eg of output.txt created from command line myfolder/2023-03-05_1014.jpg

#!/bin/bash
cd /home/myuser/
current_date=$(date +%F)
time_start=2023-03-05_0900
time_finish=2023-03-05_1700

ls myfolder/"$current_date"_*.jpg | sed -n "/myfolder\/$time_start/, /myfolder\/$time_finish/p" > output.txt

When run as a cronjob as user myuser (not root)

00 22 * * * /home/myuser/create_jpg_list.sh 2>&1

output.txt will list every single file in myfolder with every JPEG file listed from 0000 through 2359. That is, the variables in the sed command do not seem to be interpreted.

eg of output.txt created from cronjob myfolder/2023-03-05_2302.jpg this line should not show in output.txt because it is out of the time range 9am to 5pm

I have searched online and there are a few threads such as the links below (to name a few) dealing with this scenario but alas https://unix.stackexchange.com/questions/614759/sed-doesnt-work-under-cron https://askubuntu.com/questions/524726/not-able-to-run-sed-on-bash-script https://unix.stackexchange.com/questions/662851/why-this-script-doesnt-work-from-cron-while-it-works-when-invoke-from-my-shell https://www.linuxquestions.org/questions/linux-newbie-8/sed-not-working-from-cron-706865/

Any feedback is welcome as I do not know what else to try.

Answer 1

The sed address start/finish conditions are "categorical". That is, it is fixed at first match. So, while that works OK for the start condition, capturing the first of many matches, that does not work for the end condition, giving you only the first of many.

Also, there is no "comparison" on addresses, so you can't use 1701 as a specification of an end condition.

awk is a better choice for that kind of fine-grained control, while giving greater flexibility on some kinds of matches.

The script below incorporates a demo mode (no need to specify command-line parameters) which will demonstrate the output for both, your original sed address matching, and the awk method to give you the result you wanted.

Logic demonstration script:

#!/bin/bash

demo=0 
sel_date=""

while [ $# -gt -0 ]
do
    case $1 in
        --demo )    demo=1 ; shift ;;
        --today )   sel_date=$(date +%F) ; shift ;;
        --date )    sel_date="$2" ; shift ; shift ;;
        --start )   stime="${2}" ; shift ; shift ;;
        --end )     etime="${2}" ; shift ; shift ;;
        --dir )     startdir="${2}" ; shift ; shift ;;
        * ) echo "\n\t Invalid parameter used on command line.  Only valid options: [ --today | --date {yyyy-mm-dd} | --start {hhmm} | --end {hhmm} ]\n Bye!\n" ; exit 1 ; ;;
    esac
done

if [ ${demo} -eq 1  -o  -z "${startdir}" ]
then
    demo=1
    echo " ****  DEMO MODE IN EFFECT  **** "
    startdir="myuser"
fi
echo "${startdir}"

if [ ${demo} -eq 1  -o  -z "${sel_date}" ]
then
    sel_date="2023-03-05"
fi
echo "${sel_date}"

if [ ${demo} -eq 1  -o  -z "${stime}" ]
then
    stime="0900"
fi
echo "${stime}"

if [ ${demo} -eq 1  -o  -z "${etime}" ]
then
    etime="1700"
fi
echo "${etime}"


if [ ${demo} -eq 1 ]
then

    if [ ! -d myuser ]
    then
        mkdir myuser
    fi
    ###
    ###  Create test environment for logic confirmation
    ###
    for mdate in 2023-02-30  2023-03-05  2023-03-08
    do
        echo -e "\t ${mdate} ..."
        for mtime in 0700 0900 1100 1300 1500 1700 1900
        do
                echo "dummy" >"${startdir}/${mdate}_${mtime}.jpg"
            done
        done
    done
fi

cd `dirname "${startdir}" `

time_start="${sel_date}_${stime}"
time_finish="${sel_date}_${etime}"

ls -1 "${startdir}/${sel_date}_"*.jpg
echo ""

### Method 1 - Inappropriate
ls "${startdir}/${sel_date}_"*.jpg | sed -n "/${startdir}\/${time_start}/, /${startdir}\/${time_finish}/p" | tee output1.txt

wc -l output1.txt


### Method 2 - 
ls "${startdir}/${sel_date}_"*.jpg | 
awk -v pdate="${sel_date}" -v stime="${stime}" -v etime="${etime}" '{
    ### Input format:  'myuser/2023-03-05_0900_001.jpg'
    n=split( $0, line, "/" ) ;
    split( line[n], dat, "_" ) ;
    if( dat[1] == pdate ){
        if( dat[2] >= stime && dat[2] <= etime ){
            print $0 ;
        } ;
    } ;
}' | tee output2.txt
wc -l output2.txt