Bash coding return a file name with specific string

Question

My script (in bash) aims to do this job:

1. gets start and stop time from a file, file_A. The time range usually is 3-24 hours.

2. Based on this time window of [start_time, stop_time] got from file_A, I need to find specific files among totally 10k log files(and will increase along with the experimental running), each of which recorded around 30 minutes. That's to say, I have to find 6-50 log files among 10k ones.

3. After confirming the correct log files, I need to print out interesting data.

Step 1) and 3) are OK, I did it already. Right now, I'm stuck in step 2), Especially in two places:

(a). How to select appropriate files efficiently by their names since the log files named as time. Every log file named as log_201305280650 which means 2013 / May 28 / 06 :50. That's to say, according to the time get from file_A, I need to confirm the corresponding log files by their names which is a hint of time.

(b). Once the files are selected, read the items(like temperature, pressure etc) from this file whose time is inside of time window. Because each file records 30 minutes, which means some of the entry in this file can't satisfy time window.

For instance,

From step 1), My time window is set to [201305280638, 201305290308].

from step 2), I know the log file(log_201305280650) contains the start time of 201305280638. So I need to read all of temperature and pressure for the entries below 201305280638.

    the log files name is log_201305280650 (= 2013 / May 28 / 06 :50)

    Time                      temperature  pressure ...
    201305280628                100,         120  ...
    201305280629                100,         120  ...

   ...              ...     ...

    201305280638                101,         121  ...
    201305280639                99,          122  ...

     ...             ...     ... 

    201305280649                101,         119  ...
    201305280650                102,         118  ...

My fake script is following.

get time_start from /path/file_A
get time_stop  from /path/file_A
for file in /path_to_log_files/*
do
case "$file" in
*)        
     If [[log file name within time window of (time_start, time_stop)]]; then
     loop over this file to get the entry whose time is just within (time_start, time_stop)
     read out temperature and pressure etc.
fi
esac
done

Answer 1

Quite a job using bash. Perl or python would have been easier, they both have date/time modules.

I spent a while doing the usual date slicing and it was horrible, so I cheated and used file timestamps instead. Bash has some limited timestamp checking, and this uses that. OK, it does some file IO, but these are empty files and what the hell!

lower=201305280638
upper=201305290308
filename=log_201305280638
filedate=${filename:4}

if (( filedate == upper )) || (( filedate == lower ))
then
    echo "$filename within range"
else
    # range files
    touch -t $lower lower.$$
    touch -t $upper upper.$$

    # benchmark file
    touch -t $filedate file.$$

    if [[ file.$$ -nt $upper ]]
    then
        echo "$filename is too young"

    elif [[ file.$$ -ot $lower ]]
    then
        echo "$filename is too old"
    else
        echo "$filename is just right"
    fi

    rm lower.$$ upper.$$ file.$$
fi

-nt is "newer-than"

-ot is "older-than"

Hence the check for equality at the start. You can use a similar check for the timestamps within the file (your second issue). But honestly, can't you use perl or python?

Answer 2

Maybe something along the lines of this would work for you? I am using $start and $end for the start and end times from file_A. I

 eval cat log_{$start..$end} 2> /dev/null | sort -k1 | sed -n "/$start/,/$end/p"

This assumes that your log files are in the format

time temperature pressure ...

with no headers or other such text

Answer 3

It may be easier to use awk and the +"%s" option of the date command in stead of literal date and time. This option converts date/time to seconds from epoch (01-01-1970). The resulting number is easy to work with. After all, it's just a number. As an example I made a small bash script. First, a simulation:

#!/bin/bash

#simulation: date and time
start_dt="2013-09-22 00:00:00"
end_dt="2013-09-22 00:00:00"
start_secs=$(date -d "start_dt" +"%s")
end_secs=$(date -d "end_dt" +"%s")
#simulation: set up table (time in secs, temperature, pressure per minute)
> logfile
for ((i=$start_secs;i<$end_secs;i=i+60)); do
    echo $i $[90+$[RANDOM %20]] $[80+$[RANDOM %30]] >> logfile
done

Here's the actual script to get the user range and to print it out:

echo "Enter start of range:"
read -p "Date (YYYY-MM-DD): "sdate
read -p "Time (HH:MM:SS)  : "stime
echo "Enter end of range:"
read -p "Date (YYYY-MM-DD): "edate
read -p "Time (HH:MM:SS)  : "etime
#convert to secs
rstart=$(date -d "$sdate $stime" +"%s")
rend=$(date -d "$edate $etime" +"%s")
#print it to screen
awk -v rstart=$rstart -v rend=$rend '{if($1 >= rstart && $1 <= rend)print $0}' logfile

The awk command is very suited for this. It is fast and can handle large files. I hope this gives you ideas.