13

To get the execution time of any executable, say a.out, I can simply write time ./a.out. This will output a real time, user time and system time.

Is it possible write a bash script that runs the program numerous times and calculates and outputs the average real execution time?

F. Hauri - Give Up GitHub
  • 64,122
  • 17
  • 116
  • 137
mooncow
  • 413
  • 4
  • 11
  • 2
    The only caveat to be aware of is depending on how much of your code remains in cache, you subsequent runs will be artificially faster due to the cache. – David C. Rankin Feb 28 '19 at 07:46

4 Answers4

19

You could write a loop and collect the output of time command and pipe it to awk to compute the average:

avg_time() {
    #
    # usage: avg_time n command ...
    #
    n=$1; shift
    (($# > 0)) || return                   # bail if no command given
    for ((i = 0; i < n; i++)); do
        { time -p "$@" &>/dev/null; } 2>&1 # ignore the output of the command
                                           # but collect time's output in stdout
    done | awk '
        /real/ { real = real + $2; nr++ }
        /user/ { user = user + $2; nu++ }
        /sys/  { sys  = sys  + $2; ns++}
        END    {
                 if (nr>0) printf("real %f\n", real/nr);
                 if (nu>0) printf("user %f\n", user/nu);
                 if (ns>0) printf("sys %f\n",  sys/ns)
               }'
}

Example:

avg_time 5 sleep 1

would give you

real 1.000000
user 0.000000
sys 0.000000

This can be easily enhanced to:

  • sleep for a given amount of time between executions
  • sleep for a random time (within a certain range) between executions

Meaning of time -p from man time:

   -p
      When in the POSIX locale, use the precise traditional format

      "real %f\nuser %f\nsys %f\n"

      (with  numbers  in seconds) where the number of decimals in the
      output for %f is unspecified but is sufficient to express the
      clock tick accuracy, and at least one.

You may want to check out this command-line benchmarking tool as well:

sharkdp/hyperfine

codeforester
  • 39,467
  • 16
  • 112
  • 140
4

Total execution time vs sum of single execution time

Care! dividing sum of N rounded execution time is imprecise!

Instead, we could divide total execution time of N iteration (by N)

avg_time_alt() { 
    local -i n=$1
    local foo real sys user
    shift
    (($# > 0)) || return;
    { read foo real; read foo user; read foo sys ;} < <(
        { time -p for((;n--;)){ "$@" &>/dev/null ;} ;} 2>&1
    )
    printf "real: %.5f\nuser: %.5f\nsys : %.5f\n" $(
        bc -l <<<"$real/$n;$user/$n;$sys/$n;" )
}

Nota: This uses bc instead of awk to compute the average. For this, we would create a temporary bc file:

printf >/tmp/test-pi.bc "scale=%d;\npi=4*a(1);\nquit\n" 60

This would compute with 60 decimals, then exit quietly. (You can adapt number of decimals for your host.)

Demo:

avg_time_alt 1000 sleep .001
real: 0.00195
user: 0.00008
sys : 0.00016

avg_time_alt 1000 bc -ql /tmp/test-pi.bc
real: 0.00172
user: 0.00120
sys : 0.00058

Where codeforester's function will anser:

avg_time 1000 sleep .001
real 0.000000
user 0.000000
sys 0.000000

avg_time 1000 bc -ql /tmp/test-pi.bc
real 0.000000
user 0.000000
sys 0.000000

Alternative, inspired by choroba's answer, using Linux's/proc

Ok, you could consider:

avgByProc() { 
    local foo start end n=$1 e=$1 values times
    shift;
    export n;
    { 
        read foo;
        read foo;
        read foo foo start foo
    } < /proc/timer_list;
    mapfile values < <(
        for((;n--;)){ "$@" &>/dev/null;}
        read -a endstat < /proc/self/stat
        {
            read foo
            read foo
            read foo foo end foo
        } </proc/timer_list
        printf -v times "%s/100/$e;" ${endstat[@]:13:4}
        bc -l <<<"$[end-start]/10^9/$e;$times"
    )
    printf -v fmt "%-7s: %%.5f\\n" real utime stime cutime cstime
    printf "$fmt" ${values[@]}
}

This is based on /proc:

man 5 proc | grep [su]time\\\|timer.list | sed  's/^/>   /'
            (14) utime  %lu
            (15) stime  %lu
            (16) cutime  %ld
            (17) cstime  %ld
     /proc/timer_list (since Linux 2.6.21)

Then now:

avgByProc 1000 sleep .001
real   : 0.00242
utime  : 0.00015
stime  : 0.00021
cutime : 0.00082
cstime : 0.00020

Where utime and stime represent user time and system time for bash himself and cutime and cstime represent child user time and child system time wich are the most interesting.

Nota: In this case (sleep) command won't use a lot of ressources.

avgByProc 1000 bc -ql /tmp/test-pi.bc
real   : 0.00175
utime  : 0.00015
stime  : 0.00025
cutime : 0.00108
cstime : 0.00032

This become more clear... Of course, as accessing timer_list and self/stat successively but not atomicaly, differences between real (nanosecs based) and c?[su]time (based in ticks ie: 1/100th sec) may appear!

F. Hauri - Give Up GitHub
  • 64,122
  • 17
  • 116
  • 137
  • 1
    Amazing insight! I like your solution very much. I agree this is a better approach. I wish `time -p` had given more precise numbers. – codeforester Mar 01 '19 at 18:47
  • @codeforester Did you test my alternative, based on `/proc/subShellPid/stat`, showing ***childs** system and user times*? – F. Hauri - Give Up GitHub Mar 03 '19 at 09:53
  • For non-English locales (where comma is used instead of the decimal point) `bc` will yield `(standard_in) 1: syntax error` must use something like: `shopt -s expand_aliases; alias bc="sed 's/,/./g' | bc | sed 's/\./,/g'"` and `-l` do not needed, but `scale=5` is recommended in the first example instead of `%.5f` as `bc` will break long lines. – dlazesz Sep 17 '22 at 18:58
  • Aaarg! `sed | bc | sed` , for each values! You'd better to force `LANG=C bc`!!! – F. Hauri - Give Up GitHub Sep 18 '22 at 05:21
1

From bashoneliners

  • adapted to transform (,) to (.) for i18n support
  • hardcoded to 10, adapt as needed
  • returns only the "real" value, the one you most likely want

Oneliner

for i in {1..10}; do time $@; done 2>&1 | grep ^real | sed s/,/./ | sed -e s/.*m// | awk '{sum += $1} END {print sum / NR}'

I made a "fuller" version

  • outputs the results of every execution so you know the right thing is executed
  • shows every run time, so you glance for outliers

But really, if you need advanced stuff just use hyperfine.

GREEN='\033[0;32m'
PURPLE='\033[0;35m'
RESET='\033[0m'

# example: perf sleep 0.001
# https://serverfault.com/questions/175376/redirect-output-of-time-command-in-unix-into-a-variable-in-bash
perfFull() {
    TIMEFORMAT=%R                       # `time` outputs only a number, not 3 lines
    export LC_NUMERIC="en_US.UTF-8"     # `time` outputs `0.100` instead of local format, like `0,100`

    times=10

    echo -e -n "\nWARMING UP ${PURPLE}$@${RESET}"
    $@ # execute passed parameters

    echo -e -n "RUNNING ${PURPLE}$times times${RESET}"

    exec 3>&1 4>&2                                   # redirects subshell streams
    durations=()
    for _ in `seq $times`; {
        durations+=(`{ time $@ 1>&3 2>&4; } 2>&1`)   # passes stdout through so only `time` is caputured
    }
    exec 3>&- 4>&-                                   # reset subshell streams

    printf '%s\n' "${durations[@]}"

    total=0
    for duration in "${durations[@]}"; {
        total=$(bc <<< "scale=3;$total + $duration")
    }

    average=($(bc <<< "scale=3;$total/$times"))
    echo -e "${GREEN}$average average${RESET}"
}
icetbr
  • 153
  • 1
  • 2
  • 8
0

It's probably easier to record the start and end time of the execution and divide the difference by the number of executions.

#!/bin/bash
times=10
start=$(date +%s)
for ((i=0; i < times; i++)) ; do
    run_your_executable_here
done
end=$(date +%s)
bc -l <<< "($end - $start) / $times"

I used bc to calculate the average, as bash doesn't support floating point arithmetics.

To get more precision, you can switch to nanoseconds:

start=$(date +%s.%N)

and similarly for $end.

choroba
  • 231,213
  • 25
  • 204
  • 289