0

I have three functions that digest an access.log file on my server.

hitsbyip() {
        cat $ACCESSLOG | awk '{ print $1 }' | uniq -c | sort -nk1 | uniq
}
hitsbyhour() {
        cat $ACCESSLOG | cut -d[ -f2 | cut -d] -f1 | awk -F: '{print $2":00"}' | sort -n | uniq -c
}
hitsbymin() {
        hr=$1
        grep "2015:${hr}" $ACCESSLOG | cut -d[ -f2 | cut -d] -f1 | awk -F: '{print $2":"$3}' | sort -nk1 -nk2 | uniq -c
}

They all work fine when used on their own. All three output 2 small colums of data.

Now I am looking to create another function called report which will simply use printf and its formatting possibilities to print 3 columns of data with header, each of them the result of my three first functions. Something like that:

report() {
       printf "%-30b %-30b %-30b\n" `hitsbyip` `hitsbyhour` `hitsbymin 10`
}

The thing is that the format is not what i want; it prints out the columns horizontaly instead of side by side.

Any help would be greatly appreciated.

Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
Jeanmichel Cote
  • 531
  • 1
  • 5
  • 19
  • 3
    Quotes, people, quotes! – Charles Duffy Jul 20 '15 at 14:40
  • Waitaminute. Does each function return only one line per invocation, or more? If it's more than one, you need to do more work (most easily, using external tools). – Charles Duffy Jul 20 '15 at 14:41
  • 1
    Besides your functions are doing work inefficiently. `cat`, `cut`, `uniq` etc are all unnecessary. – anubhava Jul 20 '15 at 14:43
  • Indeed -- `awk` could do the whole job itself. – Charles Duffy Jul 20 '15 at 14:44
  • Lots more than one line. I am querying the access.log file on a server so even though I use the `uniq -c` command, there may be hundreds of lines. So what kind of external tools? – Jeanmichel Cote Jul 20 '15 at 14:45
  • `paste`. See my answer as amended. Though, well, `awk` could do the whole thing, and I assume someone will add an awk-only answer eventually. – Charles Duffy Jul 20 '15 at 14:45
  • @anubhava Your comment didn't say much. Of course you don't have to help but if you could explain the whys and hows, that would help. Thank you anyway for your time. – Jeanmichel Cote Jul 20 '15 at 14:52
  • 2
    @JeanmichelCote, to explain anubhava's comment a bit, `awk` is a full-featured scripting language: It can do the work of `cut` and `uniq` (and, with GNU awk, even `sort`) itself. As for `cat`, it only has utility when reading more than one file; for reading a single file, one can simply use ` – Charles Duffy Jul 20 '15 at 14:53
  • (indeed, "cat" is short for "concatenate", that being its utility -- to concatenate multiple input sources into a single stream; if your input *already is* a single stream, it's generally unneeded). – Charles Duffy Jul 20 '15 at 14:58

1 Answers1

3

Once you use paste to combine the output of the three commands into a single stream, then you can operate line-by-line to format those outputs.

while IFS=$'\t' read -r by_ip by_hour by_min; do
  printf '%-30b %-30b %-30b\n' "$by_ip" "$by_hour" "$by_min"
done < <(paste <(hitsbyip) <(hitsbyhour) <(hitsbymin 10))

Elements to note:

  • <() syntax is process substitution; it generates a filename (typically of the form /dev/fd/## on platforms with such support) which will, when read, yield the output of the command given.
  • paste takes a series of filenames and puts the output of each alongside the others.
  • Setting IFS=$'\t' while reading ensures that we read content as tab-separated values (the format paste creates). See BashFAQ #1 for details on using read.
  • Putting quotes on the arguments to printf ensures that we pass each value assembled by read as a single value to printf, rather than letting them be subject to string-splitting and glob-expansion as unquoted values.
Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
  • This is excellent and does exactly what i was looking for. I didn't know about `paste` command. But could you explain a bit more about that `<(stuff)`? No need for spaces after and before the parens? – Jeanmichel Cote Jul 20 '15 at 14:54
  • Not only "no need for", but "must not use". I added a bullet point with a link talking about process substitution; please reload to get the current version of the answer and follow it. – Charles Duffy Jul 20 '15 at 14:56
  • ...to understand a little more about the *whys* of process substitution, see http://mywiki.wooledge.org/BashFAQ/024 – Charles Duffy Jul 20 '15 at 14:56
  • Thanks so much. Refactoring my functions now. – Jeanmichel Cote Jul 20 '15 at 15:00
  • 1
    By the way, it's always a good practice to run something through http://shellcheck.net/ before asking about it here -- that way you at least get some of the knee-jerk "code that does X is always wrong!" issues out of the way (such as how my first read was focusing on bad quoting). – Charles Duffy Jul 20 '15 at 15:02
  • Gee, shame on me. Didn't know about shellcheck.net! – Jeanmichel Cote Jul 20 '15 at 15:08