double awk commands in same pipe?

Question

Hello I'm trying to do something like this but it wont work, can you guys tell me help me out?

/usr/bin/mysql -B -r -h ******** -u******* -p****** -D***** \
    -e'SELECT `username`, `uid`, `gid`, `homedir` FROM `some_table`)' | \
        awk '{print $1":x:"$2":"$3"::"$4":/usr/sbin/nologin"}' >> /tmp/file1 ;
        awk '{print $1":"$5":13753:0:99999:7:::"}'>>/tmp/file2

Changing ";" to "&&" doesn't work either. Both file1 and file2 is created, but only file1 contains data. So basically what I want to do is using the same variables for two commands

score 3 · Accepted Answer · answered Oct 18 '11 at 16:07

3

Try this:

/usr/bin/mysql -B -r -h ******** -u******* -p****** -D***** \
    -e'SELECT `username`, `uid`, `gid`, `homedir` FROM `some_table`)' | \
        awk '{ print $1":x:"$2":"$3"::"$4":/usr/sbin/nologin" >> "/tmp/file1"; 
               print $1":"$5":13753:0:99999:7:::" >> "/tmp/file2" }'

answered Oct 18 '11 at 16:07

quanta

51,413
19
159
217

Good call, get awk to do both operations in one call. I was thinking of the more generic case :) – MikeyB Oct 18 '11 at 16:14

score 2 · Answer 2 · edited Apr 13 '17 at 12:14

When you want to run two different commands on the same incoming pipe data, you need to use an intermediate file:

TMPFILE="$(mktemp)"
/usr/bin/mysql -B -r -h ******** -u******* -p****** -D***** \
               -e'SELECT `username`, `uid`, `gid`, `homedir` FROM `some_table`)' >"$TMPFILE"
awk '{print $1":x:"$2":"$3"::"$4":/usr/sbin/nologin"}' <"$TMPFILE" >>/tmp/file1
awk '{print $1":"$5":13753:0:99999:7:::"}' <"$TMPFILE" >>/tmp/file2
rm -f "$TMPFILE"

But as quanta writes, you can ask a single awk instance to do both operations.

score 1 · Answer 3 · answered Oct 18 '11 at 15:34

You've made quite a mess, starting with the formatting of your question (see what I've done above which should make things below more clear).

First problem:
If you have a semicolon (;) or a "continue-on-success (&&) you are running two separate sets of commands.
Completely separate. It's like you ran one, hit enter and then ran the other.
This means your second awk is looking on stdin for input, and not finding anything it's giving you no output (which is why file2 is empty)

Second Problem:
Separate instances of awk can't share variables. Either you want the second awk to chew on the output of the MySQL query, or you want it to chew on the output of the first awk.
Since you've got 5 variables in your second awk I'm assuming the latter.

Solution
Run the first half of the command list (everything up to the semicolon).
Now run the second awk, and specify an input file (awk '{print $1":"$5":13753:0:99999:7:::"}' /tmp/file1 >> /tmp/file2).

Thank you! Sorry for the messy question, was in a bit of hurry when I wrote it, Ive think about it next time! — Apaz, Oct 19 '11 at 08:26
No worries - That was more a "If you format your commands so they're pretty sometimes the answer jumps out" note than a "your question is ugly so I'm going to beat you with a dirty sock" one :-) — voretaq7, Oct 19 '11 at 15:24

score 0 · Answer 4 · answered Oct 18 '11 at 15:43

Your command as shown will never work, as processing is entirely sequential, and the output of mysql is already used up by the first awk statement.

I would simply move all that to a tiny bash script:

#!/bin/bash
mycmd="SELECT `username`, `uid`, `gid`, `homedir` FROM `some_table`"
while read -r username, uid, gid, homedir; do
printf "%s:x:%s:%s::%s:/usr/sbin/nologin" "$username" "$uid" "$gid" "$homedir" >> /tmp/file1
printf "%s:%s:13753:0:99999:7:::" "$username" "missing variable" >>/tmp/file2
done <(/usr/bin/mysql -B -r -h * -u* -p* -D* -e "$mycmd")

UNtested, visit the Wooledge Wiki for more bash magic than you knew existed :)

double awk commands in same pipe?

4 Answers4