How can I grep for two values from the same input being piped to it?

Question

CentOS 5.x

I'm trying to build a shell script that searches data provided via stdin. Here's an example of the input stream:

Date: 1/1/11 
Time: 12:00 AM 
Foo: 12345 
Foo1: dskjflsdkjflksdjlfds 
Foo2: 123456789 
Foo3: kdsjflskdjflkjsdlkfjsdlkjflksdjflkjsdklfjlksdjflk

This information doesn't exist in a file, it would only be sent to the script in real-time as stdout from another app.

I want the sript to look at the data and parse out the values for Foo: and Foo2: storing those as variables for use later in the script.

My revised script attempt is this:

#!/bin/bash
while read data; do
        SearchCriteria1=$(echo "$data" | grep "Foo: " | cut -c 5-)
        SearchCriteria2=$(echo "$data" | grep "Foo2: " | cut -c 6-)
        echo $SearchCriteria1 >> test.1
        echo $SearchCriteria2 >> test.2
done

The finished script won't actually use test.1 or test 2 files. I just have them listed here for easier examples.

In this example, I would expect test.1 to have:

In this example, I would expect test.2 to have:

123456789

When I test this though, both test.1 and test.2 are blank and I know the data has the valid info in it.

I'm missing something obvious. Can someone please clarify?

Please show us an example input and expected output? In your above example, don't you `grep` from the `$data`? — quanta, Aug 13 '12 at 03:42
You're grepping for "Foo" followed by a space but your input file has "Foo" followed by a colon. — Alan Curry, Aug 13 '12 at 05:05
@AlanCurry User error on my part when writing the serverfault question. I updated the question and they correctly match on my test system (sorry for the typos - I've been banging my head against this for awhile and am getting tired). — Mike B, Aug 13 '12 at 05:08
now when I run what you have posted it works for me. I copied the first section to `infile`, the second section to `scriptfile`, ran `chmod +x scriptfile;./scriptfile> test.1` — Alan Curry, Aug 13 '12 at 05:11
@AlanCurry Strange the example is working for me too. There must be a typo in the original. I'll look again. — Mike B, Aug 13 '12 at 05:21
The script looked identical to me but I re-wrote it and it worked so there must have been a typo somewhere. Anyway, it's working now. Thank you VERY VERY much! — Mike B, Aug 13 '12 at 05:35

score 4 · Accepted Answer · edited May 23 '17 at 12:41

Think about where your greps are getting their input from. You've given them no input filename, so they're reading from standard input. What is their standard input? There is no redirection for it, and there is no pipe going into the greps, so they are running with the inherited standard input of the loop.

The read is executing only once. It reads the first line into $data, which you never use anywhere (that should be a clue that something is wrong). Then the first grep runs, it consumes all of the input, and the second grep is attached to the same input so it gets immediate EOF.

This is probably closer to what you wanted:

SearchCriteria1=$(echo "$data" | grep "Foo " | cut -c 10-)
SearchCriteria2=$(echo "$data" | grep "Foo2 " | cut -c 13-)

The semicolons on the end were useless so I took them off.

Also you probably want to append to test.1 and test.2 otherwise each iteration through the loop overwrites what the previous one wrote.

Looking at the big picture, it appears that you want a "multi output grep". Your read loop with individual greps for each line is one way to implement it, but not very efficient. Some other ways were discussed recently at https://stackoverflow.com/questions/11676350/grepping-a-20g-file-in-bash/11676853

Thanks for the feedback. That all makes sense but it's still not working. I'll revise my question to be more specific. — Mike B, Aug 13 '12 at 04:37

score 1 · Answer 2 · answered Aug 13 '12 at 05:17

input stream producer |
egrep 'Foo:|Foo2:' |
while read foo data
do
        case $foo in
                Foo:) echo "do something with $foo -- $data" ;;
                Foo2:) echo "do something else with $foo -- $data" ;;
        esac
done

The egrep line is not strictly necessary but cuts out unnecessary input for the loop. Then read into two variables (foo and data) and use case to choose between them and act accordingly (you could at this stage assign data to another variable if you wish).

score 0 · Answer 3 · answered Aug 13 '12 at 04:58

0

How about this:

$ ./another.app | awk '/Foo: / { print $2 }' > test.1
$ ./another.app | awk '/Foo2: / { print $2 }' > test.2

answered Aug 13 '12 at 04:58

quanta

51,413
19
159
217

Piping it to test.1 and test.2 is just an example of what I would do with the variable. There's still more I want to do with the variable in the script. Sorry for the confusion... the script isn't finished yet and I'm trying to keep the example straight-forward. Thanks for your patience. – Mike B Aug 13 '12 at 05:01
`SearchCriteria1=$(./another.app | awk '/Foo: / { print $2 }')`? – quanta Aug 13 '12 at 05:10

How can I grep for two values from the same input being piped to it?

3 Answers3