repeated curl + grep on macosx occasionally giving bad results?

Question

Running macosx 10.6.2, I am seeing some extremely weird behavior with a script which repeatedly calls curl -o (file) and then greps for a certain string within it. Occasionally grep returns 1 (not found) when I'd expected 0 (found). Here's the script...

# do this 1000 times

for ii in `cat count.txt`; do

        rm -f a.txt
        rm -f e.txt
        curl  --fail --stderr e.txt -j -o a.txt  -s  $MYURL

        if [ -e a.txt ] ; then

                #  Occasionally a.txt doesn't finish writing on time
                grep  "login-url" a.txt >/dev/null
                LASTERR=$?
                echo $LASTERR is lasterr grep 1
                if [ "$LASTERR" -ne "0" ] ; then
                        cp a.txt anomaly.txt
                        sleep 1
                        echo "Sleeping..."
                fi

                grep -q "login-url" a.txt >/dev/null
                LASTERR=$?
                echo $LASTERR is lasterr grep 2
                if [ "$LASTERR" -ne "0" ] ; then
                    echo "Dying..."
                    exit 1
                fi

                # This is what I actually want to do
                grep "login-url" a.txt  >> out.txt
        fi

done

What I see is something like this:

0 is lasterr 1
0 is lasterr 2
...
0 is lasterr 1
0 is lasterr 2
0 is lasterr 1
1 is lasterr 2

In other words, a.txt is changing (as far as grep can tell) between the two greps!

Has anyone else seen the like?

I note if I put in a "sleep 1" after the curl call, the issue goes away. So is there a problem with re-using the same file name over and over, or is curl returning before it is done writing, or...?

This is not a crisis issue because of the "sleep 1" workaround, but I am nervous because I don't understand the behavior.

This isn't your problem, but you should do `for ii in {1..1000}` or `for ((ii = 1; ii <= 1000; ii++))` instead of using a file like that. Or if this isn't Bash, do `for ii in $(seq 1000)` or `for ii in $(jot 1000)` — Dennis Williamson, Feb 09 '10 at 19:19

score 1 · Answer 1 · answered Feb 09 '10 at 19:27

1

You might try making the two grep commands match. One has -q and the other doesn't. That's probably not the problem, but it's worth eliminating differences to narrow things down.

Your output doesn't match what the script is doing, so it makes me wonder if you're running and looking at two different versions. I've done that a number of times myself, causing a great deal of confusion until I realize what's going on.

answered Feb 09 '10 at 19:27

Dennis Williamson

62,149
16
116
151

From the man page of grep: Normally, exit status is 0 if selected lines are found and 1 otherwise. But the exit status is 2 if an error occurred, unless the -q or --quiet or --silent option is used and a selected line is found. In this case you are getting different results because of the -q – tajh Feb 10 '10 at 00:02
Hi, thanks for noticing that; I was a little clumsy in cleaning out the company-specific stuff from the script. Naturally, that's where the error actually showed up. I had had something like this: SERVER=(some ip address) curl -o a.txt -s http://${SERVER}/.../a.jsp?val1=a&val2=a@b&val3=c When I changed it to URL="http://(some ip addr)/.../a.jsp?val1=a&val2=a@b&val3=c" curl -o a.txt -s $URL It worked fine. I think I must have had shell-active characters in my URL... – akaioi Feb 10 '10 at 15:58

score 0 · Answer 2 · answered Feb 09 '10 at 23:44

0

have you tried the debug output of bash?

bash -x /path/to/script

It will output each step verbatim.

answered Feb 09 '10 at 23:44

CarpeNoctem

2,437
4
23
32

repeated curl + grep on macosx occasionally giving bad results?

2 Answers2