1

I have running in background (SSH) tail -f access_log | grep 'POST /index.php' > test &

I ran this command inside /var/log/httpd where the access_log and error_log is stored.

I keep doing ls -l to see if test file changed size but it's created.. and always 0 bytes.

Am I even using the proper command tail? to do what I want?

I want to be able to filter out useless access_log information only store the POST /index.php's in a different file.

I know access_log keeps modifying itself even while I am iterating it, but if it's possible to start from top to bottom (head? i guess) that would be better.

Adrian Cornish
  • 23,227
  • 13
  • 61
  • 77
SSpoke
  • 5,656
  • 10
  • 72
  • 124

3 Answers3

1

tail -f will have continuous output, so > test may never create a file.

The suggestion in Ahmed Masud's answer is probably superior to this so try that first, but if that doesn't work out for you, you can just run this once a minute or once an hour or whatever if you don't need to worry about being super-efficient about it:

grep 'POST /index.php' access_log > test

That will totally re-create the file test every time you run it rather than build it incrementally. But if you just want a crude tool, that will get it done.

If you do it this way and your log file gets rotated, you'll no longer have what was in the previous log file in test after this runs on the new file. So do be aware of that! On the other hand, using tail -f, if the file rotates, you will stop getting data until you restart the process (and it to will blow away your old data too unless you use >> rather than >).

Trott
  • 66,479
  • 23
  • 173
  • 212
  • I understand, I just picked that up from another blog post somewhere. Any idea how I can do it a better way? – SSpoke Dec 01 '11 at 04:11
  • If Ahmed Masud's way works for you, that's probably the way to go. If not, I've added a blunt hammer approach to the answer. – Trott Dec 01 '11 at 04:17
  • Wow that `grep 'POST /index.php' access_log > test` ran in no one flat . Thanks – SSpoke Dec 01 '11 at 04:19
0

Try to do this in a shell script,

tail -f access_log | while read r ; do 
      echo $r | grep 'POST[[:space:]]*/index.php' >> out
done

this should get rid of that pesky flushing problem.

Ahmed Masud
  • 21,655
  • 3
  • 33
  • 58
  • Thanks, seems a bit long.. Trott's solution ran in a few seconds on 800 MB access_log – SSpoke Dec 01 '11 at 04:21
  • Ah you just wanted to grep it... i thought you wanted to continuously grep it as the log grew in the future so i gave this solution. – Ahmed Masud Dec 01 '11 at 04:34
  • 1
    Sorry I didn't know that's what it was for thats nice too.. but I was really trying to just extract all POST IP's and clear access_log – SSpoke Dec 01 '11 at 04:41
  • 1
    @SSpoke you may want to look at logrotate (See tutorial at: http://www.thegeekstuff.com/2010/07/logrotate-examples/). It's a very nice software for that. The one thing you have to be careful about is how you 'clear out' access_log; if you do it incorrectly you may have to restart your server or kill -HUP it; – Ahmed Masud Dec 01 '11 at 04:44
  • I just do `echo > access_log` every week, I don't know about what problems it causes but it works. – SSpoke Dec 01 '11 at 05:56
0

Other idea: "tail -F -n +1" will track log file rotation and read from the top of the file. Also, are you sure your log actually has the POST lines in it? Just asking... Finally, are file permissions appropriate?

dschultz
  • 485
  • 3
  • 10
  • yup I definitely got a bunch of POST requests, I need to check IP of each one of them to find the criminals so I can take action against them. – SSpoke Dec 01 '11 at 04:40