How to get last X records of the modsec_audit.log file?

Question

I need to get the last X records of the modsec_audit.log file in bash and write it out to a new file. I know this is possible with tail and sed, but this possibly cuts out at a certain line and causes half of a message to be left in the beginning.
Is there a cleaner way to do this? I just need to get the X last events/items from the log file.

For those not familiar with modsecurity log files, here is an example of the modsec_audit.log file:

--38321e0f-E--
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>503 Service Unavailable</title>
</head><body>
<h1>Service Unavailable</h1>
<p>The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.</p>
<hr>
<address>Apache/2.4.7 (Ubuntu) Server at 10.211.55.6 Port 80</address>
</body></html>

--38321e0f-H--
Message: Access denied with code 503 (phase 2). Pattern match "block_hack" at ARGS:block_hack. [file "/usr/share/modsecurity-crs/activated_rules/block_hack.conf"] [line "2"] [id "11212"]
Action: Intercepted (phase 2)
Apache-Handler: application/x-httpd-php
Stopwatch: 1443442449908332 455 (- - -)
Stopwatch2: 1443442449908332 455; combined=23, p1=6, p2=15, p3=0, p4=0, p5=2, sr=0, sw=0, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.7.7 (http://www.modsecurity.org/).
Server: Apache/2.4.7 (Ubuntu)
Engine-Mode: "ENABLED"

--38321e0f-Z--

Why do you need this? The Apache error file will have the details of any error on one line which is often easier to deal with than trying to deal with the multi-line audit log. — Barry Pollard, Sep 29 '15 at 22:20
I am making a Modsecurity specific application. Thus I need to get access to the last X records of the modsecurity file. I know the apache has also usefull info, but I need to focus here on modsec_audit.log — , Sep 30 '15 at 07:53
Ok. Just thought I'd mention in case you didn't know that ModSecurity also logs to Apache error file in a more easily grep-able format. But it doesn't have as much info as audit log. — Barry Pollard, Sep 30 '15 at 07:56

karakfa · Accepted Answer · 2015-09-29T20:03:47.160

0

some hacking with awk and tac

start with generous tail, reverse, print top 3 records, reverse

tail -100 log2 | tac | awk -vRS= -vORS="\n\n" 'NR<4' | tac

will print the last 3 records from the original log.

Explanation:

Since the log file can be very large you may want to limit the operation on the last few lines. If your records are 10 lines long, tail a corresponding section that will include at least n*10 lines. As most other tools awk start from the beginning of the file, that's why we reverse the lines with tac print n records and reverse back to the original format.

RS= sets awk to paragraph mode (records separated one or more empty lines), similarly ORS="\n\n" for printing records separated by an empty line. NR<4 condition will be true for records 1,2,3 and will be printed (as the default action of awk). -v is the optional tag to define variables.

edited Sep 29 '15 at 20:03

answered Sep 29 '15 at 19:17

karakfa

66,216
7
41
56

1

Thanks this works! But it shows top 2 records in my case (which is not a problem, just a sidenote). – Sep 29 '15 at 19:40
1

could you please explain "tail -100", and "awk -vRS= -vORS="\n\n" 'NR<4' | tac" – Sep 29 '15 at 19:52
please see the update. If you have 2 records printed perhaps 100 lines are not enough. – karakfa Sep 29 '15 at 20:04
1

if I change 'NR<4' to 'NR<5' i got 12 records, if I use 'NR<6' i got arround 16 records. Any idea how to a linear scale? – Sep 30 '15 at 07:51
it's counting paragraphs separated by an empty line. Are you sure all your records are in the same format as you posted above? – karakfa Sep 30 '15 at 12:35
1

It does work after all. I increased the tail length and now it's working!! Big thanks and thanks for the explanation and patience. – Sep 30 '15 at 13:36

score 0 · Answer 2 · answered Sep 29 '15 at 21:16

Another option might be like this... grep for all the -- at the start of the lines and number them like this:

grep -n "^--" modsec_audit.log

Sample output

1:--38321e0f-E--
14:--38321e0f-H--
25:--38321e0f-Z--
26:--38321e0f-E--
39:--38321e0f-H--
50:--38321e0f-Z--
51:--38321e0f-E--  <--- 51 is the line number of the 3rd last one
64:--38321e0f-H--
75:--38321e0f-Z--

So now delete (i.e. suppress printing) lines 1-50 of the file:

sed '1,50d' modsec_audit.log

How to get last X records of the modsec_audit.log file?

2 Answers2