1

I am not a system administrator but I run a little website and today I have received a subpoena request from police, in order to comply I need to extract IP addresses from our server logs for needed time.

This is my problem:

1

I have log1.txt

/var/log/lighttpd/access.log.1.gz:84.20.132.141 180.175.44.143 - [28/Apr/2011:09:23:30 -0500] "POST /base/script.php HTTP/1.1" 200 158 "http://ref/,http://ref/" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/530.5 (KHTML, like Gecko) Chrome/2.0.172.28 Safari/530.5"
/var/log/lighttpd/access.log.1.gz:85.40.142.111 180.175.44.143 - [28/Apr/2011:09:23:30 -0500] "POST /base/script.php HTTP/1.1" 200 158 "http://ref/,http://ref/" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/530.5 (KHTML, like Gecko) Chrome/2.0.172.28 Safari/530.5"

The file contains 500+ such lines

This is the output I am expecting: 85.40.142.111, 84.20.132.141, and other ips.

2

And I have log2.txt that has exactly same format

I need to output IPs from that log as well, e.g. 85.40.142.111, 81.02.32.61,

3

I need to use output results from log1 and match them with log2 and if one IP repeats in both logs then output it. From the examples I have provided, that IP would be 85.40.142.111.

Could you assist?

Community
  • 1
Andrey
  • 11
  • 2

2 Answers2

2

I'd suggest using cut, not grep:

cut -d\ -f1 log1.txt | sort | uniq > ip1.txt
cut -d\ -f1 log2.txt | sort | uniq > ip2.txt
grep -f ip2.txt ip1.txt

If the IP you're after is the second in each line, rather than the first, replace '-f1' with '-f2'.

HTH.

caelyx
  • 699
  • 3
  • 7
  • Thanks, but it doesn't work: `cut: the delimiter must be a single character` – Andrey Apr 30 '11 at 08:59
  • There's two spaces after the backslash; the first is the delimeter. Perhaps -d' ' instead? – caelyx Apr 30 '11 at 09:00
  • Thank you, but the output is not what I need. `/var/log/lighttpd/access.log.1.gz:93.8.158.46`. How to make it to output `93.8.158.46` instead. – Andrey Apr 30 '11 at 09:02
  • Oh, I gave you the wrong file format. All lines also begin with `/var/log/lighttpd/access.log.*.gz:`, e.g. `/var/log/lighttpd/access.log.1.gz:` or `/var/log/lighttpd/access.log.2.gz:` How to modify the cut query to work with this? Thank you! Let me also edit the original post. – Andrey Apr 30 '11 at 09:05
  • Thank you Lain and Caelyx. I have accomplished my desired objective. – Andrey Apr 30 '11 at 09:28
2

If all you want is a list of ip addresses 

#!/bin/bash
cut -d ' ' -f1 log1.txt | cut -d ':' -f2 | sort | uniq >log1.out
cut -d ' ' -f1 log2.txt | cut -d ':' -f2 | sort | uniq >log2.out
while read IP
do
    sed -n /$IP/p log2.out
done <log1.out
user9517
  • 115,471
  • 20
  • 215
  • 297