How can I split parsed text by "blocks" of text?

Question

I am writing a bash script that runs an Nmap scan of the network. After this the scan needs to be examinned and the relevant bits need to be extracted.

I need to extract the IP, MAC and OS from the completed scan. The problem is that Nmap does not always get the OS from the scan and therefore does not put it in the results. I need to associate the IP, MAC and OS in the end result.

Here is an example of a test scan:

Nmap scan report for 192.168.0.1
Host is up (0.0029s latency).
Not shown: 990 closed ports
PORT      STATE SERVICE
PORT#    STATE    XXXXXXX
MAC Address: MA:CA:DR:ES:S0:03 (Unknown)
Device type: general purpose
Running: Linux 2.6.X|3.X
OS CPE: cpe:/o:linux:linux_kernel:2.6 cpe:/o:linux:linux_kernel:3
OS details: Linux 2.6.32 - 3.13
Network Distance: 1 hop

Nmap scan report for 192.168.0.102
Host is up (0.0044s latency).
Not shown: 999 closed ports
PORT     STATE    SERVICE
PORT#    STATE    XXXXXXX
MAC Address: MA:CA:DR:ES:S0:02 (Sony Mobile Communications AB)
Too many fingerprints match this host to give specific OS details
Network Distance: 1 hop

Nmap scan report for 192.168.0.104
Host is up (0.00024s latency).
Not shown: 995 filtered ports
PORT     STATE SERVICE
PORT#    STATE XXXXXX
MAC Address: MA:CA:DR:ES:S0:01 (Micro-star Intl)
Warning: OSScan results may be unreliable because we could not find at least 1 open and 1 closed port
Device type: general purpose
Running (JUST GUESSING): Microsoft Windows 2008 (91%)
OS CPE: cpe:/o:microsoft:windows_server_2008::sp1 cpe:/o:microsoft:windows_server_2008:r2
Aggressive OS guesses: Microsoft Windows Server 2008 SP1 or Windows Server 2008 R2 (91%)
No exact OS matches for host (test conditions non-ideal).
Network Distance: 1 hop

Also note how the last one in the example above could not find the OS, in this case the aggress guess is wanted

The end result needs to be a text file that has has something like the following:

192.168.0.1 - MA:CA:DR:ES:S0:03 - Linux 2.6.32 - 3.13
192.168.0.102 - MA:CA:DR:ES:S0:02 - Not found
192.168.0.104 - MA:CA:DR:ES:S0:01 - Microsoft Windows Server 2008 SP1 or Windows Server 2008 R2

I did some research but could not find anything that explains how I can associate the IP with the mac addresses and the os in the text blocks.

I have the following commands that work with a simple scan where the IP and Mac addresses are next to each other

  while read line; do
    Mac="$(grep -oE '[A-Z0-9]{2}:[A-Z0-9]{2}:[A-Z0-9]{2}:[A-Z0-9]{2}:[A-Z0-9]{2}:[A-Z0-9]{2}' <<< "$line")"
    ip="$(grep -oE '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' <<< "$line")"
    echo -e $ip'\t-\t '$Mac >>/path/to/results.txt
  done </path/to/testscan.txt

I am fairly new to bash scripting so apologies if I am missing something obvious.

The nmap command for anyone interested is:

nmap -O --osscan-guess 192.168.0.0/24 -oN /path/to/testscan.txt

Sorry for the wall of text, I figured the more information the better!

If you compare your grep solution with the AWK solution below, you understand that regex are no longer adapted for pattern matching over several lines. — Pierre François, Apr 29 '20 at 19:10

larsks · Accepted Answer · 2020-04-29T19:10:42.923

This would be pretty easy to parse with awk:

BEGIN {os_details="Not found"}

/^Nmap scan report/      {target=$5}
/^MAC Address/           {mac_address=$3}
/^OS details/            {os_details=substr($0, length("OS details: "))}
/^Aggressive OS guesses/ {
    os_details=substr($0, length("Aggressive OS guesses: "))
}

# This matches the blank lines between hosts
/^$/ {
    printf "%s - %s - %s\n", target, mac_address, os_details
    target=""
    mac_address=""
    os_details="Not found"
}

END {
    printf "%s - %s - %s\n", target, mac_address, os_details
}

Running this on your sample data gets you:

192.168.0.1 - MA:CA:DR:ES:S0:03 -  Linux 2.6.32 - 3.13
192.168.0.102 - MA:CA:DR:ES:S0:02 - Not found
192.168.0.104 - MA:CA:DR:ES:S0:01 -  Microsoft Windows Server 2008 SP1 or Windows Server 2008 R2 (91%)

I had to make one correct to what I believe was an error in your sample data...I removed the blank line before the MAC Address line here:

Nmap scan report for 192.168.0.104
Host is up (0.00024s latency).
Not shown: 995 filtered ports
PORT     STATE SERVICE
PORT#    STATE XXXXXX

MAC Address: MA:CA:DR:ES:S0:01 (Micro-star Intl)

The erratic blank line has been removed from the input. I totally agree with you that AWK is the best tool for this kind of problems. — Pierre François, Apr 29 '20 at 19:07
The blank line was a mistake. Thanks, I just tried it and it works like a charm. Was not aware that awk had BEGIN, END blocks. Will keep it in mind for the future! — SpookyLoops, Apr 30 '20 at 08:19

score 1 · Answer 2 · answered Apr 29 '20 at 20:59

Using option -oX of the nmap (output to the XML format) the parsing could be more accurate:

nmap -oX /path/to/testscan.xml ...
# or
nmap -oX - ... > /path/to/testscan.xml

Then you could to use, for example, xmllint to parse this XML with XPath:

file="/path/to/testscan.xml"

get_details() {
    local file addr mac os
    file="$1"
    addr=$2
    mac=$(xmllint --xpath "string(//address[../address[@addr='$addr']][@addrtype='mac']/@addr)" "$file")
    os=$(xmllint --xpath "string(//os[../address[@addr='$addr']]/osmatch/@name)" "$file")
    : ${mac:="No data"}
    : ${os:="No data"}
    printf "%s - %s - %s\n" "$addr" "$mac" "$os"
}   

for a in $(xmllint --xpath "//address[@addrtype='ipv4']/@addr" "$file" | grep -Po '\d+\.\d+\.\d+\.\d+'); do
    get_details "$file" $a
done

How can I split parsed text by "blocks" of text?

2 Answers2