Filtering nmap outputs using grep, awk, or sed

Question

I using NMAP, I ran a scan on a large network to see open ports. The output file is 2MB, but I want to filter out all of the IP addresses with ALL closed ports.

 Nmap scan report for 10.x.x.x
 Host is up (0.048s latency).
 Not shown: 998 closed ports
 PORT   STATE SERVICE
 22/tcp open  ssh
 23/tcp open  telnet

 Nmap scan report for 10.x.x.x
 Host is up (0.046s latency).
 All 1000 scanned ports on 10.x.x.x are closed

 Nmap scan report for 10.x.x.x
 Host is up (0.045s latency).
 All 1000 scanned ports on 10.x.x.x are closed

Should output to only output to:

 Nmap scan report for 10.x.x.x
 Host is up (0.048s latency).
 Not shown: 998 closed ports
 PORT   STATE SERVICE
 22/tcp open  ssh
 23/tcp open  telnet

EDIT

The results are like

 Nmap scan report for 10.x.x.x
 Host is up (0.048s latency).
 Not shown: 998 closed ports
 PORT   STATE SERVICE
 22/tcp open  ssh
 23/tcp open  telnet
 Nmap scan report for 10.x.x.x
 Host is up (0.046s latency).
 All 1000 scanned ports on 10.x.x.x are closed
 Nmap scan report for 10.x.x.x
 Host is up (0.045s latency).
 All 1000 scanned ports on 10.x.x.x are closed

There are newlines that didn't copy over correctly

EDIT Thanks everyone. I see awk is pretty awesome and easy to do.

I only know simple grep. Like "cat nmap,out | grep open". But it only lists the ports that are open and not the other details such as ip address — user3037023, Nov 26 '13 at 15:28
Please provide the nmap command line you're using, so that we can replicate your output to test a solution. — ghoti, Nov 26 '13 at 15:36
Okay. What have you tried, beyond the simple grep you mentioned in the earlier comment? — ghoti, Nov 26 '13 at 15:49
that's the only thing. I do not know how to make grep grab other lines that are needed such as the host IP address for host with open ports while filtering out the IP addresses and information for hosts with ALL closed ports. — user3037023, Nov 26 '13 at 15:57

ghoti · Answer 1 · 2013-11-26T16:14:17.623

As I see it, you're trying to apply some intelligent filtering to your nmap output, not just a simple "grep".

Since your nmap command (per your comments rather than your question) is pointing at a subnet rather than an individual host, you need to interpret each section of output individually. But this kind of interpretation is too complex for a regular expression. (Might be possible with PREG, but it would be extremely difficult to compose and virtually impossible to read.) A tool like awk is a much better choice for this task.

For example:

nmap 10.10.0.0/16 | awk '
  /^Nmap scan report for/ {
    if (open) {
      print output;
    }
    output="";
    open=0;
  }

  {
    output=output $0 "\n";
  }

  $2 == "open" {
    open=1;
  }

  END {
    if (open) {
      print output;
    }
  }
'

Awk is easy enough to read, but you should know that it operates by matching each line of input against expressions that look like condition { action }. If a condition evaluates to true, the action is executed. So the first one has a condition that is a regular expression to find the beginning of a host section, and the actions are encased in curly braces. The second condition is missing, and so is assumed to be "true" for all lines. The last condition matches after all lines of input have been processed, and is necessary in case the last host scanned includes open ports.

This kind of thing could be expressed much more densely, but I've written it long so that you can see how the logic works more easily. Tighter code will come with practice.

Note that you can put the awk script into a separate file, which you reference using awk's -f option. Read the man page for details.

You could also put this whole thing into its own shell script if you don't want to keep the awk portion in its own file. You should be able to find examples of what that looks like easily enough -- at any rate, it's out of scope for this question.

If the question is "complex text processing" the answer is almost always `awk` (unless it's `perl`...). Good answer. — Floris, Nov 26 '13 at 17:11

score 1 · Answer 2 · answered Nov 26 '13 at 15:57

1

You could try

awk -f filter.awk input.txt

where input.txt is the ouput of nmap and filter.awk is:

/^Nmap/ {
    if (block)
        if (!match(block,/are closed/))
            print block
    block=$0
    next
}

{
    if (block)
        block=block RS $0
    else
        block=$0
}

answered Nov 26 '13 at 15:57

Håkon Hægland

39,012
21
81
174

AWESOME! Thanks. It worked although I modified it a tad bit to get rid of some other stuff. >if (!match(block,/are closed/) && !match(block, /are filtered/)) – user3037023 Nov 26 '13 at 16:12
@user3037023 Great :) – Håkon Hægland Nov 26 '13 at 16:17

score 1 · Answer 3 · answered Nov 26 '13 at 21:22

While it appears you have your answer, I'll tackle this from the Nmap side. Nmap has lots of different output options, so careful planning beforehand can make this sort of problem simple to deal with.

First, if you only want the hosts with open ports, you can run your scan with --open, which will hide all closed and filtered ports and hosts with no open ports. This means your output would be exactly what you want without filtering. The downside is that you lose some information; sometimes it's important to know what hosts are present, even if they have no services.

Next, Nmap has several output formats. The normal output (-oN) is mostly what you see on the screen. It's nice to scroll through, but as you can see, it's terrible for text processing or filtering with grep, sed, awk, etc. I find the most useful output option to be -oA, which saves 3 different files: a normal .nmap file, a "greppable" .gnmap file, and the mother of all formats, the .xml file.

Greppable format is nice for quick queries like, "What IPs have port 22 open?" It has one line per host, with strict delimiters. The downside is that it is deprecated: any new features of Nmap will not get integrated into the greppable format. Here's a list of some things missing from -oG: NSE script output, traceroute info, multiple hostnames, port state reasons, ttls. Your question could be answered using greppable format as follows:

grep '[0-9]/open' myscan.gnmap

XML output is the most complete format, including information that doesn't even fit into the normal output. It is also, by virtue of being XML, capable of being processed on the command line. Here's an example query using xmlstarlet (though there are lots of other tools that could do the job):

xmlstarlet sel -t -m "//host[ports/port/state/@state='open']" -v "address[@addrtype='ipv4']/@addr" -n myscan.xml

Filtering nmap outputs using grep, awk, or sed

3 Answers3