simplifying list of CIDRs in bash

Question

I am trying to create a list of CIDRs for Google's computers. When I issue in Linux whois -h whois.radb.net -- '-i origin AS15169' | grep ^route: | sed 's/.* //' . The list is excessively long and redundant. For example, at the end, I see:

216.239.58.0/23
216.239.58.0/24
216.239.59.0/24
216.239.60.0/23
216.239.60.0/24
216.239.61.0/24
216.239.62.0/23
216.239.62.0/24
216.239.63.0/24

These are all part of a subnet listed a few screens above, 216.239.32.0/19.

Is there an easy way in Bash to reduce the number of lines? Searching SO, I found some Python one-liners, but I'd like to run this on Bash, if possible.

Bash itself has fairly limited support for arithmetic, and Python is readily available as a tool from within Bash most places you'll find Bash these days. Perhaps if you can elaborate on why Python is unacceptable, it would also help guide us on whether other scripting languages (Awk, Perl, etc) would work for you. Also, have you looked for existing tools to solve this problem, rather than programming your own? — tripleee, Feb 08 '15 at 07:23
I want to run the script on an embedded ARM system. I could install Python, but if I can get away without it, I'd prefer it. I'd rather compile and install one or more small programs, like ipcalc, than Python. Bash is already required for other scripts. So the question is really: can this be done with something lighter than Python? — nvja, Feb 08 '15 at 08:29
A natural solution would be to build a tree structure, but this is cumbersome with the limited data types available in Bash and Awk. That is not to say it is impossible. Would an Awk solution be acceptable? — tripleee, Feb 08 '15 at 09:30
Yes, Awk is also installed. If it is too difficult, then I'll just use Python's `netaddr`. — nvja, Feb 08 '15 at 17:13

score 1 · Accepted Answer · answered Feb 11 '16 at 03:42

I was able to solve it with the help of ipcalc. Without ipcalc, it gets more tedious. Replacing the for with a while should speed it up. I am not awarding myself the points, given that my answer is a work in progress:

lowestipinrange()
{
  ipcalc "$1" | grep HostMin | sed 's/HostMin:  *//'| sed 's/   .*//'
}

highestipinrange()
{
  ipcalc "$1" | grep HostMax | sed 's/HostMax:  *//'| sed 's/   .*//'
}

ip2dec () {
# This is verbatim from https://stackoverflow.com/questions/10768160/ip-address-converter
    local a b c d ip=$@
    IFS=. read -r a b c d <<< "$ip"
    printf '%d\n' "$((a * 256 ** 3 + b * 256 ** 2 + c * 256 + d))"
}

as2ip()
{
  ALLSUBNETS=$(whois -h whois.radb.net -- '-i origin '$1 | grep ^route: | sed 's/.* //' | sort -k 2  -t '/')
  USEFULSUBNETS=()
  MINIPARRAY=()
  MAXIPARRAY=()
  COUNTUSEFULNETS=0
  for subnet in $ALLSUBNETS ; do
    CURRMINIP=`lowestipinrange $subnet`
    CURRMINIP=`ip2dec $CURRMINIP`
    CURRMAXIP=`highestipinrange $subnet`
    CURRMAXIP=`ip2dec $CURRMAXIP`
    SUBNETSEEMSUSEFUL=true
    for ((i = 0; i < $COUNTUSEFULNETS ; i++)); do
      if (( ${MINIPARRAY[$i]} <= $CURRMINIP )) && (( ${MAXIPARRAY[$i]} >= $CURRMAXIP )) ; then
        SUBNETSEEMSUSEFUL=false
      fi
    done
    if [ "$SUBNETSEEMSUSEFUL" = true ]; then
      let "COUNTUSEFULNETS += 1"
      MINIPARRAY+=($CURRMINIP)
      MAXIPARRAY+=($CURRMAXIP)
      USEFULSUBNETS+=("$subnet")
    fi
  done

  for ((i = 0; i < $COUNTUSEFULNETS ; i++)); do
    echo ${USEFULSUBNETS[$i]}
  done
}

score 0 · Answer 2 · answered Aug 13 '15 at 06:40

0

shopt -s extglob
IFS=\|$IFS
while IFS=./ read a b c d e
do  ip=$(printf %08i `bc <<<"obase=2;$a;$b;$c;$d"`)
    network=${ip::$e}\*
    [[ "$network" == !($netlist) ]] &&
        nets[${#nets[*]}]="$network" netlist="${nets[*]}" && echo $a.$b.$c.$d/$e
done

This script outputs the network addresses that are not part of a previously seen network. It does this by converting the input CIDR notation to a string $network of binary digits (0 and 1) comprising only the network prefix, and using extended pattern matching to search for the network in the accumulated list $netlist of (super-)nets, outputting the network and adding it to the list if not matched by a supernet pattern.

answered Aug 13 '15 at 06:40

Armali

18,255
14
57
171

I could not make it work. I pasted it on a new text file, set +x, tried piping a list of CIDRs. It won't work. – nvja Feb 11 '16 at 03:29
_It won't work_ is not a useful description. What was the outcome? Did you pipe a different list than shown in your question? If so, you should provide that list in order to make the problem reproducible. – Armali Feb 16 '16 at 09:29

simplifying list of CIDRs in bash

2 Answers2