How do I fill in unused ip addresses in a python sublist?

Question

I have created a python script to try and make my life as a system administrator a lot easier. The point of this script is to convert a Microsoft DHCP server dump file into a sorted CSV file.

I will include the code here and am thankfull for all kinds of improvements.

My problem

My script creates a list of lists (one for each dhcp reservation). For example:

[
  # [DHCP SERVER, IP ADDRESS, MAC ADDRESS, HOSTNAME, DESCRIPTION]
  [server1,172.16.0.120,31872fcefa33,wks120.domain.net,Description of client]
  [server1,172.16.0.125,4791ca3d7279,wks125.domain.net,Description of client]
  [server1,172.16.0.132,6035a71c930c,wks132.domain.net,Description of client]
  ...
]

The unused ip addresses are not listed. But I would like my script to automatically add sublists for all the unused IP addresses in between and give them a comment saying "Unregistered" or something.

I have no clue how to even begin searching google on how to complete this task, so any help would be appreciated :)

The script

#!/usr/bin/python
import sys, shlex
from operator import itemgetter

# Function: get_dhcp_reservations
#
# Extracts a list of ip reservations from a Microsoft DHCP server dump file
# then it stores the processed reservations them in a nested list

def get_dhcp_reservations(dmpFile):

  # Setup empty records list
  records = []

  # Open dump file for reading
  dmpFile = open(dmpFile,"r")

  # Iterate dump file line by line
  for line in dmpFile:

    # Only user lines with the word "reservedip" in it
    if "reservedip" in line:

      # Split the line into fields excluding quoted substrings
      field = shlex.split(line)

      # Create a list of only the required fields
      result = [field[2][1:9], field[7], field[8], field[9], field[10]]

      # Append each new record as a nested list
      records.append(result)

  # Return the rendered data
  return records


# Function: sort_reservations_by_ip
#
# Sorts all records by the IPv4 address field

def sort_reservations_by_ip(records):

  # Temporarily convert dotted IPv4 address to tuples for sorting
  for record in records:
    record[1] = ip2tuple(record[1])

  # Sort sublists by IP address
  records.sort(key=itemgetter(1)) 

  # Convert tuples back to dotted IPv4 addresses
  for record in records:
    record[1] = tuple2ip(record[1])

  return records


# Function: ip2tuple
#
# Split ip address into a tuple of 4 integers (for sorting)

def ip2tuple(address):
  return tuple(int(part) for part in address.split('.'))


# Function: tuple2ip
#
# Converts the tuple of 4 integers back to an dotted IPv4 address

def tuple2ip(address):

  result = ""

  for octet in address:
    result += str(octet)+"."

  return result[0:-1]


# Get DHCP reservations
records = get_dhcp_reservations(sys.argv[1])

# Sort reservations by IP address
records = sort_reservations_by_ip(records)

# Print column headings
print "DHCP Server,Reserved IP,MAC Address,Hostname,Description"

# Print in specified format records
for record in records:
  print record[0]+","+record[1]+",\""+record[2]+"\","+record[3]+","+record[4]

NOTE: I have also tried IPv4 sorting by using the python socket.inet_ntoa as suggested in other topics on this site but was not successful to get it working.

Dump File Example

Per request, here is some of the dump file

[Ommited content]

# ======================================================================
#  Start Add ReservedIp to the Scope : 172.16.0.0, Server : server1.domain.net            
# ======================================================================


    Dhcp Server \\server1.domain.net Scope 172.16.0.0 Add reservedip 172.16.0.76 0800278882ae "wks126devlin.domain.net" "Viana (VM)" "BOTH"
    Dhcp Server \\server1.domain.net Scope 172.16.0.0 Add reservedip 172.16.0.118 001e37322202 "WKS18.domain.net" "Kristof (linux)" "BOTH"
    Dhcp Server \\server1.domain.net Scope 172.16.0.0 Add reservedip 172.16.0.132 000d607205a5 "WKS32.domain.net" "Lab PC" "BOTH"
    Dhcp Server \\server1.domain.net Scope 172.16.0.0 Add reservedip 172.16.0.156 338925b532ca "wks56.domain.net" "Test PC" "BOTH"
    Dhcp Server \\server1.domain.net Scope 172.16.0.0 Add reservedip 172.16.0.155 001422a7d474 "WKS55.domain.net" "Liesbeth" "BOTH"
    Dhcp Server \\server1.domain.net Scope 172.16.0.0 Add reservedip 172.16.0.15 0800266cfe31 "xpsystst.domain.net" "Pascal (VM)" "BOTH"

[Ommited content]

Answer 1

I started by creating a list of all the empty reservations, then overwriting it with the non-empty list you started with:

#!/usr/bin/env python

reservations = [
    # [DHCP SERVER, IP ADDRESS, MAC ADDRESS, HOSTNAME, DESCRIPTION]
    ['server1','172.16.0.120','31872fcefa33','wks120.domain.net','Description of client'],
    ['server1','172.16.0.125','4791ca3d7279','wks125.domain.net','Description of client'],
    ['server1','172.16.0.132','6035a71c930c','wks132.domain.net','Description of client'],
]

def reservationlist(reservations, serverpattern, addresspattern, hostpattern,
        start, end):
    result = []
    for i in range(start, end + 1):
        result.append([
            serverpattern % i,
            addresspattern % i,
            '[no mac]',
            hostpattern % i,
            'Unregistered'])

    for reservation in reservations:
        index = int(reservation[1].split('.')[3]) - start
        result[index] = reservation

    return result

print reservationlist(
    reservations,
    'server%d',
    '172.16.0.%d',
    'wks%d.domain.net',
    120,
    132)

The final result looks like:

[['server1', '172.16.0.120', '31872fcefa33', 'wks120.domain.net', 'Description of client'],
['server121', '172.16.0.121', '[no mac]', 'wks121.domain.net', 'Unregistered'],
['server122', '172.16.0.122', '[no mac]', 'wks122.domain.net', 'Unregistered'],
['server123', '172.16.0.123', '[no mac]', 'wks123.domain.net', 'Unregistered'],
['server124', '172.16.0.124', '[no mac]', 'wks124.domain.net', 'Unregistered'],
['server1', '172.16.0.125', '4791ca3d7279', 'wks125.domain.net', 'Description of client'],
['server126', '172.16.0.126', '[no mac]', 'wks126.domain.net', 'Unregistered'],
['server127', '172.16.0.127', '[no mac]', 'wks127.domain.net', 'Unregistered'],
['server128', '172.16.0.128', '[no mac]', 'wks128.domain.net', 'Unregistered'],
['server129', '172.16.0.129', '[no mac]', 'wks129.domain.net', 'Unregistered'],
['server130', '172.16.0.130', '[no mac]', 'wks130.domain.net', 'Unregistered'],
['server131', '172.16.0.131', '[no mac]', 'wks131.domain.net', 'Unregistered'],
['server1', '172.16.0.132', '6035a71c930c', 'wks132.domain.net', 'Description of client']]

Bah! I couldn't help myself. This version accepts IP addresses for the start and end values:

#!/usr/bin/env python

reservations = [
    # [DHCP SERVER, IP ADDRESS, MAC ADDRESS, HOSTNAME, DESCRIPTION]
    ['server1','172.16.0.120','31872fcefa33','wks120.domain.net','Description of client'],
    ['server1','172.16.0.125','4791ca3d7279','wks125.domain.net','Description of client'],
    ['server1','172.16.0.132','6035a71c930c','wks132.domain.net','Description of client'],
]

def addr_to_int(address):
    """Convert an IP address to a 32-bit int"""
    a, b, c, d = map(int, address.split('.'))
    return a * 256 * 256 * 256 + b * 256 * 256 + c * 256 + d

def int_to_addr(value):
    """Convert a 32-bit int into a tuple of its IPv4 byte values"""
    return value >> 24, value >> 16 & 255, value >> 8 & 255, value & 255

def reservationlist(reservations, serverpattern, addresspattern, hostpattern,
        start, end):

    reservationdict = dict((addr_to_int(item[1]), item)
            for item in reservations)
    startint = addr_to_int(start)
    endint = addr_to_int(end)
    for i in range(startint, endint + 1):
        try:
            item = reservationdict[i]
        except KeyError:
            addressbytes = int_to_addr(i)
            item = [
                serverpattern.format(*addressbytes),
                addresspattern.format(*addressbytes),
                '[no mac]',
                hostpattern.format(*addressbytes),
                'Unregistered']
        yield item

for entry in reservationlist(
    reservations,
    'server{3}',
    '172.16.{2}.{3}',
    'wks{3}.domain.net',
    '172.16.0.120',
    '172.16.1.132'):
    print entry

This version uses the yield keyword to turn reservationlist() into a generator. Instead of holding all the values in RAM at once, it just emits a single value at a time until the loop is finished. For each pass through the loop, it tries to fetch the actual value from your list of reservations (using a dict for quick access). If it can't, it uses the string.format method to fill in the string templates with the IPv4 address bytes.

A quick note on address manipulation

The int_to_addr function takes a 32-bit IP address like:

AAAAAAAABBBBBBBBCCCCCCCCDDDDDDDD

and returns 4 bytes in the range 0-255, like:

AAAAAAAA, BBBBBBBB, CCCCCCCC, DDDDDDDD

In that function, >> means "rotate the value to the right that many bits", and "& 255" means "only return the last 8 bits (128 + 64 + 32 + 16 + 8 + 4 + 2 + 1)".

So if we're passed in the "AAAA...DDDD" number above:

• value >> 24 => AAAAAAAA
• value >> 16 => AAAAAAAABBBBBBBB. That value & 255 => BBBBBBBB
• value >> 8 => AAAAAAAABBBBBBBBCCCCCCCC. That value & 255 => CCCCCCCC
• value & 255 => DDDDDDDD

That's the more-or-less standard way of converting a 32-bit IPv4 address into a list of 4 bytes. When you join those values together with a dot, you get the normal "A.B.C.D" address format.

Answer 2

Here is a plan:

1. Transform all your IP's in an int. That is, IP[0]*(256**3) + IP[1]*(256**2) + IP[2]*256 + IP[3] for IPv4.
2. store all the IP you got in a dict, with the IP_as_int as key
3. iterate over all the ips of your subnet (as ints, use range() for that), and get() your dict for each IP. If get() returns None, print your default message along with the current IP. Otherwise, print the returned list.

So, first step of our plan:

def ip_tuple2int(ip_tuple):
     return sum(ip_part * 256**(len(ip_tuple) - no) for no, ip_part in enumerate(ip_tuple, start=1))

We will need a function to print those later. Let's say we use:

def ip_int2str(ip_int):
    l = []
    while ip_int > 0:
        l.insert(0, ip_int % 256)
        ip_int /= 256

    if len(l) > 4:
        return socket.inet_ntop(socket.AF_INET6, ''.join(chr(i) for i in l))
    else:
        return '.'.join(str(i) for i in l)

Second step:

d = {}
for rec in records:
    d[ip_tuple2int(ip2tuple(rec[1]))] = rec

For the third step, we need the network mask. We will assume it is stored in nmask, like this: nmask = ip_tuple2int(ip2tuple("255.255.254.0")) (yep this mask is unusual, for it's best to solve the more general problem.

min_ip = d.keys()[0] & nmask
max_ip = min_ip | nmask ^ 2**int(math.ceil(math.log(nmask, 2))) - 1
    # the right-hand of the '|' is just the bitwise inversion of nmask
    # because ~nmask gives a negative number in python

for ip_int in range(min_ip, max_ip + 1):
    row = d.get(ip_int)
    if row:
        print row
    else:
        print [None, ip_int2str(ip_int), None, None, None]

.

So, this ends the solution. The code presented here supports both IPv4 and IPv6: it has been tested on some input for both cases, with the following ip2tuple()

def ip2tuple(ip_str):
    try:
        ip_bin = socket.inet_pton(socket.AF_INET, ip_str)
    except socket.error:
        ip_bin = socket.inet_pton(socket.AF_INET6, ip_str)

    return [ord(c) for c in ip_bin]

The code from your question still needs to be adapted if you want to also accept IPv6.

Finally, this code also supports any network mask, as long as the most significant bit of the address type is set.

EDIT: more on two complicated lines: min_ip and max_ip

So, we had

min_ip = d.keys()[0] & nmask
max_ip = min_ip | nmask ^ 2**int(math.ceil(math.log(nmask, 2))) - 1

to compute the minimal and maximal ip of our range. Let us break those down!

min_ip = d.keys()[0] & nmask

we take here an arbitrary ip d.keys()[0], and AND it with with the network mask: we conserve as-is the bits of the ip that are constant, and zero the bits that make up the variable part.

max_ip = min_ip | nmask ^ 2**int(math.ceil(math.log(nmask, 2))) - 1

to compute the maximal ip, we take the constant part of the ips of our subnet, stored in min_ip, and we "add" (binary OR) the variable part of the ips with all bits set to 1. This is done by computing a binary row of 1 of the same size as nmask 2**int(math.ceil(math.log(nmask, 2))) - 1 and XOR with nmask so that all the bits set to 1 in nmask become 0 and all the low-order 0s of the network mask become 1.

Why this solution? because even though it is less clear, it adapts itself to the address type automatically. It might even support 4096 bit addresses -and more!

Answer 3

Done with numpy

import numpy as np

reservations = [
    # [DHCP SERVER, IP ADDRESS, MAC ADDRESS, HOSTNAME, DESCRIPTION]
    ['server1','172.16.0.120','31872fcefa33','wks120.domain.net',
    'Description of client'],
    ['server1','172.16.0.125','4791ca3d7279','wks125.domain.net',
    'Description of client'],
    ['server1','172.16.0.132','6035a71c930c','wks132.domain.net',
    'Description of client'],
]
occupied_ip = []
for i in reservations:
    occupied_ip.append(int(i[1][-3:]))
occupied_ip = np.array(occupied_ip)

iplist = np.arange(256)
idx = np.in1d(iplist,occupied_ip)    #Where are the two arrays equual?
idx = np.logical_not(idx)            #Where are they NOT equal
freeip = iplist[idx]

unreserved = []
for i in range(len(freeip)):
    unreserved.append(["server1", "172.16.0."+str(freeip[i]), "Unassigned MAC",
    "unknown domain"])
    print unreserved[i]

Produces

....
    ['server1', '172.16.0.117', 'Unassigned MAC', 'unknown domain']
    ['server1', '172.16.0.118', 'Unassigned MAC', 'unknown domain']
    ['server1', '172.16.0.119', 'Unassigned MAC', 'unknown domain']
    ['server1', '172.16.0.121', 'Unassigned MAC', 'unknown domain']
    ['server1', '172.16.0.122', 'Unassigned MAC', 'unknown domain']
    ['server1', '172.16.0.123', 'Unassigned MAC', 'unknown domain']
    ['server1', '172.16.0.124', 'Unassigned MAC', 'unknown domain']
    ['server1', '172.16.0.126', 'Unassigned MAC', 'unknown domain']
    ['server1', '172.16.0.127', 'Unassigned MAC', 'unknown domain']
    ['server1', '172.16.0.128', 'Unassigned MAC', 'unknown domain']
    ['server1', '172.16.0.129', 'Unassigned MAC', 'unknown domain']
    ['server1', '172.16.0.130', 'Unassigned MAC', 'unknown domain']
    ['server1', '172.16.0.131', 'Unassigned MAC', 'unknown domain']
    ['server1', '172.16.0.133', 'Unassigned MAC', 'unknown domain']
...