Algorithm to extract network info from ifconfig (ubuntu)

Question

Im trying to parse info from ifconfig (ubuntu). Normally, I would split a chunk of data like this down into words, and then search for substrings to get what I want. For example, given line = "inet addr:192.168.98.157 Bcast:192.168.98.255 Mask:255.255.255.0", and looking for the broadcast address, I would do:

for word in line.split():
    if word.startswith('Bcast'):
        print word.split(':')[-1]

>>>192.168.98.255

However, I feel its about time to start learning how to use regular expressions for tasks like this. Here is my code so far. I've hacked through a couple of patterns (inet addr, Bcast, Mask). Questions after code...

# git clone git://gist.github.com/1586034.git gist-1586034
import re
import json

ifconfig = """
eth0      Link encap:Ethernet  HWaddr 08:00:27:3a:ab:47  
          inet addr:192.168.98.157  Bcast:192.168.98.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe3a:ab47/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:189059 errors:0 dropped:0 overruns:0 frame:0
          TX packets:104380 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:74213981 (74.2 MB)  TX bytes:15350131 (15.3 MB)\n\n
lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:389611 errors:0 dropped:0 overruns:0 frame:0
          TX packets:389611 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:81962238 (81.9 MB)  TX bytes:81962238 (81.9 MB)
"""

for paragraph in ifconfig.split('\n\n'):
        
    info = {
        'eth_port': '',
        'ip_address': '',
        'broadcast_address': '',
        'mac_address': '',
        'net_mask': '',
        'up': False,
        'running': False,
        'broadcast': False,
        'multicast': False,
    }
    
    if 'BROADCAST' in paragraph:
        info['broadcast'] = True
        
    if 'MULTICAST' in paragraph:
        info['multicast'] = True
        
    if 'UP' in paragraph:
        info['up'] = True
        
    if 'RUNNING' in paragraph:
        info['running'] = True
        
    ip = re.search( r'inet addr:[^\s]+', paragraph )
    if ip:
        info['ip_address'] = ip.group().split(':')[-1]  
    
    bcast = re.search( r'Bcast:[^\s]+', paragraph )
    if bcast:
        info['broadcast_address'] = bcast.group().split(':')[-1]
    
    mask = re.search( r'Mask:[^\s]+', paragraph )
    if mask:
        info['net_mask'] = mask.group().split(':')[-1]

    print paragraph
    print json.dumps(info, indent=4)

Here're my questions:

Am I taking the best approach for the patterns I have already implemented? Can I grab the addresses without splitting on ':' and then choosing the last of the array.?
I'm stuck on HWaddr. What would be a pattern to match this mac address?

EDIT:

Ok, so here's how I ended up going about this. I started out trying to go about this without the regex... just manipulating stings and lists. But that proved to be a nightmare. For example, what separates HWaddr from its address is a space. Now take inet addr its separated from its address by :. Its a tough problem to scrape with differing separators like this. Not only a problem to code but also a problem to read.

So, I did this with regex. I think this makes a strong case for when to use regular expressions.

# git clone git://gist.github.com/1586034.git gist-1586034

# USAGE: pipe ifconfig into script. ie "ifconfig | python pyifconfig.py"
# output is a list of json datastructures

import sys
import re
import json

ifconfig = sys.stdin.read()

print 'STARTINPUT'
print ifconfig
print 'ENDINPUT'

def extract(input):
    mo = re.search(r'^(?P<interface>eth\d+|eth\d+:\d+)\s+' +
                     r'Link encap:(?P<link_encap>\S+)\s+' +
                     r'(HWaddr\s+(?P<hardware_address>\S+))?' +
                     r'(\s+inet addr:(?P<ip_address>\S+))?' +
                     r'(\s+Bcast:(?P<broadcast_address>\S+)\s+)?' +
                     r'(Mask:(?P<net_mask>\S+)\s+)?',
                     input, re.MULTILINE )
    if mo:
        info = mo.groupdict('')
        info['running'] = False
        info['up'] = False
        info['multicast'] = False
        info['broadcast'] = False
        if 'RUNNING' in input:
            info['running'] = True
        if 'UP' in input:
            info['up'] = True
        if 'BROADCAST' in input:
            info['broadcast'] = True
        if 'MULTICAST' in input:
            info['multicast'] = True
        return info
    return {}


interfaces = [ extract(interface) for interface in ifconfig.split('\n\n') if interface.strip() ]
print json.dumps(interfaces, indent=4)

synthesizerpatel · Answer 1 · 2016-03-25T04:31:29.640

13

Rather than reinventing the wheel:

https://pypi.python.org/pypi/python-ifconfig/0.1

Or if you want a portable-ish version that works on multiple platforms..

https://pypi.python.org/pypi/netifaces/

edited Mar 25 '16 at 04:31

answered Jan 10 '12 at 01:21

synthesizerpatel

27,321
5
74
91

score 5 · Accepted Answer · answered Jan 10 '12 at 01:08

Am I taking the best approach for the patterns I have already implemented? Can I grab the addresses without splitting on ':' and then choosing the last of the array.?

Your patterns are fine for what they are doing, although [^\s] is equivalent to \S.

You can grab the addresses without splitting on ':' by putting the address into a capturing group, like this:

    ip = re.search(r'inet addr:(\S+)', paragraph)
    if ip:
        info['ip_address'] = ip.group(1)

If you had more grouped portions of the regex you could refer to them by the order they appear in your regex, starting at 1.

I'm stuck on HWaddr. What would be a pattern to match this mac address?

Now that you know about grouping, you can get HWaddr the same way as the other addresses:

    mac = re.search(r'HWaddr\s+(\S+)', paragraph)
    if mac:
        info['mac_address'] = mac.group(1)

Note that with a more advanced regular expression you could actually do several of these steps all at once. For example here is an example regex that pulls out the interface name, ip address, and net mask in one step:

>>> re.findall(r'^(\S+).*?inet addr:(\S+).*?Mask:(\S+)', ifconfig, re.S | re.M)
[('eth0', '192.168.98.157', '255.255.255.0'), ('lo', '127.0.0.1', '255.0.0.0')]

Thanks for hitting each question. Capturing groups were the missing piece for me. — sbartell, Jan 10 '12 at 18:39

score 2 · Answer 3 · answered Jun 13 '15 at 19:38

With new 3.0.0 version of psutil (https://github.com/giampaolo/psutil) you can avoid parsing ifconfig output and do this directly in Python: http://pythonhosted.org/psutil/#psutil.net_if_addrs

>>> import psutil
>>> psutil.net_if_addrs()
{'lo': [snic(family=<AddressFamily.AF_INET: 2>, address='127.0.0.1', netmask='255.0.0.0', broadcast='127.0.0.1'),
        snic(family=<AddressFamily.AF_INET6: 10>, address='::1', netmask='ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff', broadcast=None),
        snic(family=<AddressFamily.AF_LINK: 17>, address='00:00:00:00:00:00', netmask=None, broadcast='00:00:00:00:00:00')],
 'wlan0': [snic(family=<AddressFamily.AF_INET: 2>, address='192.168.1.3', netmask='255.255.255.0', broadcast='192.168.1.255'),
           snic(family=<AddressFamily.AF_INET6: 10>, address='fe80::c685:8ff:fe45:641%wlan0', netmask='ffff:ffff:ffff:ffff::', broadcast=None),
           snic(family=<AddressFamily.AF_LINK: 17>, address='c4:85:08:45:06:41', netmask=None, broadcast='ff:ff:ff:ff:ff:ff')]}
>>>

Giacomo Lacava · Answer 4 · 2012-01-10T01:16:34.280

To be honest, regular expressions are not particularly better than simple string manipulation; if anything, they're always slower.

This said, you should start cleaning your input with a better split:

lines = [line.strip() for line in ifconfig.split("\n") if line.strip() != '']

This removes all whitespace around the lines, and discards empty ones; your regexes can now start with ^ and end with $, which will reduce the possibility of false positives.

Then you'd really have to look at grouping; the patterns you're using are just glorified startswith, and certainly less optimized than startswith will ever be. A regex guru will come up with better, but for example a simple pattern for the HWAddr line would be

>>> m = re.match(r'^([A-z]*\d)\s+(Link)\s+(encap):([A-z]*)\s+(HWaddr)\s+([A-z0-9:]*)$',lines[0])
>>> m.groups()
('eth0', 'Link', 'encap', 'Ethernet', 'HWaddr', '08:00:27:3a:ab:47')

But really, the more I look at it, the more the simpler approach based on split() and split(':') makes sense for such a rigidly formatted input. Regexes make your code less readable, and are very expensive. As JWZ once said, "Some people, when confronted with a problem, think 'I know, I'll use regular expressions.' Now they have two problems."

Yea regex can look nasty, but so can filter expressions for list comprehensions. Any way you go about it it can get ugly (thats totally up to interpretation though). I think regular expressions shine in a case like this where the chunk is small, the input is a regular language, there are multiple search patterns, and the patterns are well defined. — sbartell, Jan 10 '12 at 18:50

score 1 · Answer 5 · answered Jan 10 '12 at 01:13

Try something along:

>>> import re  
>>> m = re.search(r'^(?P<interface>eth\d+|eth\d+:\d+|lo|ppp\d+)\s+' +
...              r'Link encap:(?P<link_encap>\S+)\s+' +
...              r'HWaddr\s(?P<hardware_address>[0-9a-f]{2}:[0-9a-f]{2}:' +
...              r'[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2})\s', 
...              ifconfig,
...              re.MULTILINE
...    )
>>> m.groupdict()
{'hardware_address': '08:00:27:3a:ab:47',
 'interface': 'eth0',
 'link_encap': 'Ethernet'}

now that F.J explained capturing groups, this makes sense. I like the group names addition here. And spitting it all out in a dict is pretty cool. — sbartell, Jan 10 '12 at 18:59

Algorithm to extract network info from ifconfig (ubuntu)

5 Answers5

Linked