Extract table values from html web pages automatically

Question

I use MRTG to monitor devices' CPU/MEM/traffic usage, and each device has different ports/vlans/interfaces. When I want to create a table/excel showing statistics of all the interfaces, I found it very inefficient because I need to click into about 400 web pages and copy tables (In/Out traffic, Max/Avg/Current CPU usage, Max/Avg Connections, etc.) from the html.

My question: Is there any scripts/tools to automatize this work? [Extract certain table values from each web page and fill in a table/excel].

clj · Answer 1 · 2012-02-20T11:08:13.253

You might find that there are better ways to get the data you want than parsing the HTML generated by MRTG (e.g., such as using SNMP directly). However, this might help if really need to parse the HTML:

If you are willing to use Python then a the BeautifulSoup library will help you. You will still have to write some code but BeautifulSoup is pretty flexible.

Here is a crude example that dumps all HTML tables in a page to CSV:

import sys
import csv
import urllib2

import BeautifulSoup

page    = urllib2.urlopen(sys.argv[1]).read()
soup    = BeautifulSoup.BeautifulSoup(page)
csvout  = csv.writer(sys.stdout)

for table in soup.findAll('table'):
    print '#'
    print '# Table'
    print '# Fields: ' + ','.join([tr.text for tr in table.findAll('th')])
    for row in table.findAll('tr'):
        csvout.writerow([tr.text for tr in row.findAll('td')])
    print

Assuming that you have Python installed as well as BeautifulSoup (pip install BeautifulSoup or easy_install BeautifulSoup) and assuming that you have saved the above code as htmltable2csv.py you can display all the tables on a page like this:

python htmltable2csv.py http://www.w3schools.com/tags/tag_table.asp

score 0 · Accepted Answer · answered Feb 22 '12 at 14:35

I finally find Firefox extension iMacros for Firefox very useful in recording and replaying repetitious work. It works well and efficiently in extracting data from massive web pages with same structures.

What's more, iMacros has detailed tutorials and sample codes. iMacros for Firefox is free for personal and commercial use, if you have concern on this (like me).

I would strongly recommend iMacros if you ever want to automate some cumbersome work!

Extract table values from html web pages automatically

2 Answers2