Retrieve all content between a closing and opening html tag using Beautiful Soup

Question

I am parsing content using Python and Beautiful Soup then writing it to a CSV file, and have run into a bugger of a problem getting a certain set of data. The data is ran through an implementation of TidyHTML that I have crafted and then other not needed data is stripped out.

The issue is that I need to retrieve all data between a set of <h3> tags.

Sample Data:

<h3><a href="Vol-1-pages-001.pdf">Pages 1-18</a></h3>
<ul><li>September 13 1880. First regular meeting of the faculty;
 September 14 1880. Discussion of curricular matters. Students are
 debarred from taking algebra until they have completed both mental
 and fractional arithmetic; October 4 1880.</li><li>All members present.</li></ul>
 <ul><li>Moved the faculty henceforth hold regular weekkly meetings in the
 President's room of the University building; 11 October 1880. All
 members present; 18 October 1880. Regular meeting 2. Moved that the
 President wait on the property holders on 12th street and request
 them to abate the nuisance on their property; 25 October 1880.
 Moved that the senior and junior classes for rhetoricals be...</li></ul>
 <h3><a href="Vol-1-pages-019.pdf">Pages 19-33</a></h3>`

I need to retrieve all of the content between the first closing </h3> tag and the next opening <h3> tag. This shouldn't be hard, but my thick head isn't making the necessary connections. I can grab all of the <ul> tags but that doesn't work because there is not a one to one relationship between <h3> tags and <ul> tags.

The output I am looking to achieve is:

Pages 1-18|Vol-1-pages-001.pdf|content between and tags.

The first two parts have not been a problem but content between a set of tags is difficult for me.

My current code is as follows:

import glob, re, os, csv
from BeautifulSoup import BeautifulSoup
from tidylib import tidy_document
from collections import deque

html_path = 'Z:\\Applications\\MAMP\\htdocs\\uoassembly\\AssemblyRecordsVol1'
csv_path = 'Z:\\Applications\\MAMP\\htdocs\\uoassembly\\AssemblyRecordsVol1\\archiveVol1.csv'

html_cleanup = {'\r\r\n':'', '\n\n':'', '\n':'', '\r':'', '\r\r': '', '<img src="UOSymbol1.jpg"    alt="" />':''}

for infile in glob.glob( os.path.join(html_path, '*.html') ):
    print "current file is: " + infile

    html = open(infile).read()

    for i, j in html_cleanup.iteritems():
            html = html.replace(i, j)

    #parse cleaned up html with Beautiful Soup
    soup = BeautifulSoup(html)

    #print soup
    html_to_csv = csv.writer(open(csv_path, 'a'), delimiter='|',
                      quoting=csv.QUOTE_NONE, escapechar=' ')  
    #retrieve the string that has the page range and file name
    volume = deque()
    fileName = deque()
    summary = deque()
    i = 0
    for title in soup.findAll('a'):
            if title['href'].startswith('V'):
             #print title.string
             volume.append(title.string)
             i+=1
             #print soup('a')[i]['href']
             fileName.append(soup('a')[i]['href'])
             #print html_to_csv
             #html_to_csv.writerow([volume, fileName])

    #retrieve the summary of each archive and store
    #for body in soup.findAll('ul') or soup.findAll('ol'):
    #        summary.append(body)
    for body in soup.findAll('h3'):
            body.findNextSibling(text=True)
            summary.append(body)

    #print out each field into the csv file
    for c in range(i):
            pages = volume.popleft()
            path = fileName.popleft()
            notes = summary
            if not summary: 
                    notes = "help"
            if summary:
                    notes = summary.popleft()
            html_to_csv.writerow([pages, path, notes])

Answer 1

Extract content between </h3> and <h3> tags:

from itertools import takewhile

h3s = soup('h3') # find all <h3> elements
for h3, h3next in zip(h3s, h3s[1:]):
  # get elements in between
  between_it = takewhile(lambda el: el is not h3next, h3.nextSiblingGenerator())
  # extract text
  print(''.join(getattr(el, 'text', el) for el in between_it))

The code assumes that all <h3> elements are siblings. If it is not the case then you could use h3.nextGenerator() instead of h3.nextSiblingGenerator().

Answer 2

If you try to extract data between <ul><li></ul></li> tags in lxml, it provides a great functionality of using CSSSelector

import lxml.html
import urllib
data = urllib.urlopen('file:///C:/Users/ranveer/st.html').read() //contains your html snippet
doc = lxml.html.fromstring(data)
elements = doc.cssselect('ul li') // CSSpath[using firebug extension]
for element in elements:
      print element.text_content()    

after executing the above code you will get all text between the ul,li tags. It is much cleaner than beautiful soup.

If you by any chance plan to use lxml than you can evaluate XPath expressions in the following way-

import lxml
from lxml import etree
content = etree.HTML(urllib.urlopen("file:///C:/Users/ranveer/st.html").read())
content_text = content.xpath("html/body/h3[1]/a/@href | //ul[1]/li/text() | //ul[2]/li/text() | //h3[2]/a/@href")
print content_text

You can change XPath according to your need.