0

I am trying to scrape some information from a website. I require 8 fields of information, I have got it for 5 fields, but 3 fields are always coming empty. I think there is some mistake with my regular expression formulation. I am doing it in python and I don't have to use BS. Here are the HTML fileds I need to scrape. This is example of one of the webpage.

enter code here

<td><span class="facultyName">John Matthew Falletta, MD</span>

<span class="primaryTitle">Professor of Pediatrics</span>

<span class="secondaryTitle">Professor in the School of Nursing</span>

<td><span class="label">Department:</span>
        &nbsp;&nbsp;
    </td><td>Pediatrics</td>

<td><span class="label">Division:</span>
        &nbsp;&nbsp;
    </td><td>Hematology/Oncology</td>

<td><span class="label">Address:</span></td><td>Box 2991<br>DUMC<br>Durham, NC &nbsp;27710   </td>

<td><span class="label">Phone:</span></td><td>
       (919)
       668-5111<br>

<td><span class="label">FAX:</span></td><td>                
        (919)
        688-5125</td>

Here is my code containing respective regular expressions for each type of tag:

enter code here

patFinderFullname = re.compile('<span class="facultyName">(.*)</span>')
patFinderPTitle = re.compile('<span class="primaryTitle">(.*)</span>')
patFinderSTitle = re.compile('<span class="secondaryTitle">(.*)</span>')
patFinderDepartment = re.compile('<span class="label">Department:</span>\s+&nbsp;&nbsp;\s+</td><td>(.*)</td>')
patFinderDivision = re.compile('<span class="label">Division:</span>\s+&nbsp;&nbsp;\s+</td><td>(.*)')

patFinderAddress = re.compile(' <span class="label">Address:</span>\s+(.*)\s+</td>')
patFinderPhone = re.compile('<span class="label">Phone:</span></td><td>\s*(.*?)\s*<br>')
patFinderFax = re.compile('<td><span class="label">FAX:</span>\s+</td><td>\s+(.*)</td>')

First five field results are coming correct, but the last three fields for Address, Phone and Fax are returning always empty. Can anyone point out what I am missing? Or what is wrong with the regular expressions for the last three fields. I have posted an earlier [1][question], but these problems arrived later to that, so I am asking it in a different question.

[1] : How to scrape html tags spread over multiple lines in python?

Community
  • 1
  • 1
Steve
  • 1
  • 7
  • You are missing a closing bracket on your Division `re.compile()`. In fact the end of that expression doesn't look right at all, should you not have a closing `)'` – Pep_8_Guardiola Feb 15 '13 at 07:30
  • Sorry, If I missed in DIvision, it must be a typo. I am having probles in the last three fields i.e Address, phone and fax. – Steve Feb 15 '13 at 07:31
  • 2
    Why not use an HTML parser? – Janne Karila Feb 15 '13 at 07:36
  • I am learning web scraping, and I was familiar to python earlier, and it also has BS, so I thought it would be a good idea, to do it in python. And I have already got most of the result correct, so I would like to continue with this. – Steve Feb 15 '13 at 07:40
  • You should really invest in learning a parsing library, because every time your task changes your expressions become more or less useless. – root Feb 15 '13 at 07:45
  • I will try to learn a parsing library. But I need this code to work, as I have tried, and couldn't fix the bugs. – Steve Feb 15 '13 at 07:49
  • Err, I tried re.compile('(.*)') and it returns 'John Matthew Falletta, MD'. I'm assuming you (should) only want the name John Matthew Falleta from your regex? This is precisely why an html parser like BeautifulSoup should be used. – Jason White Feb 15 '13 at 07:59

1 Answers1

1
patFinderAddress = re.compile('<td><span class="label">Address:</span></td>.*?</td>'
patFinderPhone  = re.compile('<td><span class="label">Phone:</span>\s*</td><td>\s*^\s*.*\s*^\s*.*<br>',re.M)
patFinderFax = re.compile('<td><span class="label">FAX:</span>\s*</td><td>\s*^\s*.*\s*^\s*.*</td>',re.M)

Here's the some regexs that work with your data. The last two weren't working as the data spanned multiple lines. The first didn't work because it was wrong.

But, for html parsing, use an html parser as it's far more robust and gives you the data you want rather than this eyesore of html strings.

Jason White
  • 666
  • 4
  • 10
  • 23
  • I will learn how to learn how to use BS. I tried the above regex,they are displaying the whole data including all the tags. – Steve Feb 15 '13 at 08:26
  • So are your regexs! If you want only the data, use a parser. No way in hell I'm crafting regexs to do a parsers job. – Jason White Feb 15 '13 at 08:31
  • I am new to this, so I didn't know what's easy and what's not. Okay, I will learn and use BS. Thanks for your time. – Steve Feb 15 '13 at 08:35