Issue with scraping data using beautiful soup

Question

I am using following code to scrape data from a website.

# -*- coding: cp1252 -*-
import urllib2
import sys
from bs4 import BeautifulSoup

page = urllib2.urlopen('http://www.att.com/shop/wireless/plans-new.html#fbid=U-XD_DHOGEp').read()
soup = BeautifulSoup(page)
plans = soup.findAll('div', {"class": "planTitle"})
for plan in plans:
    planname = u' '.join(plan.stripped_strings)
    plantypes = soup.findAll('div', {"class":"top"})
    prices = soup.findAll('div', {"class":"bottom"})
    for plantype, price in zip(plantypes, prices):
        plantype1 = u' '.join(plantype.stripped_strings)
        price1 = u' '.join(price.stripped_strings)
        print planname, plantype1, price1

Problem: If you go through the webpage which is mentioned in this code, these are 4-5 types of plans and for each plan 3 voice options and some 2-3 data options are present. I want to scrape the data in such way that for each plan I can get its respective voice options against it and then monthly prices for those options.

Code which I am running right now returns the all possible combinations of plan name + voice options. For each plan name I get some 20-30 entries because it creates a entry even for wrong Plan name + voice options combination. Eg. Individual Plan - 550 minutes - $59.99, in this combination 500 minutes and 59.99 are part of Family Plan.

I want the loop to run so that only correct Plan + Voice options combination is extracted.

Snippet of Webpage: For each plan one box is present on webpage, which contains voice options and prices corresponding to those options, I want the loop to run for each box, but element + class combination for voice options and their prices is not unique. That's why plan name takes value from other boxex also.

<div class="innerContainer"> 

    <div class="planTitle"> 
        <h2><a href="http://www.att.com/shop/wireless/plans/individualplans.html" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010">AT&amp;T Individual Plans</a></h2> 
    </div> 
    <div class="planSubTitle"> 
        <img src="/shopcms/media/att/2012/shop/wireless/promotions/Plans-CM_page/tiny-clock.jpg" alt=""> 
        <p>Voice plan options:</p> 
    </div> 
    <!-- Begin three white boxes --> 
    <!-- Note, extra boxes can be added to the row with the following method  --> 
    <!-- 1. Add more div containers inside .whiteBox  --> 
    <!-- 2. Modify class names to boxes_one, boxes_two, boxes_three etc... (max six) --> 
    <div class="whiteBox"> 
        <div class="boxes_three"> 
            <a class="lnk-help tooltips fullBoxLink" href="#smartphone_individual_voice_450" onclick="window.location.href = 'http://www.att.com/shop/wireless/plans/voice/sku3830290.html?source=IC95ATPLP00PSP00L&amp;wtExtndSource=spindvoice450';return false;" aria-describedby="smartphone_individual_voice_450" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010" title=""></a> 
            <span id="smartphone_individual_voice_450" class="tips" role="tooltip">$0.45/min. for additional minutes</span> 
            <div class="top"> 
                <p class="stat">450</p> 
                <p class="statText">Minutes</p> 
            </div> 
            <div class="bottom"> 
                <p>$39.99/mo.</p> 
            </div> 
        </div> 
        <div class="boxes_three"> 
            <a class="lnk-help tooltips fullBoxLink" href="#smartphone_individual_voice_900" onclick="window.location.href = 'http://www.att.com/shop/wireless/plans/voice/sku3830292.html?source=IC95ATPLP00PSP00L&amp;wtExtndSource=spindvoice900';return false;" aria-describedby="smartphone_individual_voice_900" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010" title=""></a> 
            <span id="smartphone_individual_voice_900" class="tips" role="tooltip">$0.40/min. for additional minutes</span> 
            <div class="top"> 
                <p class="stat">900</p> 
                <p class="statText">Minutes</p> 
            </div> 
            <div class="bottom"> 
                <p>$59.99/mo.</p> 
            </div> 
        </div> 
        <div class="boxes_three borderNone"> 
            <a class="fullBoxLink" href="http://www.att.com/shop/wireless/plans/voice/sku3830293.html?source=IC95ATPLP00PSP00L&amp;wtExtndSource=spindvoiceunlim" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010"></a> 
            <div class="top"> 
                <p class="stat">Unlimited</p> 
                <p class="statText">Minutes</p> 
            </div> 
            <div class="bottom"> 
                <p>$69.99/mo.</p> 
            </div> 
        </div> 
    </div> 
    <!-- End three white boxes --> 
    <!-- Begin left gray container --> 
    <div class="containerTwoThirds"> 
        <div class="planSubTitle"> 
            <img src="/shopcms/media/att/2012/shop/wireless/promotions/Plans-CM_page/tiny-globe.jpg" alt=""> 
            <p>Data plan options:</p> 
        </div> 
        <div class="grayTwoThirds"> 
            <div class="grayBox"> 
                <a class="fullBoxLink" href="http://www.att.com/shop/wireless/services/dataplus300mb-smartphone4glte-sku5380269.html?source=IC95ATPLP00PSP00L&amp;wtExtndSource=spinddata300mb" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010"></a> 
                <p class="stat"><strong>300MB</strong></p> 
                <p class="statText">$20.00/mo.</p> 
            </div> 
            <div class="grayBoxBreak"></div> 
            <div class="grayBox"> 
                <a class="fullBoxLink" href="http://www.att.com/shop/wireless/services/datapro3gb-smartphone4glte-sku5470232.html?source=IC95ATPLP00PSP00L&amp;wtExtndSource=spinddata3gb" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010"></a> 
                <p class="stat"><strong>3GB</strong></p> 
                <p class="statText">$30.00/mo.</p> 
            </div> 
            <div class="grayBoxBreak"></div> 
            <div class="grayBox"> 
                <a class="fullBoxLink" href="http://www.att.com/shop/wireless/services/datapro5gb-smartphone4glte-sku5480228.html?source=IC95ATPLP00PSP00L&amp;wtExtndSource=spinddata5gb" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010"></a> 
                <p class="stat"><strong>5GB</strong></p> 
                <p class="statText">$50.00/mo.</p> 
            </div> 
        </div> 
    </div> 
    <!-- End left gray container --> 
    <!-- Begin right gray container --> 
    <div class="containerThird"> 
        <div class="planSubTitle"> 
            <img src="/shopcms/media/att/2012/shop/wireless/promotions/Plans-CM_page/tiny-phone.jpg" alt=""> 
            <p>Messaging plan options: <span class="fix"></span></p> 
        </div> 
        <div class="grayThird"> 
            <div class="grayBox">  
                <a data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2012325" href="http://www.att.com/shop/wireless/services/messagingunlimited-sku1160055.html?source=IC95ATPLP00PSP00L&amp;wtExtndSource=spindmessunlim" class="fullBoxLink"></a>  
                <p class="stat"><strong>ULTD</strong> MSGS</p>  
                <p class="statText">$20.00/mo.</p>  
            </div> 
            <div class="grayBoxBreak"></div> 
            <div class="grayBox last"> 
                <p class="stat"><strong>PAY PER USE</strong></p> 
                <p class="statText">20¢/text <span class="lightGray">|</span> 30¢/pic/video</p> 
            </div> 
        </div> 
    </div> 
    <!-- End right gray container --> 
    <!-- Begin sub footer --> 
    <div class="bottomLinks">  
        <div class="links"> 
            <a href="http://www.att.com/shop/wireless/plans/individualplans.html?taxoPlan=POSTPAID-INDIVIDUAL-CANADA&amp;source=IC95ATPLP00PSP00L&amp;wtExtndSource=spindcanada" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010">Nation with Canada Plans</a> | <a href="http://www.att.com/shop/wireless/plans/voice/sku5740279.html?source=IC95ATPLP00PSP00L&amp;wtExtndSource=spindhomephone" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010">Unlimited Home Phone</a> | <a href="http://www.att.com/shop/wireless/plans/voice/sku3830294.html?source=IC95ATPLP00PSP00L&amp;wtExtndSource=spindsenior" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010">Senior Plans</a> 
        </div> 
        <a class="shop_button" href="http://www.att.com/shop/wireless/devices/smartphones.html?source=IC95ATPLP00PSP00L&amp;wtExtndSource=indshopsp" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010"><img src="/shopcms/media/att/2012/shop/wireless/promotions/Plans-CM_page/buttons/shop_smartphones.png" alt="Shop Smartphones" width="158" height="29"></a> 
    </div> 
    <!-- End sub footer --> 
</div>

Please help me in solving this issue as I am new to programming.

Can you please include a HTML sample? This question is to really too localized at the moment (as soon as the ATT site changes your question will be useless to future visitors). — Martijn Pieters, Jan 14 '13 at 12:16
@MartijnPieters do I need to add/mention anything else to make this problem more generic? — , Jan 14 '13 at 12:47
Nope, the HTML snippet is excellent. No time right now for a detailed answer, but that won't stop others from helping hopefully. — Martijn Pieters, Jan 14 '13 at 12:52
Aah, okay thanks! Please do look into this though if you get time. — , Jan 14 '13 at 12:56

score 1 · Accepted Answer · edited May 23 '17 at 11:48

Rewrote the thing from scratch. There's no comments, but it's pretty self explanatory. The lambda in the dictionary is for finding attributes that start with a certain string. I referenced this answer for that: https://stackoverflow.com/a/2830550/541208

I had thought that you were using findAll on soup, when you should have been using plan.findAll instead, but then it didn't help anything, so I just rewrote the whole thing.

import urllib2
import sys
from bs4 import BeautifulSoup


page = urllib2.urlopen('http://www.att.com/shop/wireless/plans-new.html#fbid=U-XD_DHOGEp').read()
soup = BeautifulSoup(page)

#find the container for all the plans
tabcontent = soup.find('div', {"id": "smartphonePlans", "class": "tabcontent"})
containers = tabcontent.findAll('div', {"class": "innerContainer"})

for plan in containers:
     planTitle = plan.find("div", {"class": "planTitle"})
     if planTitle:
          title = planTitle.find("a").text     
          print title          

     voiceBoxes = plan.find("div", {"class": "whiteBox"})     
     if voiceBoxes:
               box3 = voiceBoxes.findAll("div", {"class": lambda x: x and x.startswith("boxes_")})
               if box3:
                    for box in box3:
                         top = box.findAll("p")
                         minutes = u" ".join([tag.text for tag in top])
                         print "\t", minutes

Which outputs:

AT&T Individual Plans
    450 Minutes $39.99/mo.
    900 Minutes $59.99/mo.
    Unlimited Minutes $69.99/mo.
AT&T Family Plans
    550 Minutes $59.99/mo.
    700 Minutes $69.99/mo.
    1,400 Minutes $89.99/mo.
    2,100 Minutes $109.99/mo.
    Unlimited Minutes $119.99/mo.
AT&T Mobile Share Plans
    1GB $40/mo. + $45/smartphone
    4GB $70/mo. + $40/smartphone
    6GB $90/mo. + $35/smartphone
    10GB $120/mo.
    15GB $160/mo. + $30/smartphone
    20GB $200/mo.

In the above code, output we are getting is for voice plans, but what if I want output for Data & messaging plans also which are present in gray boxes. I tried using the similar approach but no output came. — , Jan 15 '13 at 09:56
Try opening the page in Chrome/Firefox and enabling dev tools, so you can 'Inspect Element' and then trace the path of each `grayBox` from the outer container, then try to recreate that with code. Put a ton of breakpoints after each `findAll` to see what data is being returned and to confirm you're selecting the right pieces. — TankorSmash, Jan 15 '13 at 14:55

Issue with scraping data using beautiful soup

1 Answers1