problem scraping with BeautifulSoup

Question

I am trying to scrape the url http://www.kat.ph/search/beatles/?categories[]=music using BeautifulSoup

torrents = bs.findAll('tr',id = re.compile('torrent_*'))

torrents gets all the torrents on that page ,now every element of torrents contains a tr element.

My problem is that len(torrents[0].td) is 5 but i am not able to iterate over the td's.I mean something like for x in torrents[o].td is not working.

the data that i am getting for torrent[0] is :

<tr class="odd" id="torrent_2962816">
<td class="fontSize12px torrentnameCell">
<div class="iaconbox floatedRight">
<a title="Torrent magnet link" href="magnet:?xt=urn:btih:0898a4b562c1098eb69b9b801c61a51d788df0f5&amp;dn=the+beatles+2009+greatest+hits+cdrip+ikmn+reupld&amp;tr=http%3A%2F%2Ftracker.publicbt.com%2Fannounce" onclick="_gaq.push(['_trackEvent', 'Download', 'Magnet Link', 'Music']);" class="imagnet icon16"></a>
<a title="Download torrent file" href="http://torrage.com/torrent/0898A4B562C1098EB69B9B801C61A51D788DF0F5.torrent?title=[kat.ph]the.beatles.2009.greatest.hits.cdrip.ikmn.reupld" onclick="_gaq.push(['_trackEvent', 'Download', 'Download torrent file', 'Music']);" class="idownload icon16"></a>
<a class="iPartner2 icon16" href="http://www.downloadweb.org/checking.php?acode=b146a357c57fddd450f6b5c446108672&amp;r=d&amp;qb=VGhlIEJlYXRsZXMgWzIwMDldIEdyZWF0ZXN0IEhpdHMgQ0RSaXAtIGlLTU4gUmVVUGxk" onclick="_gaq.push(['_trackEvent', 'Download', 'Download movie']);"></a>
<a class="iverif icon16" href="/the-beatles-2009-greatest-hits-cdrip-ikmn-reupld-t2962816.html" title="Verified Torrent"></a> <a rel="2962816,0" class="icomment" href="/the-beatles-2009-greatest-hits-cdrip-ikmn-reupld-t2962816.html#comments_tab">
<span class="icommentdiv"></span>145
    </a>
</div>
<div class="torrentname">
<a href="/the-beatles-2009-greatest-hits-cdrip-ikmn-reupld-t2962816.html" class="torType musicType"></a>
<a href="/the-beatles-2009-greatest-hits-cdrip-ikmn-reupld-t2962816.html">The <strong class="red">Beatles</strong> [2009] Greatest Hits CDRip- iKMN ReUPld</a>
<span>
                Posted by <a class="plain" href="/user/iKMN/">iKMN</a>
<img src="http://static.kat.ph/images/verifup.png" alt="verified" /> in 
                    <span id="cat_2962816">
<a href="/music/">Music</a>
</span></span>
</div>
</td>
<td class="nobr">168.26 <span>MB</span></td>
<td>42</td>
<td>1&nbsp;year</td>
<td class="green">1368</td>
<td class="red lasttd">94</td>
</tr>

Why do you ask for len(torrents[0].td and then iterate over something completely different??? — , Jul 03 '11 at 14:00
@Matt - it's still a perfectly valid programming question that doesn't state any kind of nefarious intent. @Bunny , would you mind editing this post to include a sample of the data? If that link beaks, _so does the entire context of this question_. — Tim Post, Jul 03 '11 at 14:36
@Tim I wasn't implying that it wasn't, just joking. In fact, I'm going to start looking into BeautifulSoup now that I've seen this question :) — Matt, Jul 03 '11 at 14:40
Can you post an example of what your desired output would be? — serk, Jul 09 '11 at 00:24

score 2 · Answer 1 · answered Jul 03 '11 at 15:25

I'd recommend using lxml or instead of BeautifulSoup, among other great features you can use xpath to grab your links:

import lxml.html
doc = lxml.html.parse('http://www.kat.ph/search/beatles/?categories[]=music')
links = doc.xpath('//a[contains(@class,"idownload")]/@href')

problem scraping with BeautifulSoup

1 Answers1