Scrape table with Scrapy

Question

I have a table formed like this from a website:

    <table>
<tr class="head">
    <td class="One">
    Column 1
    </td>
    <td class="Two">
    Column 2
    </td>
    <td class="Four">
    Column 3
    </td>
    <td class="Five">
    Column 4
    </td>
</tr>
<tr class="DataSet1">
    <td class="One">
        <table>
            <tr>
                <td class="DataType1">
                    Data 1
                </td>
            </tr>
            <tr>
                <td class="DataType_2">
                    <ul>
                        <li> Data 2a</li>
                        <li> Data 2b</li>
                        <li> Data 2c</li>
                        <li> Data 2d</li>
                    </ul>
                </td>
            </tr>
        </table>
    </td>
    <td class="Two">
        <table>
            <tr>
                <td class="DataType_3">
                    Data 3
                </td>
            </tr>
            <tr>
                <td class="DataType_4">
                    Data 4
                </td>
            </tr>
        </table>
    </td>
    <td class="Three">
        <table>
            <tr>
                <td class="DataType_5">
                    Data 5
                </td>
            </tr>
        </table>
    </td>
    <td class="Four">
        <table>
            <tr>
                <td class="DataType_6">
                    Data 6
                </td>
            </tr>
        </table>
    </td>
</tr>
<tr class="Empty">
    <td class="One">
    </td>
    <td class="Two">
    </td>
    <td class="Four">
    </td>
    <td class="Five">
    </td>
</tr>
<tr class="DataSet2">
    <td class="One">
        <table>
            <tr>
                <td class="DataType_1">
                    Data 7
                </td>
            </tr>
            <tr>
                <td class="DataType_2">
                    Data 8
                </td>
            </tr>
        </table>
    </td>
    <td class="Two">
        <table>
            <tr>
                <td class="DataType_3">
                    Data 9
                </td>
            </tr>
            <tr>
                <td class="DataType_4">
                    Data 10
                </td>
            </tr>
        </table>
    </td>
    <td class="Three">
        <table>
            <tr>
                <td class="DataType_5">
                    Data 11
                </td>
            </tr>
        </table>
    </td>
    <td class="Four">
        <table>
            <tr>
                <td class="DataType_6">
                    Data 12
                </td>
            </tr>
        </table>
    </td>
</tr>
<!-- and so on -->
</table>

The tags sometimes are also empty, for example:

<td class="DataType_6> </td>

I tried to scrape the content with Scrapy and the following script:

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from project.items import ProjectItem

class MySpider(BaseSpider):
    name = "SpiderName"
    allowed_domains = ["url"]
    start_urls = ["url"]
    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        rows = hxs.select('//tr')
        items = []
        item = ProjectItem()
        item["Data_1"] = rows.select('//td[@class="DataType_1"]/text()').extract()
        item["Data_2"] = rows.select('//td[@class="DataType_2"]/text()').extract()
        item["Data_3"] = rows.select('//td[@class="DataType_3"]/text()').extract()
        item["Data_4"] = rows.select('//td[@class="DataType_4"]/text()').extract()
        item["Data_5"] = rows.select('//td[@class="DataType_5"]/text()').extract()
        item["Data_6"] = rows.select('//td[@class="DataType_6"]/text()').extract()
        items.append(item)
        return items

If I crawl using this command:

scrapy crawl SpiderName -o output.csv -t csv

I only get crap like as many times as I have got the Dataset all the values for "Data_1".

Answer 1

Had a similar problem. First of all, rows = hxs.select('//tr') is going to loop on everything from the first child. You need to dig a bit deeper, and use relative paths. This link gives an excellent explanation on how to structure your code.

When I finally got my head around it, I realised that in that order to parse each item separately, row.select should not have the // in it.

Hope this helps.