2

I am facing an issue here. I am trying to execute this code using scrapy-selenium but this is scraping nothing. What actually I am missing here. If it is because of 'page_source' then how to apply 'page_source' properly? Here is my code_

import scrapy 
from scrapy_selenium import SeleniumRequest 
from register.settings import * 
from selenium import webdriver 
from selenium.webdriver.chrome.options import Options 
import time


class DataSpider(scrapy.Spider):
    name = 'data'

def start_requests(self):
    yield SeleniumRequest(
        url='https://registers.maryland.gov/RowNetWeb/Estates/frmEstateSearch2.aspx/',
        wait_time=3,
        callback=self.parse
    )

def parse(self, response):
    chrome_options = Options()
    chrome_options.add_argument('__headless')

    chrome_path = SELENIUM_DRIVER_EXECUTABLE_PATH
    driver = webdriver.Chrome(executable_path=chrome_path, options= chrome_options)
    driver.get("https://registers.maryland.gov/RowNetWeb/Estates/frmEstateSearch2.aspx/")
    driver.set_window_size(1920, 1080)
    time.sleep(3)

    search_btn = driver.find_element_by_id('cmdSearch')
    search_btn.click()
    time.sleep(10)

    results = response.xpath("//table[@id='dgSearchResults']/tbody/tr[position() = 1 < position() = 21]/td[2]/a")

    for result in results:
        result.click()
        for info in result:
            yield {
                'Estate Number:': info.xpath("//span[@id='lblEstateNumber']/text()").get()
            }

    driver.quit()
Raisul Islam
  • 277
  • 2
  • 19
  • Which data are you trying to scrape? – undetected Selenium Dec 21 '21 at 13:39
  • @DebanjanB I want to go to all the pages from 'results' by clicking it, and then get all the data fields from it. let's say, I want to get to page by clicking it and the get all the data from "https://registers.maryland.gov/RowNetWeb/Estates/frmDocketImages.aspx?src=row&RecordId=1028414408" this page and again return to the loop and get the next page to get the data. I hope I made it clear. – Raisul Islam Dec 21 '21 at 13:51
  • What is `register.settings`? – me.limes Dec 24 '21 at 13:56
  • @me.limes "register" is my project folder name – Raisul Islam Dec 24 '21 at 17:10

0 Answers0