Enter string into search field with Scrapy Spider; loading the generated URL

Question

Would the correct method with a Scrapy Spider for entering a zip code value "27517" automatically within the entry box on this website: Locations of Junkyards be to use a Form Request? Here is what I have right now:

import scrapy
from scrapy.http import FormRequest
from scrapy.item import Item, Field
from scrapy.http import FormRequest
from scrapy.spider import BaseSpider

class LkqSpider(scrapy.Spider):
name = "lkq" 
allowed_domains = ["http://www.lkqcorp.com/en-us/locationResults/"]
start_urls = ['http://www.lkqcorp.com/en-us/locationResults/']

def start_requests(self):
return [ FormRequest("http://www.lkqcorp.com/en-us/locationResults/",
             formdata={'dnnVariable':'27517'},
             callback=self.parse) ]

def parsel(self):
print self.status 

It doesn't do anything when run though, is Form Request mainly for completing login fields? What would be the best way to get to THIS page? (which comes up after the search for the zip 27517 and is where I would start scraping my desired information with a scrapy spider)

Answer 1

this isn't really a FormRequest as FormRequests is only a name for a POST request in scrapy, and of course it helps you fill a form, but a form is also normally a POST request.

You need some debugging console (I prefer Firebug for Firefox) to check which requests are being done, and it looks like it is a GET request and quite simple to replicate, the url would be something like this where you'll have to change the number after /fullcrit/ to the desired zip code, but you also need the lat and lng arguments, for that you could use the Google Maps API, check this answer for an example on how to get it, but to summarise just do this Request and get the location argument.