Is it possible to pass a variable from start_requests() to parse() for each individual request?

Question

I'm using a loop to generate my requests inside start_request() and I'd like to pass the index to parse() so it can store it in the item. However when I use self.i the output has the i max value (last loop turn) for every items. I can use response.url.re('regex to extract the index') but I wonder if there is a clean way to pass a variable from start_requests to parse.

Answer 1

You can use scrapy.Request meta attribute:

import scrapy

class MySpider(scrapy.Spider):
    name = 'myspider'

    def start_requests(self):
        urls = [...]
        for index, url in enumerate(urls):
            yield scrapy.Request(url, meta={'index':index})

    def parse(self, response):
        print(response.url)
        print(response.meta['index'])

Answer 2

You can pass cb_kwargs argument to scrapy.Request()

https://docs.scrapy.org/en/latest/topics/request-response.html#scrapy.http.Request.cb_kwargs

import scrapy

class MySpider(scrapy.Spider):
    name = 'myspider'

    def start_requests(self):
        urls = [...]
        for index, url in enumerate(urls):
            yield scrapy.Request(url, callback=self.parse, cb_kwargs={'index':index})

    def parse(self, response, index):
        pass