Response.url and refer url scrapy

Question

2020-11-09 12:13:08 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://example.com/books/adventure/book1/index.html> (referer: example.com/books/adventure/index.html)

If anyone is familiar with scarpy, you know that https://example.com/books/adventure/book1/index.html is called response.url. However, I want to get the refer link example.com/books/adventure/index.html, does anyone know what it's called.

Does this answer your question? [Scrapy:How to print request referrer](https://stackoverflow.com/questions/12054958/scrapyhow-to-print-request-referrer) — drum, Nov 09 '20 at 03:38

score 0 · Accepted Answer · answered Nov 09 '20 at 17:29

You need to create the referer in your header.

Ideally it needs to be created by you, i.e you will already have it and you don't need to get it from the response.

eg.

    headers={'Referer':'example.com/books/adventure/index.html'}

Hope that helps?

Response.url and refer url scrapy

1 Answers1