0

I tried the AuthorSpider in scrapy docs, but it doesn't return the author page for me, it even doesn't go to parse_author() method, I want to crawl a website with several internal links, then follow the link to crawl the article, I tried all the scrapy code to use response.follow() with several parse method, but it only called the first parse() method, the parse_author() method never get called.

vezunchik
  • 3,669
  • 3
  • 16
  • 25
rabbit
  • 1
  • 2
  • Are you using the exactly same code from the docs (https://docs.scrapy.org/en/latest/intro/tutorial.html#more-examples-and-patterns)? Looks like you are trying to crawl a different website, in this case the sample code will not work. – Luiz Rodrigues da Silva Apr 18 '19 at 19:43
  • @LuizRodriguesdaSilva Thank you for answering. I used the same code to try the example, it just crawled the url for me without going further to get the author info. – rabbit Apr 19 '19 at 20:42
  • Please, see https://stackoverflow.com/help/mcve – Gallaecio Apr 22 '19 at 13:32

0 Answers0