24

Both middleware can process Request and Response. But what is the difference?

Erez Segal
  • 3
  • 2
Zhang Jiuzhou
  • 759
  • 8
  • 22
  • 7
    @MitchWheat: It may look trivial, but it's a good question. – Blender Jul 26 '13 at 04:34
  • Doesn't appear to have any programming content: looks like a question for a search engine. – Mitch Wheat Jul 26 '13 at 04:42
  • 2
    @MitchWheat: The distinction between the two isn't obvious and isn't explained in the documentation. Casting downvotes and close votes isn't really helpful if you don't understand the framework involved. – Blender Jul 26 '13 at 04:43
  • 1
    I just wanted to say that both the question and answer helped me greatly. I wished it had remained open, so more helpful answers could be posted. – Danial Jul 24 '22 at 04:25

1 Answers1

29

While they have almost identical interfaces, they serve different purposes:

  • Downloader middlewares modify requests and responses or generate requests in response to responses. They don't directly interact with spiders. Some examples are middlewares that implement cookies, caching, proxies, redirects, setting user-agent headers, etc. They just add functionality to the downloader system.

  • Spider middlewares modify things that pass in and out of spiders, like requests, items, exceptions, and start_requests. They do share some basic functionality with downloader middlewares, but they can't generate requests in response to responses. They stand between the spiders and the downloader. One example is filtering out responses with bad HTTP status codes.

Some middlewares can function as either a downloader middleware or a spider middleware, but they're often trivial and will be forced into one category or the other once you add more complex functionality.

Blender
  • 289,723
  • 53
  • 439
  • 496
  • 2
    Thanks for the solution that helps me a lot. Can you please elaborate the role of DOWNLOADER_MIDDLEWARES here.@Blender – Jack Daniel Oct 08 '15 at 12:53