0

I am trying to parse individual product pages for information on each product and I am constructing the page link from the categories API,

def parse_categories(self, response):
    resps = json.loads(response.body)
    prods = resps.get("data").get("attributes").get(
        "main-content")[1].get("records")
    for prod in prods:
        product_link = prod.get("pdp-url")
        yield scrapy.Request(f"https://shop.lululemon.com/api{product_link}", callback=self.parse, meta={
            'pdp-url': product_link})

my problem is that my function is not returning a JSON object for my following function to parse, whenever I run it all I get is a 'NONE' value.

The start of my second function is

def parse(self, response):
    resp = json.loads(response.body)
    print(resp)

Anyone have an idea how to load the json response that I should be getting with the first scrapy request?

Halfbrowse
  • 31
  • 4
  • Hi, welcome to SO! We really need more information to help you here. Can you share more of your spider? – tomjn Apr 27 '21 at 14:04
  • @tomjn I don't have any more to share, since the parse function (which comes after the parse_categories function) is not inheriting the JSON response I need I have not written the rest of the code. – Halfbrowse Apr 27 '21 at 16:09
  • 1
    Ok - well what about the url you are trying to scrape from? At the moment your question is not easy to answer. What do you mean by "my function is not returning a JSON object "? You mean the request doesn't return JSON? We can't check that request because we don't know what it is! – tomjn Apr 27 '21 at 16:27

0 Answers0