0

I think this is more of a python data structure question. I have been at this for hours and have hit nothing that works.

In terms of JSON, i want my data to be scraped in this fashion:

{"person":"john"
 "friends":[
     {"name":"sara",
      "link":"url"},
     {"name":"rick",
      "link":"url"}
    ]
}

A person can have N number of friends that are in the object. How is this written in python and used in scrapy? I have the below:

class people(scrapy.Item):
    title = scrapy.Field()
    link = scrapy.Field()
    friends = [friend()] #this is where I'm having problems

class friend(scrapy.Item):
     title = scrapy.Field()
     link = scrapy.Fied()

And my code to consume these (incomplete?) classes:

def parse_person(self,response):
    url = response.url
    item = people()
    item['title'] = response.xpath('//h1/text()').extract()
    item['link'] = response.url
    yield scrapy.Request(url, callback=self.parse_friends, meta={'item':item})

def parse_friends(self, response):
    item = response.meta['item']
    item['friends']['link'] = response.xpath('//h1/text()').extract() #also this is a question
    yield item
slowsword
  • 314
  • 2
  • 14
  • Possible duplicate of [An array field in scrapy.Item](http://stackoverflow.com/questions/29227119/an-array-field-in-scrapy-item) – Schore May 13 '16 at 20:44
  • what's the question? from this code I only see 1 friend being added to the `friends` list, also please check again your fields, as your previous structure shows something and on the code you are using something like `title`, etc. – eLRuLL May 13 '16 at 20:45

0 Answers0