5

I am trying to store scrapyd items in a JSON file. Actually by default it store items in a json file but like this: File_1:

{item1}
{item2}
....

And if i run my spider by scrapy crawl spidername -o fileName -t json

it will store item like this: File_2:

[{item1},
{item2},
....]

If i try to copy paste contents of File_1 into jsonParser i got error saying expected [ but File_2 works fine. Can anyone tell me how to store items in proper JSON format using scrapyd

3 Answers3

0

Actually its a json lines format , json is not good for very bid data. thats why scrapy separate each item in a comma separated format. you can add [ & ] in the start and end of file it will be json then,

akhter wahab
  • 4,045
  • 1
  • 25
  • 47
  • ahhh i forgot that my mistake you needs to add comma on each line end .. or you can read each line as dictionary as well – akhter wahab May 30 '13 at 13:14
0

Try to save the items directly into the database rather then storing them on the file....then storing the items from file to database... or try to read line by line from that JSON file and then do what ever you want instead of reading the whole file at a time...

Vaibhav Jain
  • 5,287
  • 10
  • 54
  • 114
-1

Add to your settings.py:

FEED_FORMAT='json'
Lucas Pottersky
  • 1,844
  • 6
  • 22
  • 38