Portia spider not crawling items

Asked Nov 02 '16 at 13:25

Active Nov 04 '16 at 06:00

Viewed 293 times

I have created a spider using Portia UI and I have deployed and scheduled in one of my virtual machine using scrapyd. Spider ran fine and scraped website contents.

But when I try to deploy and schedule the same spider in another similar virtual machine using scrapyd, the spider ran fine but not crawling any content.

Both machines have similar configuration, setups, packages and versions.

What could be the possible issue?

Edit
I have done the following
- Installed all Portia packages in my machine using docker
- Created a spider(say myspider)
- Deployed and scheduled that spider using scrapyd
- Got extracted content from spider run
- Cloned the machine and added to another network with different ISP
- Deployed the same spider(myspider)
- Spider ran fine but website content didn't extract
- I have created a new spider with some different URL and that spider is crawling the website content fine

edited Nov 04 '16 at 06:00

asked Nov 02 '16 at 13:25

Prabhakar

1,138
2
14
30

Are you using any proxy setting while doing request ? – Ahsan aslam Nov 02 '16 at 13:28
What's different about the virtual machines? Are the OS versions different for each of them? – Nov 02 '16 at 21:30
@SheikhJames - I am not using any proxy but using different ISPs and I am getting 200 response for all the URLs. I am wondering why the content alone not scraping. – Prabhakar Nov 03 '16 at 09:29
@kiran.koduru - No difference in machines, setups, packages and versions. – Prabhakar Nov 03 '16 at 09:30
@Prabhakar what about vagrant? Are you running it through Vagrant? – Nov 03 '16 at 14:21
@kiran.koduru - I am using docker. Edited my question. – Prabhakar Nov 04 '16 at 06:00

Portia spider not crawling items

0 Answers0