Different Output From Same PyQuery Object

Question

I am using scrapy in order to crawl a web site.

with open('test.html', 'wb') as f:
        f.write(response.body)

With this block I am writing body to a file. When I open the file I can see many "a" tag.

When I print the same thing with print. It shows only two "a" tags

print response.body

Do you have any idea what is happening here?

These codes are both in parse method. So, response doesn't change in same parse method. — AnovaConsultancy, Dec 03 '15 at 17:13
How it can be depend on the site if scrapy cannot run javascript code? — AnovaConsultancy, Dec 03 '15 at 17:13

score 0 · Accepted Answer · edited Dec 08 '15 at 14:11

0

I have solved the problem. The crawled website has second <html> tag in a combobox.

I was using PyQuery and if there is any problem with tags in html structure PyQuery is not working.

Now I have changed my selector to xpath and now it finds all a tags in the html.

edited Dec 08 '15 at 14:11

Mr Lister

answered Dec 04 '15 at 13:31

AnovaConsultancy

1 Answers1