Parse HTML Infinite Scroll

Question

I'm trying to parse the HTML of a page with infinite scrolling. I want to load all of the content so that I can parse it all. I'm using Python. Any hints?

Leonardo.Z · Accepted Answer · 2013-10-22T08:19:35.477

1

Those pages update their html with AJAX. Usually you just need to find the new AJAX requests send by browser, guess the meaning of the AJAX url parameters and fetch the data from the API.

API servers may validate the user agent, referer, cookie, oauth_token ... of the AJAX request, keep an eye on them.

edited Oct 22 '13 at 08:19

answered Oct 22 '13 at 08:09

Leonardo.Z

9,425
3
35
38

score 1 · Answer 2 · answered Oct 22 '13 at 08:10

1

the data is

either loaded in advance
or the page sends a request while you scroll

you can use httpfox to find the request and send it

answered Oct 22 '13 at 08:10

Foo Bar User

2,401
3
20
26

Parse HTML Infinite Scroll

2 Answers2

Linked