Extracting web elements using Pyquery, Requests and Gadget selector

Question

I am able to extract table values from this website with the following code.

from pyquery import PyQuery as pq
import requests

url = "https://finviz.com/screener.ashx"
content = requests.get(url).content
doc = pq(content)
Tickers = doc(".screener-link-primary").text()

print(Tickers)

But I am able to extract only the first 20 values. There is a 'next' button at the end of the page which has the link to the next set of values.

How can I extract this link automatically, fetch the new page and extract the new set of values and append to my existing list?

Answer 1

You can iterate through all pages like:

counter = 1

while True:
    url = "https://finviz.com/screener.ashx?v=111&r=%d" % counter
    content = requests.get(url).content
    counter += 20

Note that for the first page r parameter (which I guess stands for starting entry index) will be 1 for the second - 21, for the third -41... So I used + 20 increment for counter

You should also add break for the moment when the last page reached. Usually one make a check whether new data to scrape available and if not - break