Extracting Results from "Result-Page" With Selenium in Python

Question

Unfortunately I have a little problem with the realization of my Python program. At one point I can't get any further. The program should do the following:

Perform an automatic search for a specific keyword on the search engine "www.startpage.com".
Then the page with the results should be read out (and here is the problem).
The program should now count how often a certain word appears on the page with the search results.

The problem here is that I can't get the source code from the search results page. I only get the source code of the start page Does anyone know a solution?

Thanks in advance.

So far my program looks like this:

import selenium.webdriver as webdriver

def get_results(search_term):

    #this is the site, where I want to do the search
    url="https://www.startpage.com"
    browser = webdriver.Firefox()
    browser.get(url)

    search_box = browser.find_element_by_id("q")
    #search in the search box after the search term
    search_box.send_keys(search_term)
    search_box.submit()

    #print(browser.page_source) would give the result of the startpage (not the result page)

    sub="dog"
    print(source_code.count("dog"))
    #counts zero times because it searchs for "dog" at the startpage

get_results("dog")

You need to understand how REST web pages work. Submitting the search term loads a new page, but your code never does that. (Also, `source_code` is not a defined variable. Please [edit] to post actually working code, or perhaps simply delete this question.) — tripleee, May 11 '20 at 08:41
As mentioned other contributor.Its require some time to load the page.so before you capture `browser.page_source` just provide some wait.you can do that using `time.sleep(5)` — KunduK, May 11 '20 at 09:45

score 1 · Accepted Answer · edited Oct 21 '20 at 20:37

You can do it this way: Just make a loop in which you always add an elment to a list(can be a number or a letter, for example) when this term is found.

To do this you have to save the source code in a variable and then simply search for the term in it. When it is found you simply add a number to a list with .append() and then at the end you check the length of the list with len(list).

Extracting Results from "Result-Page" With Selenium in Python

1 Answers1