real time population number using web scraping

Question

I am working on a forecasting project and in this project, we need country population data for forecast something , I want to take a real time population number from a website I tried this code but it returning "retriving data...." and i'm totally confused

import bs4
import requests
from bs4 import BeautifulSoup

def population():

    r=requests.get('https://www.worldometers.info/world-population/india-population/')
    soup=bs4.BeautifulSoup(r.text,'html')
    population=soup.find('div',{'class':'maincounter-number'}).find('span').text
    return population

while True:
    print('the current population:' +str(population()))

You might need to look at this - https://stackoverflow.com/questions/21221571/python-scrapy-dynamic-web-sites/21223620 — Craicerjack, Apr 08 '20 at 08:40
Does this answer your question? [How to construct data frame from Web Scraping in Python](https://stackoverflow.com/questions/61008195/how-to-construct-data-frame-from-web-scraping-in-python) — αԋɱҽԃ αмєяιcαη, Apr 08 '20 at 08:41

score 1 · Accepted Answer · answered Apr 08 '20 at 08:46

1

import requests
from bs4 import BeautifulSoup


def main(url):
    r = requests.get(url)
    soup = BeautifulSoup(r.content, 'html.parser')
    target = soup.find(
        "div", class_="col-md-8 country-pop-description").find_all_next("strong")[1]
    print(target.text)


main("https://www.worldometers.info/world-population/india-population/")

Output:

1,376,863,120

answered Apr 08 '20 at 08:46

αԋɱҽԃ αмєяιcαη

11,825
3
17
50

its not updating it return only a fixed value. why? – Rjshrivastav Apr 08 '20 at 08:59

score 1 · Answer 2 · answered Apr 08 '20 at 09:57

If you think of using selenium to get the updated count of population, the following is something you might wanna comply with:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

link = 'https://www.worldometers.info/world-population/india-population/'

def get_population_count(wait,link):
    driver.get(link)
    while True:
        item = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR,"span[rel='india-population']"))).text
        if "retrieving data" not in item:
            break
    return item

if __name__ == '__main__':
    with webdriver.Chrome() as driver:
        wait = WebDriverWait(driver,10)
        print(get_population_count(wait,link))

real time population number using web scraping

2 Answers2