0

The webpage I am trying to scrape can only be seen after login so using a direct url won't work. I need to scrape data while I am logged in using my chrome browser. Then I need to get the value of the the element from

I have tried using the following code.

import requests
from selenium import webdriver
from bs4 import BeautifulSoup as bs
import pandas as pd
from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(ChromeDriverManager().install())

lastdatadate=[]
lastprocesseddate=[]

source = requests.get('webpage.com').text
content = driver.page_source
soup = bs(content, 'lxml')
#print(soup.prettify())
price = soup.find('span', attrs={'id':'calculatedMinRate'})

print(price.text)
Dev Babbar
  • 1
  • Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. – Community Jan 06 '23 at 10:58

1 Answers1

0

You could still perform a login on the opened webdriver and fill in the input fields, as explained here: How to locate and insert a value in a text box (input) using Python Selenium?

Steps:

  1. Fill in the input fields
  2. Find the submit button and trigger a click event
  3. Afterwards add a sleep command, few seconds should be enough
  4. Afterwards you should be able to get the data.
Yusuf Ipek
  • 166
  • 1
  • 11