-2

I am web scraping a website: https://apps.ktrade.pk/webterminalv3/SignIn I can fetch HTML from it, but a div with class box-user-id's child elements are not appearing in my scraped html elements; while on inspection element, child elements of box-user-id's are showing.

I have tried it with multiple libraries like selenium, BeautifulSoup, mechanicalsoup etc. Please tell me how can I do it. Thanks in advance.

Jim G.
  • 15,141
  • 22
  • 103
  • 166
Naima Urooj
  • 3
  • 4

1 Answers1

0

You can try that code to login

import requests
from bs4 import BeautifulSoup

session = requests.Session()

# get auth page
auth = session.get('https://apps.ktrade.pk/webterminalv3/SignIn')

# collect csrf token
soup = BeautifulSoup(auth.text, 'html.parser')
csrf_token = soup.find('input', {'name': 'csrfPreventionSalt'})['value']

# create login request
session.post('https://apps.ktrade.pk/webterminalv3/ajax/login', data={
    'username': 'your_username', 'password': 'your_password',
    'csrfPreventionSalt': csrf_token, 'actBrandName': 'KTrade'
})
t4kq
  • 754
  • 1
  • 5
  • 11
  • Thanks. Now I am logging in and scraping data :) – Naima Urooj Jul 15 '21 at 13:35
  • @NaimaUrooj mark answer as correct please if you can – t4kq Jul 15 '21 at 13:36
  • OK. Can you please tell me from where did u find this thing? session.post('https://apps.ktrade.pk/webterminalv3/ajax/login', data={ 'username': 'your_username', 'password': 'your_password', 'csrfPreventionSalt': csrf_token, 'actBrandName': 'KTrade' }) – Naima Urooj Jul 15 '21 at 13:57
  • @NaimaUrooj in devtool. You can check on screen - https://ibb.co/QC0f7pd – t4kq Jul 15 '21 at 14:00