Stack Overflow
Stack Overflow

Questions tagged [python-requests-html]

Requests-HTML is a Python HTTP library built around the requests API, adding support for parsing HTML (with optional headless-browser support to render JavaScript).

Official web site

Requests is a web scraping library written in Python under the MIT license.

This library intends to make parsing HTML (e.g. scraping the web) simple, building on top of the library for the HTTP layer. It supports XPath and CSS selectors, User Agent spoofing, and optional headless-browser support to execute JavaScript scripts on the page.

535 questions
2
votes
3 answers

Python - Requests Library - How to ensure HTTPS requests

This is probably a dumb question, but I just want to make sure with the below. I am currently using the requests library in python. I am using this to call an external API hosted on Azure cloud. If I use the requests library from a virtual machine,…
adan11
  • 647
  • 1
  • 7
  • 24
2
votes
2 answers

Getting the element on page source but unable to locate using xpath

I am trying to find the element by using xpath but they are unable to locate. While when I am getting the page source using selenium they have the element and Also I have checked but the element are not in Iframe. Here is my code: from requests_html…
Haris Ahmad
  • 39
  • 5
2
votes
0 answers

requests-html failed to get expected elements

I'm new to requests-html and just installed several days ago. when followed the Tutorial : from requests_html import HTMLSession session = HTMLSession() r = session.get('https://python.org/') about = r.html.find('#about',…
fengsanyunyan
  • 21
  • 2
2
votes
0 answers

requests-html - Unable to render() html with cookies

I need to render a page for scraping with requests-html in python. No matter what I have tried, I've been unsuccessful with getting the page rendered. It stays as standard HTML. I did find this site, which made me believe that i'm unable to render…
BlueBell
  • 113
  • 8
2
votes
1 answer

Is there a way to render a HTML page without using a browser and then scrape it's content?

I need to extract some text from a webpage but the webpage is dynamically built(plugin). i.e I need to include a javascript SDK