Stack Overflow
Stack Overflow

Questions tagged [python-requests-html]

Requests-HTML is a Python HTTP library built around the requests API, adding support for parsing HTML (with optional headless-browser support to render JavaScript).

Official web site

Requests is a web scraping library written in Python under the MIT license.

This library intends to make parsing HTML (e.g. scraping the web) simple, building on top of the library for the HTTP layer. It supports XPath and CSS selectors, User Agent spoofing, and optional headless-browser support to execute JavaScript scripts on the page.

535 questions
-1
votes
1 answer

How to follow page redirects using requests

I have this simple code: import requests r = requests.get('https://yahoo.com') print(r.url) Which after executing, prints: https://uk.yahoo.com/?p=us I want to see: How many redirects have happened before arriving on https://uk.yahoo.com/?p=us…
user9371654
  • 2,160
  • 16
  • 45
  • 78
-1
votes
2 answers

Python - requests_html screen scraping

I’m trying to log in to a pretty complex (to my beginner’s eye) website and make a reservation. Did not know a single python statement before starting the project. After many starts and stops have successfully logged in using…
steve
  • 77
  • 1
  • 8
-2
votes
1 answer

Python script not returning any results while web scraping

I am looking to scrape a list of URLs -- I want to visit each one & then return all IMG links contained within each HREF on the page (in essence, visit each link and return the image address of the player headshot on each player profile). I have a…
Anthony Madle
  • 371
  • 1
  • 10
-2
votes
1 answer

Execute JavaScript function using Python requests_html

I am trying to execute a JavaScript function (run with a button click) within a session using Python's requests_html I understand the regular requests library does not have JavaScript support so I am trying to use requests_html instead. Here's what…
Omaro_IB
  • 379
  • 1
  • 3
  • 13
-2
votes
1 answer

soup.find_all returns empty list

I was trying to do some data scraping from booking.com for prices. But it just keeps on returning an empty list. If anyone can explain me what is happening i would be really thankful to them. Here is the website from which I am trying to scrape…
Doulat Ghuman
  • 1
  • 2
-2
votes
2 answers

Find by style (color) with requests_html

I have to use requests_html for JavaScript content. Code: 38-G1 I want to find all names (38-G1 in this case)…
Dixy
  • 3
  • 1
-2
votes
3 answers

AttributeError: 'NoneType' object has no attribute 'text' , I don't understand how to fix it

I'm trying to read a file with python and get each line as a parameter for a function. I've got a AttributeError: 'NoneType' object has no attribute 'text' error and I don't understand how to fix it. from bs4 import BeautifulSoup from requests_html…
Dany Frum
  • 1
  • 1
  • 1
-2
votes
1 answer

How to find all elements with class name containing a specific string using HTML-requests library?

I am on this page on Tokyo Olympic Website I would like to get all the elements that has class name beginning with a specific string. For instance I want to get all elements that begin with 'col-sm-'. html.r.find('.col-sm-6') gives me all elements…
Sam
  • 21
  • 7
-3
votes
1 answer

What parameters need to be put in the code in order to execute correctly the requests.post function?

What parameters need to be put in regards to this site (www.pyszne.pl) so that the requests function can be executed properly? I need to have a url which leads to the restaurants available under a specific postcode. here is my code: import…
Sebastian
  • 31
  • 6
-3
votes
3 answers

how to convert str to int? int() and float() won't work

I got some data from requests-html module and want to do calculation with it but won't work. It is a 'str' type and I tried converting it to int by using int() neither do float() work. Thanks in advance.
hadi tedi
  • 515
  • 1
  • 4
  • 12
1 2 3
…
35
36