How extract URL from web page

Question

I need a way to extract url from the list at this web page https://iota-nodes.net/ using Python. I tried BeautifulSoup but without success. My code is:

from bs4 import BeautifulSoup, SoupStrainer
import requests

url = "https://iota-nodes.net/"

page = requests.get(url)    
data = page.text
soup = BeautifulSoup(data)

for link in soup.find_all('a'):
   print(link.get('href'))

score 0 · Accepted Answer · answered Jul 22 '19 at 09:23

No need for BeautifulSoup, as the data is coming from an AJAX request. Something like this should work:

import requests

response = requests.get('https://api.iota-nodes.net/')
data = response.json()

hostnames = [node['hostname'] for node in data]

Note that the data comes from an API endpoint being https://api.iota-nodes.net/.

How extract URL from web page

1 Answers1