How to get an HTML file using Python?

Question

I am not very familiar with Python. I am trying to extract the artist names (for a start :)) from the following page: http://www.infolanka.com/miyuru_gee/art/art.html.

How do I retrieve the page? My two main concerns are; what functions to use and how to filter out useless links from the page?

score 24 · Accepted Answer · edited Apr 07 '14 at 16:58

24

Example using urlib and lxml.html:

import urllib
from lxml import html

url = "http://www.infolanka.com/miyuru_gee/art/art.html"
page = html.fromstring(urllib.urlopen(url).read())

for link in page.xpath("//a"):
    print "Name", link.text, "URL", link.get("href")

output >>
    [('Aathma Liyanage', 'athma.html'),
     ('Abewardhana Balasuriya', 'abewardhana.html'),
     ('Aelian Thilakeratne', 'aelian_thi.html'),
     ('Ahamed Mohideen', 'ahamed.html'),
    ]

edited Apr 07 '14 at 16:58

Siegfried Gevatter

3,624
3
18
13

answered Dec 20 '10 at 17:21

Vince Spicer

4,325
3
21
11

11

in python 3 you should import urllib.request and use urllib.request.urlopen function. see http://docs.python.org/3.2/library/urllib.request.html#urllib.request.urlopen – sebast26 Feb 07 '13 at 11:11
2

urllib is outdated in this day and age and should be using the requests library or something that handles the modern day issues. – User Mar 29 '14 at 00:53

score 7 · Answer 2 · answered Dec 20 '10 at 15:17

I think "eyquem" way would be my choice too, but I like to use httplib2 instead of urllib. urllib2 is too low level lib for this work.

import httplib2, re

pat = re.compile('<DT><a href="[^"]+">(.+?)</a>')
http = httplib2.Http()
headers, body = http.request("http://www.infolanka.com/miyuru_gee/art/art.html")

li = pat.findall(body)
print li

score 6 · Answer 3 · answered Feb 01 '14 at 21:19

Check this my friend

import urllib.request

import re

pat = re.compile('<DT><a href="[^"]+">(.+?)</a>')

url = 'http://www.infolanka.com/miyuru_gee/art/art.html'

sock = urllib.request.urlopen(url).read().decode("utf-8")

li = pat.findall(sock)

print(li)

score 6 · Answer 4 · answered Dec 20 '10 at 12:22

6

Use urllib2 to get the page.
Use BeautifulSoup to parse the HTML (the page) and get what you want!

answered Dec 20 '10 at 12:22

user225312

126,773
69
172
181

score 4 · Answer 5 · answered Dec 20 '10 at 14:40

4

Or go straight forward:

import urllib

import re
pat = re.compile('<DT><a href="[^"]+">(.+?)</a>')

url = 'http://www.infolanka.com/miyuru_gee/art/art.html'
sock = urllib.urlopen(url)
li = pat.findall(sock.read())
sock.close()

print li

answered Dec 20 '10 at 14:40

eyquem

26,771
7
38
46

score 1 · Answer 6 · edited May 23 '17 at 12:31

1

And respect robots.txt and throttle your requests :)

(Apparently urllib2 does already according to this helpful SO post).

edited May 23 '17 at 12:31

Community

1
1

answered Dec 20 '10 at 12:25

Tim Barrass

4,813
2
29
55

Is it illegal to not do so? ^.^ – Zippo Mar 28 '11 at 18:33
No, unless I misinterpret the multiple negatives there. :) – Tim Barrass Mar 28 '11 at 21:14

score 1 · Answer 7 · answered Feb 27 '21 at 12:20

A more concise answer adapted to Python 3.x and using requests and bs4. There are two questions though in the original question. First, how to obtain the html:

import requests
html = requests.get("http://www.infolanka.com/miyuru_gee/art/art.html").content

Second, how to obtain artists name list:

import bs4
soup = bs4.BeautifulSoup(html)
artist_list = []
for i in soup.find_all("a"):
    if i.parent.name == "dt":
        artist_list.append(i.contents[0])
print(artist_list)

Output:

['Aathma Liyanage',
 'Abewardhana Balasuriya',
 'Aelian Thilakeratne',
 'Ahamed Mohideen',
 'Ajantha Nakandala',
 'Ajith Ambalangoda',
 'Ajith Ariayaratne',
 'Ajith Muthukumarana',
 'Ajith Paranawithana',
...]

score 0 · Answer 8 · answered Nov 07 '16 at 09:05

Basically, there's a function call:

render_template()

You can easly return single page or list of pages with it and it reads all files automaticaly from a your_workspace\templates .

Example:

/root_dir /templates /index1.html, /index2.html /other_dir /

routes.py

@app.route('/') def root_dir(): return render_template('index1.html')

@app.route(/<username>) def root_dir_with_params(username): retun render_template('index2.html', user=username)

index1.html - without params

<html> <body> <h1>Hello guest!</h1> <button id="getData">Get Data!</button> </body> </html>

index2.html - with params

<html> <body>  {% if name %} <h1 style="color: red;">Hello {{ user }}!</h1> {% else %} <h1>Hello guest.</1> <button id="getData">Get Data!</button> </body> </html>

How to get an HTML file using Python?

8 Answers8

Linked