16

In Python 3, the following code obtains the HTML source for a webpage.

import urllib.request
url = "https://docs.python.org/3.4/howto/urllib2.html"
response = urllib.request.urlopen(url)

response.read()

How can I add the following custom header to the request when using urllib.request?

headers = { 'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; Win64; x64)' }
rovyko
  • 4,068
  • 5
  • 32
  • 44

3 Answers3

37

The request headers can be customized by first creating a request object then supplying it to urlopen.

import urllib.request
url = "https://docs.python.org/3.4/howto/urllib2.html"
hdr = { 'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; Win64; x64)' }

req = urllib.request.Request(url, headers=hdr)
response = urllib.request.urlopen(req)
response.read()

Source: Python 3.4 Documentation

rovyko
  • 4,068
  • 5
  • 32
  • 44
6
import urllib.request

opener = urllib.request.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
urllib.request.install_opener(opener)
response = urllib.request.urlopen("url")
response.read()

Should you wish to learn about the details you can refer to the python documentation: https://docs.python.org/3/library/urllib.request.html

Lost Crotchet
  • 1,270
  • 12
  • 17
2
#Using urllib.request, with urlopen, allows to open the specified URL.
#Headers can be included inside the urlopen along with the url.

from urllib.request import urlopen
url = "https://docs.python.org/3.4/howto/urllib2.html"
header = { 'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; Win64; x64)' }
response = urlopen(url, headers=header)
response.read()
Natasha
  • 21
  • 3
  • Please don't post only code as an answer, but also provide an explanation of what your code does and how it solves the problem of the question. Answers with an explanation are usually more helpful and of better quality, and are more likely to attract upvotes. – Ran Marciano Feb 06 '21 at 19:40