Python - Web Scraping Code Error - HTTP Error 406: Not Acceptable

Question

I'm trying to learn basics of Python & I came across this exercise in a book on how to do Web Scraping. I tried to replicate the code but getting this error- "urllib.error.HTTPError: HTTP Error 406: Not Acceptable".

Is there anything wrong with the code?

I'm using Anaconda/VS Code on Windows 10.

Here's my code:

from urllib import request
from bs4 import BeautifulSoup

page_url = 'https://alansimpson.me/python/scrape_sample.html'
rawpage = request.urlopen(page_url)
soup = BeautifulSoup(rawpage, 'html5lib')
content = soup.article

links_list = []
for link in content.find_all('a'):
    try:
        url = link.get('href')
        img = link.img.get('src')
        text = link.span.text
        links_list.append({'url' : url, 'img' : img, 'text' : text})
    except AttributeError:
        pass

And this is the error I'm getting-

Traceback (most recent call last):
  File "c:\Users\srika\OneDrive\AIO_Python\scraper.py", line 6, in <module>
    rawpage = request.urlopen(page_url)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 523, in open
    response = meth(req, response)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 632, in http_response
    response = self.parent.error(
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 561, in error
    return self._call_chain(*args)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 494, in _call_chain
    result = func(*args)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 406: Not Acceptable

I tried to install 'urllib' but it is already installed. Tried to add exception 'urllib.error.HTTPError', but none of them worked.

How do I solve this? Please help!

score 2 · Answer 1 · answered Jun 29 '22 at 17:15

You need to add user-agent and it works.
If you don't put user-agent of some browser, the site thinks that you are bot and block you.

import requests
from bs4 import BeautifulSoup

page_url = 'https://alansimpson.me/python/scrape_sample.html'
headers = {
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36',
}
rawpage = requests.get(page_url,headers=headers)
soup = BeautifulSoup(rawpage.content, 'html.parser')
content = soup.article

links_list = []
for link in content.find_all('a'):
    try:
        url = link.get('href')
        img = link.img.get('src')
        text = link.span.text
        links_list.append({'url' : url, 'img' : img, 'text' : text})
    except AttributeError:
        pass
print("Total data scraped: " + str(len(links_list)))
for link in links_list:
    print(link)

OUTPUT:

Total data scraped: 13
{'url': 'http://www.sixthresearcher.com/python-3-reference-cheat-sheet-for-beginners/', 'img': '../datascience/python/basics/basics256.jpg', 'text': 'Basics'}
{'url': 'https://alansimpson.me/datascience/python/beginner/', 'img': '../datascience/python/beginner/beginner256.jpg', 'text': 'Beginner'}

Goh Kok Han · Accepted Answer · 2022-06-30T07:16:04.427

Error 406 Not Acceptable status code is an error message that means your website or web application does not support the client's request with a particular protocol.

Add the user agent in headers and try again.

This is the solution for urllib:

from urllib.request import Request, urlopen
from bs4 import BeautifulSoup

page_url = 'https://alansimpson.me/python/scrape_sample.html'
req = Request(page_url)
req.add_header('user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36')
rawpage = urlopen(req).read()
soup = BeautifulSoup(rawpage, 'html5lib')
content = soup.article

links_list = []
for link in content.find_all('a'):
    try:
        url = link.get('href')
        img = link.img.get('src')
        text = link.span.text
        links_list.append({'url' : url, 'img' : img, 'text' : text})
        print(links_list)
    except AttributeError:
        pass

Output:

[{'url': 'http://www.sixthresearcher.com/python-3-reference-cheat-sheet-for-beginners/',
  'img': '../datascience/python/basics/basics256.jpg',
  'text': 'Basics'},
 {'url': 'https://alansimpson.me/datascience/python/beginner/',
  'img': '../datascience/python/beginner/beginner256.jpg',
  'text': 'Beginner'},
 {'url': 'https://alansimpson.me/datascience/python/justbasics/',
  'img': '../datascience/python/justbasics/justbasics256.jpg',
  'text': 'Just the Basics'},
 {'url': 'https://alansimpson.me/datascience/python/cheatography/',
  'img': '../datascience/python/cheatography/cheatography256.jpg',
  'text': 'Cheatography'},
 {'url': 'https://alansimpson.me/datascience/python/dataquest/',
  'img': '../datascience/python/dataquest/dataquest256.jpg',
  'text': 'Dataquest'},
 {'url': 'https://alansimpson.me/datascience/python/essentials/',
  'img': '../datascience/python/essentials/essentials256.jpg',
  'text': 'Essentials'},
 {'url': 'https://alansimpson.me/datascience/python/memento/',
  'img': '../datascience/python/memento/memento256.jpg',
  'text': 'Memento'},
 {'url': 'https://alansimpson.me/datascience/python/syntax/',
  'img': '../datascience/python/syntax/syntax256.jpg',
  'text': 'Syntax'},
 {'url': 'https://alansimpson.me/datascience/python/classes/',
  'img': '../datascience/python/classes/classes256.jpg',
  'text': 'Classes'},
 {'url': 'https://alansimpson.me/datascience/python/dictionaries/',
  'img': '../datascience/python/dictionaries/dictionaries256.jpg',
  'text': 'Dictionaries'},
 {'url': 'https://alansimpson.me/datascience/python/functions/',
  'img': '../datascience/python/functions/functions256.jpg',
  'text': 'Functions'},
 {'url': 'https://alansimpson.me/datascience/python/ifwhile/',
  'img': '../datascience/python/ifwhile/ifwhile256.jpg',
  'text': 'If & While Loops'},
 {'url': 'https://alansimpson.me/datascience/python/lists/',
  'img': '../datascience/python/lists/lists256.jpg',
  'text': 'Lists'}]

This didn't work for me. I replaced the user-agent header with this from the browser: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.66 Safari/537.36' and the error I'm getting is - **NameError: name 'Request' is not defined**. Although the code suggested by @MeT worked for me I want to know how I can solve this by using **'urllib'**. Do I need to import some module or install some other library to make it work? — Raj K, Jun 30 '22 at 00:58
@RajK I have edited the code to import the correct module. Please consider accept this solution since this solution resolve your original problem with '**urllib**'. — Goh Kok Han, Jun 30 '22 at 07:17
@GohKokHan it's ugly to do this. His original problem is not about urlib. It is about getting 406 without using user-agent. — MeT, Jun 30 '22 at 14:38
@MeT He was following the exercise in a book, the exercise uses **urllib**, I provided the solution for '**urllib**' instead of '**requests**' so that he can follow the exercise easily. — Goh Kok Han, Jun 30 '22 at 15:19

Python - Web Scraping Code Error - HTTP Error 406: Not Acceptable

2 Answers2