0

I'm trying to use the mechanize .open method on a URL so I can eventually login and scrape this website. I've tried using this code on other websites and they work, but for some reason I'm getting the error only for this site: "mechanize._response.httperror_seek_wrapper: HTTP Error 404: Not Found". I'm new to Python but here's my code:

import mechanize
from bs4 import BeautifulSoup
import html2text
import http.cookiejar

# Browser
br = mechanize.Browser()

# Cookie Jar
cj = http.cookiejar.CookieJar()
br.set_cookiejar(cj)

br.set_handle_equiv(True)
br.set_handle_gzip(True)
br.set_handle_redirect(True)
br.set_handle_referer(True)
br.set_handle_robots(False)
br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)
br.addheaders = [('User-agent', 'Chrome')]

br.open('https://onlinebusiness.icbc.com/webdeas-ui/login;type=driver')

Thanks a lot in advance.

AlwaysStuck
  • 1
  • some portals may use more complex systems to recognize bots/hackers/spamers and it may block you. And sometimes it can be impossible to solve it. – furas Jun 18 '21 at 07:18
  • when I check page in web browser (DevTools in Firefox/Chrome, tab `Netwoork`) then it send correct page but with status `404` - so this page may use status `404 to block bots/script :) – furas Jun 18 '21 at 07:38

0 Answers0