Scraping webpage using BeautifulSoup

Question

I am attempting to scrape this site: https://www.senate.gov/general/contact_information/senators_cfm.cfm

My Code:

import requests
from bs4 import BeautifulSoup

URL = 'https://www.senate.gov/general/contact_information/senators_cfm.cfm'
page = requests.get(URL)

soup = BeautifulSoup(page.content, 'html.parser')

print(soup)

The issue is that it's not actually going to the site. The HTML that I get in my soup var is not at all what the HTML is in the correct webpage.

How do you know that... try print `print(page.url)` and see if it's the same URL or not — Anwarvic, May 29 '20 at 14:07

score 1 · Accepted Answer · answered May 29 '20 at 14:13

This worked for me

headers = {
        'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36',
    }
r = requests.get(URL,headers=headers)

Found the info here - https://towardsdatascience.com/5-strategies-to-write-unblock-able-web-scrapers-in-python-5e40c147bdaf

score 0 · Answer 2 · answered May 29 '20 at 14:16

DUPLICATE HTTP 503 Error while using python requests module

try that:

import requests
from bs4 import BeautifulSoup

URL = 'https://www.senate.gov/general/contact_information/senators_cfm.cfm'

page = requests.post(URL, headers=headers)

soup = BeautifulSoup(page.content, 'html.parser')
print(soup)

Scraping webpage using BeautifulSoup

2 Answers2