Why does the url request not give out the response?

Question

I use the following program to get the url response, but only the following url cannot correctly give out the response. I have waited for a long time, but it still cannot finish the running. How can I solve it?

import re
from bs4 import BeautifulSoup
import whois
import urllib
import urllib.request
import requests
from datetime import datetime

try:
    url="http://zozo.jp/shop/bestpackingstore/?price=proper&p_ssy=2015&p_ssm=5&p_ssd=13&p_sey=2015&p_sem=5&p_sed=13&dstk=2"
    header = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
    response = requests.get(url, headers=header)
    print(response.text)
except:
    response = ""

1) don't silence your exceptions 2) always check the HTTP response code 3) fix your indentation — Klaus D., May 04 '23 at 05:47
It's just because your url request does time out. Use another url for testing that behaves properly. — Svenito, May 04 '23 at 05:56
When I use "http://www.google.com" as url, the code behave properly. How can I deal with that link? That link will redirect to the other link. I need to handle that. — dragontim, May 04 '23 at 06:22
When I use "response = requests.get(url, headers=header, allow_redirects=True, verify=False, timeout=10)", it still goes to exception. How can I get the correct response? — dragontim, May 04 '23 at 06:31
@dragontim a NameError exception will be raised because *requests* is not defined but you have that wrapped in try/except so you don't notice it. Anyway, just fix your User-Agent and you'll be fine (see my answer) — DarkKnight, May 04 '23 at 06:55

DarkKnight · Accepted Answer · 2023-05-04T07:01:08.177

Your User-Agent is invalid.

This code shows an improved pattern of use with a valid user agent

import requests
from bs4 import BeautifulSoup as BS

url = 'http://zozo.jp/shop/bestpackingstore'
params = {
    'price': 'proper',
    'p_ssy': 2015,
    'p_ssm': 5,
    'p_ssd': 13,
    'p_sey': 2015,
    'p_sem': 5,
    'p_sed': 13,
    'dstk': 2
}
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.4 Safari/605.1.15'
}

with requests.get(url, headers=headers, params=params) as response:
    response.raise_for_status()
    soup = BS(response.text, 'lxml')

Why does the url request not give out the response?

1 Answers1