python scrape links keyword

Question

I'm new to python and I need help for scraping all links with a certain keyword. The problem is that I'm getting the following error:

if "air-max" in link["href"]: ^ IndentationError: expected an indented block.

Here is my code

import requests
import time
from bs4 import BeautifulSoup

headers = {"Content-Type": "application/x-www-form-urlencoded; 
charset=UTF-8","X-Requested-With": "XMLHttpRequest","User-Agent": 
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, 
like Gecko) Chrome/56.0.2924.87 Safari/537.36"}

for i in range(0,4):   
url = "https://www.aw-lab.com/shop/uomo/scarpe?p={}".format(i)
r = requests.get(url)
soup = BeautifulSoup(r.content, "html.parser")

all_links = soup.find_all("a")
for link in all_links:
if link.has_key('href'):
if "air-max" in link["href"]:
    print(link["href"])

You have to indent your code. Make sure you follow the style, check if you have spaces or tabs everywhere in correct places. — Konrad, Oct 16 '17 at 16:34

lit · Answer 1 · 2017-10-16T23:23:35.450

You need another indentation level after link.has_key('href'):. Also, be consistent; always use spaces (preferred) or always use tabs. This may not always be true, but, in general, if there is a COLON : at the end of a line, then the next line should be indented one level further.

for i in range(0,4):   
    url = "https://www.aw-lab.com/shop/uomo/scarpe?p={}".format(i)
    r = requests.get(url)
    soup = BeautifulSoup(r.content, "html.parser")

    all_links = soup.find_all("a")
    for link in all_links:
        if link.has_key('href'):
            if "air-max" in link["href"]:
                print(link["href"])

Ashok Philip · Accepted Answer · 2017-10-18T17:47:18.043

Please use an IDE like spyder IDE or jupyter notebook for development.

import requests
import time
from bs4 import BeautifulSoup

headers = {"Content-Type": "application/x-www-form-urlencoded; 
charset=UTF-8","X-Requested-With": "XMLHttpRequest","User-Agent": 
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, 
like Gecko) Chrome/56.0.2924.87 Safari/537.36"}

for i in range(0,4):
    url = "https://www.aw-lab.com/shop/uomo/scarpe?p={}".format(i)
    r = requests.get(url)
    soup = BeautifulSoup(r.content, "html.parser")

all_links = soup.find_all("a")
for link in all_links:
    if link.has_key('href'):
        if "air-max" in link["href"]:
            print(link["href"])

python scrape links keyword

2 Answers2