How to crawl a list of url in python web crawler?

Asked Jul 09 '18 at 23:57

Active Jul 10 '18 at 10:47

Viewed 717 times

I have a list of url like this:

url = ['url_1','url_2', 'url_3']

and there are 300 elements in the list.

As their HTML structure is similar, I have written a function to crawl it and extract the information that I need:

def get_department_and_units_hours(url):
    res = requests.get(url)
    soup = BeautifulSoup(res.content, "html.parser")
    data = [item.string for item in soup.find_all('td')]
    data = data[1:]
    return data

Then, I iterate through the list and append the data into an array

department_and_units_hours = []
for item in url:
    department_and_units_hours.append(get_department_and_units_hours(item))

print(department_and_units_hours)

When I run it, it is no response. Just nothing happened. It can't print out the content of the array.

Error message:

I want to ask why the problem occurs and how to fix it? I really have no idea about it.

edited Jul 10 '18 at 10:47

asked Jul 09 '18 at 23:57

Kelvinyu1117

print the data returned from `get_department_and_units_hours()`? Are you sure this is working? – RoadRunner Jul 10 '18 at 00:39
It should be ```get_department_and_units_hours(item)```, where item is the element in the url list. sorry for the typo. – Kelvinyu1117 Jul 10 '18 at 05:29
are you sure 'get_department_and_units_hours` is returning what it should? – Anuvrat Parashar Jul 10 '18 at 07:36
also, could you share the url you are trying to scrape? – Anuvrat Parashar Jul 10 '18 at 07:37
Yes, I sure the function return the data correctly. I can ``print(get_department_and_units_hours(url[i]))``, where i is the index of the element, and it works very well – Kelvinyu1117 Jul 10 '18 at 10:40
I think the for loop is the problem, when I stop the program, it shows me the error, I have added it in the question. – Kelvinyu1117 Jul 10 '18 at 10:47

How to crawl a list of url in python web crawler?

0 Answers0