The site administrator has decided that the site should pretend to not exist to clients that do not share their User-Agent in their headers:
>>> import requests
>>> url = 'http://www.transfermarkt.com/'
>>> requests.get(url).raise_for_status()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/site-packages/requests/models.py", line 831, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found
Breaks as you've found out. Set a user agent:
>>> headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0)'}
>>> requests.get(url, headers=headers).raise_for_status()
>>>
and you're good.
It seems like the site admin doesn't want you to do this, so perhaps you could ask for permission or ask if there's a preferred way to get the content, but not having a user-agent set was the technical reason.