Open online txt file using python codecs.open

Question

I am trying to open an online txt file using codecs.open. The code I have now is:

url = r'https://www.sec.gov/Archives/edgar/data/20/0000893220-96-000500.txt'
soup = BeautifulSoup(codecs.open(url, 'r',encoding='utf-8'), "lxml")

However, Python keeps reminding OSError:

OSError: [Errno 22] Invalid argument: 'https://www.sec.gov/Archives/edgar/data/20/0000893220-96-000500.txt'

I tried to replace "/" with "\". It still does not work. Is there any way to solve it? Since I have more than thousands of links to open, I do not quite want to download the online text files into my local drive.

I will appreciate it very much if someone can help here.

Thanks!

Thanks, @larsks! I see. Since I want to use beautifulsoup to parse the html written in txt format. Is there any way to do it? — dara wong, Apr 23 '19 at 04:21
With a `http.client.HTTPConnection` you can receive a `HTTPResponse` which can be wrapped (as file-like object) in a `codecs.EncodedFile`. — Michael Butscher, Apr 23 '19 at 05:07

score 2 · Answer 1 · answered Apr 23 '19 at 11:17

Is it something like this you're thinking of?

`from urllib.request import urlopen
url = urlopen('https://www.sec.gov/Archives/edgar/data/20/0000893220-96- 000500.txt')
 html = url.read().decode('utf-8')
 file = open('yourfile.txt', 'r')
 file.read(html)
 file.close`

Open online txt file using python codecs.open

1 Answers1