171

I am trying to learn how to automatically fetch urls from a page. In the following code I am trying to get the title of the webpage:

import urllib.request
import re

url = "http://www.google.com"
regex = r'<title>(,+?)</title>'
pattern  = re.compile(regex)

with urllib.request.urlopen(url) as response:
   html = response.read()

title = re.findall(pattern, html)
print(title)

And I get this unexpected error:

Traceback (most recent call last):
  File "path\to\file\Crawler.py", line 11, in <module>
    title = re.findall(pattern, html)
  File "C:\Python33\lib\re.py", line 201, in findall
    return _compile(pattern, flags).findall(string)
TypeError: can't use a string pattern on a bytes-like object

What am I doing wrong?

Aran-Fey
  • 39,665
  • 11
  • 104
  • 149
Inspired_Blue
  • 2,308
  • 3
  • 15
  • 21
  • 1
    possible duplicate of [Convert bytes to a Python string](http://stackoverflow.com/questions/606191/convert-bytes-to-a-python-string) – gnat Jun 24 '15 at 09:54

3 Answers3

250

You want to convert html (a byte-like object) into a string using .decode, e.g. html = response.read().decode('utf-8').

See Convert bytes to a Python String

Community
  • 1
  • 1
rocky
  • 7,226
  • 3
  • 33
  • 74
  • 2
    This solved the error `TypeError: cannot use a string pattern on a bytes-like object` but then I got errors like `UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb2 in position 1: invalid start byte`. I fixed it by using `.decode("utf-8", "ignore")`: https://stackoverflow.com/questions/62170614/python-unicodedecodeerror-utf-8-codec-cant-decode-byte-0x80-in-position-0/62170725#62170725 – baptx Aug 17 '20 at 22:18
  • "ignore" ignores. If that is what you want, then all is good. However sometimes this kind of problem belies a deeper problem, e.g. that the thing you want to decode really isn't decodable or meant to be, e.g. compressed or encypted text. Or it might need some other encoding like `utf-16` . Caveat emptor. – rocky Aug 17 '20 at 23:50
54

The problem is that your regex is a string, but html is bytes:

>>> type(html)
<class 'bytes'>

Since python doesn't know how those bytes are encoded, it throws an exception when you try to use a string regex on them.

You can either decode the bytes to a string:

html = html.decode('ISO-8859-1')  # encoding may vary!
title = re.findall(pattern, html)  # no more error

Or use a bytes regex:

regex = rb'<title>(,+?)</title>'
#        ^

In this particular context, you can get the encoding from the response headers:

with urllib.request.urlopen(url) as response:
    encoding = response.info().get_param('charset', 'utf8')
    html = response.read().decode(encoding)

See the urlopen documentation for more details.

Aran-Fey
  • 39,665
  • 11
  • 104
  • 149
0

Based upon last one, this was smimple to do when pdf read was done .

text = text.decode('ISO-8859-1')

Thanks @Aran-fey

Jha Ayush
  • 67
  • 8