7

I am trying to extract English titles from a wiki titles dump that's in a text file using regex in Python 3. The wiki dump contains titles in other languages also and some symbols. Below is my code:

with open('/Users/some/directory/title.txt', 'rb')as f:
    text=f.read()
    letters_only = re.sub(b"[^a-zA-Z]", " ", text)
    words = letters_only.lower().split() 
print(words)

But I am getting an error:

TypeError: sequence item 1: expected a bytes-like object, str found

at the line: letters_only = re.sub(b"[^a-zA-Z]", " ", text)

But, I am using b'' to make output as byte type, below is a sample of the text file:

Destroy-Oh-Boy!!
!!Que_Corra_La_Voz!!
!!_(chess)
!!_(disambiguation)
!'O!Kung
!'O!Kung_language
!'O-!khung_language
!337$P34K
!=
!?
!?!
!?Revolution!?
!?_(chess)
!A_Luchar!
!Action_Pact!
!Action_pact!
!Adios_Amigos!
!Alabadle!
!Alarma!
!Alarma!_(album)
!Alarma!_(disambiguation)
!Alarma!_(magazine)
!Alarma!_Records
!Alarma!_magazine
!Alfaro_Vive,_Carajo!
!All-Time_Quarterback!
!All-Time_Quarterback!_(EP)
!All-Time_Quarterback!_(album)
!Alla_tu!
!Amigos!
!Amigos!_(Arrested_Development_episode)
!Arriba!_La_Pachanga
!Ask_a_Mexican!
!Atame!
!Ay,_Carmela!_(film)
!Ay,_caramba!
!BANG!
!Bang!
!Bang!_TV
!Basta_Ya!
!Bastardos!
!Bastardos!_(album)
!Bastardos_en_Vivo!
!Bienvenido,_Mr._Marshall!
!Ciauetistico!
!Ciautistico!
!DOCTYPE
!Dame!_!Dame!_!Dame!
!Decapitacion!
!Dos!
!Explora!_Science_Center_and_Children's_Museum
!F
!Forward,_Russia!
!Forward_Russia!
!Ga!ne_language
!Ga!nge_language
!Gã!ne
!Gã!ne_language
!Gã!nge_language
!HERO
!Happy_Birthday_Guadaloupe!
!Happy_Birthday_Guadalupe!
!Hello_Friends

I have searched online but could not succeed. Any help will be appreciated.

Jean-François Fabre
  • 137,073
  • 23
  • 153
  • 219
Sherlock
  • 993
  • 1
  • 10
  • 22

4 Answers4

8

The problem is with the repl argument you supply, it isn't a bytes object:

letters_only = re.sub(b"[^a-zA-Z]", " ", b'Hello2World')
# TypeError: sequence item 1: expected a bytes-like object, str found

Instead, supply repl as a bytes instance b" ":

letters_only = re.sub(b"[^a-zA-Z]", b" ", b'Hello2World')
print(letters_only) 
b'Hello World'

Note: Don't prefix your literals with b and don't open the file with rb if you aren't looking for byte sequences.

Dimitris Fasarakis Hilliard
  • 150,925
  • 31
  • 268
  • 253
  • very nice, I did not know it could be done on bytes. However I'm not sure it's the way to go here. Better go text-only and drop the bytes. Well, maybe it avoids encoding problems. – Jean-François Fabre Oct 02 '16 at 15:28
  • that works , now there is no error. But i am getting "b" prefix to every extracted word. Like this **[b'you', b'and', b'then', b'some']**but i think according to you it should not be there. – Sherlock Oct 02 '16 at 15:30
  • @Jean-FrançoisFabre you were right ;-). Sherlock, just open the file without specifying `b` as @Jean suggests in his answer. `b` prefixed to the mode when opening files results in them being read as `bytes` objects, if that isn't what you need, drop it :-) – Dimitris Fasarakis Hilliard Oct 02 '16 at 15:31
  • 2
    Let me say I'm pleased of the way the things turned out: Jim is the most knowledgeable of us all, he knew about the ability to use regexes for `bytes`, although us, mere mortals, just wanted to use a text file and knew zip about that! So everyone learned something and noone got bashed (I almost deleted my post at some point) – Jean-François Fabre Oct 02 '16 at 16:02
4

You have to choose between binary and text mode.

Either you open your file as rb and then you can use re.sub(b"[^a-zA-Z]", b" ", text) (text is a bytes object)

Or you open your file as r and then you can use re.sub("[^a-zA-Z]", " ", text) (text is a str object)

The second solution is more "classical".

Jean-François Fabre
  • 137,073
  • 23
  • 153
  • 219
2

You can't use a byte string for your regex match when the replacement string isn't.
Essentially, you can't mix different objects (bytes and strings) when doing most tasks. In your code above, you are using a binary search string and a binary text, but your replacement string is a regular string. All arguments need to be of the same type, so there are 2 possible solutions to this.

Taking the above into account, your code could look like this (this will return regular string strings, not byte objects):

with open('/Users/some/directory/title.txt', 'r')as f:
    text=f.read()
    letters_only = re.sub(r"[^a-zA-Z]", " ", text)
    words = letters_only.lower().split() 
print(words)

Note that the code does use a special type of string for the regex - a raw string, prefixed with r. This means that python won't interpret escape characters such as \, which is very useful for regexes. See the docs for more details about raw strings.

Dartmouth
  • 1,069
  • 2
  • 15
  • 22
0

You can also use br'…', which is the byte analog to r'…'. The replacement must also be a byte string.

letters_only = re.sub(br'[^a-zA-Z]', b' ', text)
Suzanne Soy
  • 3,027
  • 6
  • 38
  • 56