Replace special characters in a string in Python

Question

I am using urllib to get a string of html from a website and need to put each word in the html document into a list.

Here is the code I have so far. I keep getting an error. I have also copied the error below.

import urllib.request

url = input("Please enter a URL: ")

z=urllib.request.urlopen(url)
z=str(z.read())
removeSpecialChars = str.replace("!@#$%^&*()[]{};:,./<>?\|`~-=_+", " ")

words = removeSpecialChars.split()

print ("Words list: ", words[0:20])

Here is the error.

Please enter a URL: http://simleyfootball.com
Traceback (most recent call last):
  File "C:\Users\jeremy.KLUG\My Documents\LiClipse Workspace\Python Project 2\Module2.py", line 7, in <module>
    removeSpecialChars = str.replace("!@#$%^&*()[]{};:,./<>?\|`~-=_+", " ")
TypeError: replace() takes at least 2 arguments (1 given)

score 123 · Answer 1 · edited Jan 02 '23 at 14:55

123

One way is to use re.sub, that's my preferred way.

import re
my_str = "hey th~!ere"
my_new_string = re.sub('[^a-zA-Z0-9 \n\.]', '', my_str)
print my_new_string

Output:

hey there

Another way is to use re.escape:

import string
import re

my_str = "hey th~!ere"

chars = re.escape(string.punctuation)
print re.sub('['+chars+']', '',my_str)

Output:

hey there

Just a small tip about parameters style in python by PEP-8 parameters should be remove_special_chars and not removeSpecialChars

Also if you want to keep the spaces just change [^a-zA-Z0-9 \n\.] to [^a-zA-Z0-9\n\.]

edited Jan 02 '23 at 14:55

off99555

3,797
3
37
49

answered Jun 02 '14 at 14:01

Kobi K

7,743
6
42
86

1

+1 clearly the fastest and best answer it handles every case, translate will not do anything if given strange utf8 characters, re.sub with negative regex [^...] is much safer. – thibault ketterer Jun 19 '15 at 14:22
how to replace the character ` using re.sub? – vineeshvs Apr 27 '19 at 07:41
2

very helpfull answer, but on the last line don't you mean that using [^a-zA-Z0-9\n\.] will REMOVE the spaces ? – radouxju Feb 18 '21 at 14:08

score 54 · Accepted Answer · answered Jun 02 '14 at 14:02

54

str.replace is the wrong function for what you want to do (apart from it being used incorrectly). You want to replace any character of a set with a space, not the whole set with a single space (the latter is what replace does). You can use translate like this:

removeSpecialChars = z.translate ({ord(c): " " for c in "!@#$%^&*()[]{};:,./<>?\|`~-=_+"})

This creates a mapping which maps every character in your list of special characters to a space, then calls translate() on the string, replacing every single character in the set of special characters with a space.

answered Jun 02 '14 at 14:02

rassahah

746
5
6

Are you sure regex will perform better than translate? translate might be using regex internally ? – Vreddhi Bhat Jun 22 '17 at 05:33
2

Very well done for the `ord` use! Otherwise str.translate on special characters does nothing. – bergercookie Apr 21 '19 at 09:50
Thanks! This answer saved my day. – Jinhua Wang Sep 13 '19 at 16:10
Note that this replaces anything IN a set of characters, while [this answer](https://stackoverflow.com/a/23996414/2025495) replaces anything NOT IN a regex match. The latter is probably a safer approach if the goal is to make a string "safe" for a given context. – AdamAL Dec 15 '20 at 14:15

score 8 · Answer 3 · answered Jun 02 '14 at 13:58

8

You need to call replace on z and not on str, since you want to replace characters located in the string variable z

removeSpecialChars = z.replace("!@#$%^&*()[]{};:,./<>?\|`~-=_+", " ")

But this will not work, as replace looks for a substring, you will most likely need to use regular expression module re with the sub function:

import re
removeSpecialChars = re.sub("[!@#$%^&*()[]{};:,./<>?\|`~-=_+]", " ", z)

Don't forget the [], which indicates that this is a set of characters to be replaced.

answered Jun 02 '14 at 13:58

Danny M

427
2
6

This is wrong. The [] within the square brackets needs to be espaced and the - needs to be at the end as otherwise it will be a range. Do re.sub("[!@#$%^&*(){};:,./<>?\|`~=_\[\]-]", " ", z) instead. – Majte May 08 '23 at 07:02

Pavel · Answer 4 · 2014-06-02T14:01:44.473

1

replace operates on a specific string, so you need to call it like this

removeSpecialChars = z.replace("!@#$%^&*()[]{};:,./<>?\|`~-=_+", " ")

but this is probably not what you need, since this will look for a single string containing all that characters in the same order. you can do it with a regexp, as Danny Michaud pointed out.

as a side note, you might want to look for BeautifulSoup, which is a library for parsing messy HTML formatted text like what you usually get from scaping websites.

edited Jun 02 '14 at 14:01

answered Jun 02 '14 at 13:51

Pavel

7,436
2
29
42

I have to use just the libraries included in python. Is there regex that could accomplish what I am trying to do? – user2363217 Jun 02 '14 at 13:55
it depends on whether you are about to work with English texts, texts that include foreign words (with accents, umlauts, etc.), digits, currency symbols etc. There is no universal regex to "clean up stuff", you need to be specific about what you need. – Pavel Jun 02 '14 at 13:59

score 0 · Answer 5 · answered Feb 12 '15 at 16:08

You can replace the special characters with the desired characters as follows,

import string
specialCharacterText = "H#y #@w @re &*)?"
inCharSet = "!@#$%^&*()[]{};:,./<>?\|`~-=_+\""
outCharSet = "                               " #corresponding characters in inCharSet to be replaced
splCharReplaceList = string.maketrans(inCharSet, outCharSet)
splCharFreeString = specialCharacterText.translate(splCharReplaceList)

score 0 · Answer 6 · answered Sep 02 '22 at 17:35

0

Translate seems faster:

N=100000, 30 special characters, string length=70

replace: 0.3251810073852539 re.sub: 0.2859320640563965 translate: 0.12320685386657715

answered Sep 02 '22 at 17:35

Yano

598
5
12

Replace special characters in a string in Python

6 Answers6

Linked