Remove all characters from a string who's ordinals are out of range

Question

What is a good way to remove all characters that are out of the range: ordinal(128) from a string in python?

I'm using hashlib.sha256 in python 2.7. I'm getting the exception:

UnicodeEncodeError: 'ascii' codec can't encode character u'\u200e' in position 13: ordinal not in range(128)

I assume this means that some funky character found its way into the string that I am trying to hash.

Thanks!

You should just use UTF8 rather than ASCII – SLaks Jun 06 '12 at 22:46 — SLaks, Jun 06 '12 at 22:46
This is an example of the wrong way to deal with unicode. – Steven Rumbalski Jun 06 '12 at 22:47 — Steven Rumbalski, Jun 06 '12 at 22:47

Joran Beasley · Accepted Answer · 2012-06-06T23:00:15.577

6

new_safe_str = some_string.encode('ascii','ignore')

I think would work

or you could do a list comprehension

"".join([ch for ch in orig_string if ord(ch)<= 128])

[edit] however as others have said it may be better to figure out how to deal with unicode in general... unless you really need it encoded as ascii for some reason

edited Jun 06 '12 at 23:00

answered Jun 06 '12 at 22:47

Joran Beasley

110,522
12
160
179

This is the accepted answer because it is the only one that will work for my use case. It would have been nice to know in advance that the hash function needed some more micro-management to work correctly, but now that several million database entries have secondary keys using the current hash method, I am not in a position to change it. – Chris Dutrow Jun 07 '12 at 11:05

score 4 · Answer 2 · answered Jun 06 '12 at 23:08

Instead of removing those characters, it would be better to use an encoding that hashlib won't choke on, utf-8 for example:

>>> data = u'\u200e'
>>> hashlib.sha256(data.encode('utf-8')).hexdigest()
'e76d0bc0e98b2ad56c38eebda51da277a591043c9bc3f5c5e42cd167abc7393e'

score 2 · Answer 3 · answered Jun 06 '12 at 23:04

This is an example of where the changes in python3 will make an improvement, or at least generate a clearer error message

Python2

>>> import hashlib
>>> funky_string=u"You owe me £100"
>>> hashlib.sha256(funky_string)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa3' in position 11: ordinal not in range(128)
>>> hashlib.sha256(funky_string.encode("utf-8")).hexdigest()
'81ebd729153b49aea50f4f510972441b350a802fea19d67da4792b025ab6e68e'
>>>

Python3

>>> import hashlib
>>> funky_string="You owe me £100"
>>> hashlib.sha256(funky_string)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Unicode-objects must be encoded before hashing
>>> hashlib.sha256(funky_string.encode("utf-8")).hexdigest()
'81ebd729153b49aea50f4f510972441b350a802fea19d67da4792b025ab6e68e'
>>>

The real problem is that sha256 takes a sequence of bytes which python2 doesn't have a clear concept of. Use .encode("utf-8") is what I'd suggest.

Remove all characters from a string who's ordinals are out of range

3 Answers3

Linked