10

I'd like to convert some character into five digit unicode on in Python 3.3. For example,

import re
print(re.sub('a', u'\u1D15D', 'abc' ))

but the result is different from what I expected. Do I have to put the character itself, not codepoint? Is there a better way to handle five digit unicode characters?

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
user1610952
  • 1,249
  • 1
  • 16
  • 31

2 Answers2

18

Python unicode escapes either are 4 hex digits (\uabcd) or 8 (\Uabcdabcd); for a codepoint beyond U+FFFF you need to use the latter (a capital U), make sure to left-fill with enough zeros:

>>> '\U0001D15D'
''
>>> '\U0001D15D'.encode('unicode_escape')
b'\\U0001d15d'

(And yes, the U+1D15D codepoint (MUSICAL SYMBOL WHOLE NOTE) is in the above example, but your browser font may not be able to render it, showing a place-holder glyph (a box or question mark) instead.

Because you used a \uabcd escape, you replaced a in abc with two characters, the codepoint U+1D15 (, latin letter small capital ou), and the ASCII character D. Using a 32-bit unicode literal works:

>>> import re
>>> print(re.sub('a', '\U0001D15D', 'abc' ))
bc
>>> print(re.sub('a', u'\U0001D15D', 'abc' ).encode('unicode_escape'))
b'\\U0001d15dbc'

where again the U+1D15D codepoint could be displayed by your font as a placeholder glyph instead.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
1

By the way, you do not need the re module for this. You could use str.translate:

>>> 'abc'.translate({ord('a'):'\U0001D15D'})
'bc'
unutbu
  • 842,883
  • 184
  • 1,785
  • 1,677