4

I am trying to get the span of selected words in a string. When working with the İ character, I noticed the following behavior of Python:

len("İ")
Out[39]: 1

len("İ".lower())
Out[40]: 2

# when `upper()` is applied, the length stays the same
len("İ".lower().upper())
Out[41]: 2

Why does the length of the upper and lowercase value of the same character differ (that seems very confusing/undesired to me)?

Does anyone know if there are other characters for which that will happen? Thank you!

EDIT:

On the other hand for e.g. Î the length stays the same:

len('Î')
Out[42]: 1

len('Î'.lower())
Out[43]: 1
lux7
  • 1,600
  • 2
  • 18
  • 34
  • *Does anyone know if there are other characters for which that will happen?* That İ is the only one, currently, as far as I know. For a lower character that does the other way (becomes longer after str.upper) there are hundreds, the most well-known of which is ß – wim Nov 25 '20 at 17:26
  • Thanks for your comment, I was not aware of thes behaviour either. – lux7 Nov 26 '20 at 09:13

2 Answers2

3

That's because 'İ' in lowercase is 'i̇', which has 2 characters

>>> import unicodedata
>>> unicodedata.name('İ')
'LATIN CAPITAL LETTER I WITH DOT ABOVE'
>>> unicodedata.name('İ'.lower()[0])
'LATIN SMALL LETTER I'
>>> unicodedata.name('İ'.lower()[1])
'COMBINING DOT ABOVE'

One character is a combining dot that your browser might render overlapped with the last quote, so you may not be able to see it. But if you copy-paste it into your python console, you should be able to see it.

If you try:

print('i̇'.upper())

you should get

İ
Red
  • 26,798
  • 7
  • 36
  • 58
  • Thanks a lot for your answer! The `unicodedata.name('İ')` method is very interessting indeed. The question that I still have difficulties to understand, is why that happens? Isn't this a undesired behaviour? – lux7 Nov 26 '20 at 09:17
-1

I think the issue is that a lower case character for that symbol is undefined in ASCII.

The .lower() function probably performs a fixed offset to the ASCII number associated with the character, since that works for the English alphabet.

topgunner
  • 52
  • 5
  • This is unicode - has nothing to do with ASCII encoding. And that's not what `str.lower` actually does.. – wim Nov 25 '20 at 17:43