1

I have been dealing with an issue regarding the terminal in my Macbook. I am passing greek words in a python string e.g. 

text = 'Καλημέρα κόσμε'

and every time I try to perform any simple task to it like splitting in spaces the result I get looks like this:

['\xce\x9a\xce\xb1\xce\xbb\xce\xb7\xce\xbc\xce\xad\xcf\x81\xce\xb1',  
'\xce\xba\xcf\x8c\xcf\x83\xce\xbc\xce\xb5']

The same thing happens when I use the collections.Counter() function as well.

On the other hand when I print the string the output is as expected:

Καλημέρα κόσμε

I tried doing what is mentioned here: In OSX Lion, LANG is not set to utf8, how fix? (by changing en_US.UTF-8 to el_GR.UTF-8) without any luck.

Anyone has an idea why that happens and how I can tackle that?

Thank you in advance.

Community
  • 1
  • 1
Swan87
  • 421
  • 6
  • 23

1 Answers1

0

This is not an issue with your terminal, but how Python (2) does things.

Even if you don't perform any task on it, repr will escape any non-ASCII (or non-printable (except space)) characters:

>>> text = 'Καλημέρα κόσμε'
>>> text
'\xce\x9a\xce\xb1\xce\xbb\xce\xb7\xce\xbc\xce\xad\xcf\x81\xce\xb1 \xce\xba\xcf\x8c\xcf\x83\xce\xbc\xce\xb5'

If you try the same thing in Python 3, it'll print normally:

>>> text = 'Καλημέρα κόσμε'
>>> text
Καλημέρα κόσμε

Is there any reason why you're using Python 2?

L3viathan
  • 26,748
  • 2
  • 58
  • 81
  • Even if you try to print this string (is it possible to try that for me if that is not too much of a hassle?): low_vowels = 'αειοηυω' everything works as expected? The only reason I still have Python 2 is because many of the libraries I have work well there so there was no particular reason for me to make the switch. – Swan87 Sep 13 '16 at 15:03
  • In Python 2, when using print, yes. In Python 3, always. Python 2's string type is really just a bytestream, whereas in Python 3 it's unicode. If you're doing anything with non-ASCII alphabets, I'd recommend using Python 3, if there's nothing stopping you. Which library that you're using isn't available for Python 3 yet? – L3viathan Sep 13 '16 at 15:04
  • I will probably make the switch soon! It is not a matter of a library not being available in Python 3, just the fact the some of them just a while ago were flaky and not as stable in Python 2.7 . Have you been using Python 3 without any notable issues for sometime yourself? – Swan87 Sep 13 '16 at 15:09
  • I used Python 3 (almost) exclusively since 2012. I do a lot of text processing, and (for the most part) not having to worry about encodings anymore is very nice. – L3viathan Sep 13 '16 at 15:29
  • Everything works as expected, thanks for the guidelines. – Swan87 Sep 16 '16 at 09:34