0

I know there are many questions out there already concerning encoding/decoding. But this is driving me nuts and I'm in desperate need of some help.

I read in a file converting the lines to unicode

line = unicode(line,'latin-1')

Then, I do some mutations and try to write the contents back to a file, encoding the string like this

o_str = '%s,%s' % (new_sname, loc )
w_out.write(o_str.encode('latin-1'))

The file contains for instance the city name 'Genève' which is u'Gen\xc3\xa8ve' as unicode. Encoding it as 'Latin-1' 

gue = gu.encode('iso-8859-1')

gives me on the console 

>>> print gue
Genève

But in file my file it still is 'Genève'. Can somebody point me to what I am missing?

LarsVegas
  • 6,522
  • 10
  • 43
  • 67
  • The editor which you use, might use a different encoding to display the string. – thefourtheye Aug 15 '13 at 11:07
  • You mean the text editor? No, I check that already. – LarsVegas Aug 15 '13 at 11:09
  • `u'Gen\xc3\xa8ve'` is **not** correct unicode. That is UTF-8 data instead, decoded from latin-1. – Martijn Pieters Aug 15 '13 at 11:09
  • @thefourtheye Regardless of which encoding the editor uses, the letter è should still only look like a single letter if it was correctly encoded with latin-1. Which leads me to the conclusion that the Python script encodes with a wide-character encoding like UTF-8. – kqr Aug 15 '13 at 11:11

1 Answers1

3

You are decoding UTF-8 data as Latin 1, use the correct codec instead:

>>> 'Gen\xc3\xa8ve'.decode('latin1')
u'Gen\xc3\xa8ve'
>>> print 'Gen\xc3\xa8ve'.decode('latin1')
Genève
>>> 'Gen\xc3\xa8ve'.decode('utf8')
u'Gen\xe8ve'
>>> print 'Gen\xc3\xa8ve'.decode('utf8')
Genève

The correct Unicode codepoint for the è letter is U+00E8, represented by \u00e8 or \xe8 in a Python Unicode literal, and the hex bytes C3A8 in UTF-8. Misintepreting C3 A8 leads to two unicode characters à and ¨, which you then write back to your file as C3 and A8 again because Latin1 maps one-on-one with Unicode.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343