I'm working on a programm that uses data from .txt-files and does stuff with said data. The data contains mostly latin characters, but sometimes there are also Japanese characters. That's what I want to do:
# -- coding: UTF-8 --
import numpy as np
test=open("test.txt", "r")
test2=open("list.txt", "w")
test2.write("# ")
for line in test:
line2=line.replace('""', "(None)")
line3=line2.replace('"', "")
line4=line3.replace(" ", "_")
line5=line4.replace(",", " ")
test2.write(line5)
It works perfectyl fine but there are some Japanese characters that cause problems. Funny thing is, characters like ゲ, ノ, セ, ト or ク are no big deal, but these characters are:い が か.
As soon as one of them hides somewhere in test.txt, the follwoing error-message occurs:
UnicodeDecodeError Traceback (most recent call last) C:\Users\syhon\Documents\DV-Liste\ListeV2.0\ListeV2.py in <module>()
196
197 test2.write("# ")
--> 198 for line in test:
199 line2=line.replace('""', "(None)")
200 line3=line2.replace('"', "")
C:\Users\syhon\Anaconda3\lib\encodings\cp1252.py in decode(self, input, final)
21 class IncrementalDecoder(codecs.IncrementalDecoder):
22 def decode(self, input, final=False):
---> 23 return codecs.charmap_decode(input,self.errors,decoding_table)[0]
24
25 class StreamWriter(Codec,codecs.StreamWriter):
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 6281: character maps to <undefined>
However, I was able to find out that I can print said characters without problem in python 2, but not in python 3. So, is it possible to get these characters decoded in python 3?
