I downloaded tweets in Urdu language. When I read the csv file using pandas in python, the tweet is shown as follows:
Sample tweet text
Unicode code point I want to convert this into utf-8.
Asked
Active
Viewed 548 times
-1
-
use `.encode('utf-8')`. But why would you want to remove it anyways? – Ganesh Kathiresan Dec 04 '17 at 10:49
-
You could use `str('yourtext')` or `'{}'.format('yourtext')` or like Ganesh said, `'sometext'.encode('utf-8')` – Morb Dec 04 '17 at 10:50
-
Ok so after the edit, `.encode('utf-8')` should do the trick either ways – Ganesh Kathiresan Dec 04 '17 at 10:52
-
Using .encode('utf-8') throws error --------------------------------------------------------------------------- UnicodeDecodeError Traceback (most recent call last)
in – user3368581 Dec 04 '17 at 10:53() ----> 1 print urdutweets.iloc[1]['text'].encode('utf-8') UnicodeDecodeError: 'ascii' codec can't decode byte 0x85 in position 544: ordinal not in range(128) -
@Morb {}'.format('yourtext') has no effect. It is still printing the string in the form of unicode codes – user3368581 Dec 04 '17 at 10:56
-
You *don't* want to convert it into UTF-8. You want to convert the `` escapes into regular Python strings. – tripleee Dec 04 '17 at 10:56
-
Have you tried .encode('utf-8', "ignore") – Hari Krishnan Dec 04 '17 at 10:56
-
You're not doing something silly like `less file-utf8.txt | python` to botch the input like that, are you? – tripleee Dec 04 '17 at 10:59
-
Ok so if you do `print(u'\xf1')`, you get: `ñ`, which is basically not ascii and it works. So why do you wish to convert exactly? Like I worked with Hindi recently and had no need to convert and worked with it natively, that's why I'm asking – Ganesh Kathiresan Dec 04 '17 at 10:59
-
@HariKrishnan yes. It is also giving the same error as with .encode('utf-8') – user3368581 Dec 04 '17 at 11:01
-
how about this ?.encode("utf-8", "ignore").decode('utf-8', "ignore") – Hari Krishnan Dec 04 '17 at 11:04
-
it is still not working @HariKrishnan – user3368581 Dec 04 '17 at 11:09
-
@GaneshK I have vector embeddings in the format of utf-8 therefore I need to replace unicode codepoints with their utf-8 representations – user3368581 Dec 04 '17 at 11:10
-
Sorry , i meant this .decode("utf-8", "ignore").encode('utf-8', "ignore") – Hari Krishnan Dec 04 '17 at 11:12
-
`your_message.decode("iso8859-6")` ? – dsgdfg Dec 04 '17 at 12:01
1 Answers
-1
When you are writing the tweet data to a file use .decode('utf-8')
And when you try to read data from that file use .encode('utf-8')
Here I am posting an example:
# -*- coding: utf-8 -*-
string1 = "آکاش کمار"
string2 = string1.decode('utf-8')
string3 = string2.encode('utf-8')
print(string3)
eiram_mahera
- 950
- 9
- 25