0

I'm trying to get the shift-jis character code from a unicode string. I'm not really that knowledgable in python, but here is what I have tried so far:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from struct import *

data="臍"
udata=data.decode("utf-8")
data=udata.encode("shift-jis").decode("shift-jis")
code=unpack(data, "Q")
print code

But I get an UnicodeEncodeError: 'ascii' codec can't encode character u'\u81cd' in position 0: ordinal not in range(128) error. The string is always a single character.

sollniss
  • 1,895
  • 2
  • 19
  • 36
  • Do you have the possibility to switch to Python 3? It handles all of this stuff natively. – Harald Nordgren Feb 12 '16 at 03:30
  • I have the possibility, yes. But it would be much effort to do so, so I would rather not ... – sollniss Feb 12 '16 at 03:49
  • What do you mean by "shift-jis chararacter code"? Its JIS X 0208 kuten code? What output are you expecting? Why are you using `struct.unpack()`? – 一二三 Feb 12 '16 at 03:55
  • Just an integer. Basically I'm trying to get this (http://kanji.jitenon.jp/cat/search.php?getdata=%E8%87%8D&search=fpart&page=1) number. I figured it is an offset jis character code. – sollniss Feb 12 '16 at 03:59
  • Please tell us the line causing the problem! You are getting the error during `unpack` and its because you have the format string in the wrong place. Use `code=unpack("Q", data)`. Then you get a different error. – tdelaney Feb 12 '16 at 04:21
  • The new error is `struct.error: unpack requires a string argument of length 8 `. – sollniss Feb 12 '16 at 04:26
  • I'm not sure what your final goal is, but since this is a utf-8 encoded python 2 file, the variable should be written as `data = u"臍"` (that is, as a unicode string) and @mhawke shows you the encoded data. I don't see what the unpack is for. – tdelaney Feb 12 '16 at 04:34

2 Answers2

1

That character is represented in shift-jis as the two byte sequence 0xE4 and 0x60:

>>> data = u'\u81cd'
>>> data_shift_jis = data.encode('shift-jis')
'\xe4`'
>>> hex(ord('`'))
0x60

So '\xe4\x60' is u'\u81cd' encoded as shift-jis.

mhawke
  • 84,695
  • 9
  • 117
  • 138
-1

In python 2, when you create a utf-8 encoded string, you can leave encoded (data = "臍") or you can have python decode it into a unicode string for you when the program is parsed (`data = u"臍"). The second option is the normal way to create strings when your source file is utf-8 encoded.

When you tried to convert to JIS, you ended up decoding the JIS back into a python unicode string. And when you tried to unpack, you asked for "Q" (unisgned long long) when you really want "H" (unsigned short).

Following are two samples to get information on the character

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from struct import *

# here we have an "ascii" string that is really utf-8 encoded char
data="臍"
jis_data = data.decode('utf-8').encode("shift-jis")
code = unpack(">H", jis_data)[0]
print repr(data), repr(jis_data), hex(code)[2:]

# here python decodes the utf-8 encoded char for us
data=u"臍"
jis_data = data.encode("shift-jis")
code = unpack(">H", jis_data)[0]
print repr(data), repr(jis_data), hex(code)[2:]

Which results in

'\xe8\x87\x8d' '\xe4`' 58464 0xe460
u'\u81cd' '\xe4`' 58464 0xe460
tdelaney
  • 73,364
  • 6
  • 83
  • 116