How to detect double-byte numbers

Question

I have to check strings in Japanese that are encoded in double-byte characters (naturally the files aren't in Unicode and I have to keep them in Shift-JIS). Many of these strings contain digits that are also double byte characters, (１２３４５６７８９) instead of standard single-byte digits (0-9). As such, the usual methods of searching for digits won't work (using [0-9] in regex, or \d for example).

The only way I've found to make it work is to create a tuple and iterate over the tuple in a string to look for a match, but is there a more effective way of doing this?

This is an example of the output I get when searching for double byte numbers:

>>> s = "２34"  # "2" is a double-byte integer
>>> if u"2" in s:
      print "y"

>>> if u"２" in s:
      print "y"

    y
>>> print s[0]

>>> print s[:2]
    ２
>>> print s[:3]
    ２3

Any advice would be greatly appreciated!

convert the string to unicode, search with a unicode regex, do whatever it is you need with the string, convert back to shift-jis... — isedev, Sep 12 '14 at 05:20
Like @isedev already said, make sure you always work in unicode in Python (not UTF8/16). You can then use the [`unicodedata`](https://docs.python.org/2/library/unicodedata.html) module to give you information about the characters, for example: `unicodedata.numeric(u'１')` => `1.0` or `unicodedata.name(u'１')` => `'FULLWIDTH DIGIT ONE'` or `unicodedata.digit(u'１')` => `1` — Lukas Graf, Sep 12 '14 at 05:48
I've tried that before but with limited success. All of Shift-JIS can convert to Unicode but not the other way around. I'll try it again. — Tensigh, Sep 12 '14 at 12:15
@Tensigh I think you're confusing Unicode and UTF encodings. You can't output Unicode from Python (or any language) to a file, the network, STDOUT, etc.. You only ever can output a bytestring, which will need to be encoded in an encoding like UTF-8. — Lukas Graf, Sep 12 '14 at 12:30

score 4 · Accepted Answer · answered Sep 12 '14 at 19:05

First of all, the comments are right: for the sake of your sanity, you should only ever work with unicode inside your Python code, decoding from Shift-JIS that comes in, and encoding back to Shift-JIS if that's what you need to output:

text = incoming_bytes.decode("shift_jis")
# ... do stuff ...
outgoing_bytes = text.encode("shift_jis")

See: Convert text at the border.

Now that you're doing it right re: unicode and encoded bytestrings, it's straightforward to get either "any digit" or "any double width digit" with a regex:

>>> import re
>>> s = u"２34"
>>> digit = re.compile(r"\d", re.U)
>>> for d in re.findall(digit, s):
...     print d,
... 
２ 3 4
>>> wdigit = re.compile(u"[０-９]+")
>>> for wd in re.findall(wdigit, s):
...     print wd,
... 
２

In case the re.U flag is unfamiliar to you, it's documented here.

Oh, `shift_jis` is an *encoding*! For some reason I was under the assumption it was a JavaScript framework. Now I feel stupid :) Either way, great answer! — Lukas Graf, Sep 13 '14 at 08:54

score 0 · Answer 2 · answered May 23 '18 at 03:22

I had a similar problem when facing Japanese two-byte characters and one relatively easy way to deal with the characters that I found is to transform them using the simple Unicode numbers (at least for processing them, if you want to keep the document as it is):

ord("２")

will return

which is 65248 points away from the one-byte characters 2. So converting back can be done using:

def convert_two_byte_numbers(character: str):
    if ord(character) in range(65296, 65306):
        return chr(ord(character) - 65248)
    else: 
        return character

If, like me, you also need to convert two-byte letters, add the same thing for the ranges (65313, 65339) and (65345, 65371).

How to detect double-byte numbers

2 Answers2