python string every two chars to byte - do it fast

Question

Got a binary blob string like:

input = "AB02CF4AFF"

Every pair "AB", "02", "CF", "4A", "FF" constitute a byte. I'm doing this:

data = StringIO()
for j in range(0, len(input)/2):
    bit = input[j*2:j*2+2]
    data.write('%c' % int(bit,16))
data.seek(0)

Works ok, but with large binary blobs this becomes unacceptable slow and sometimes event throws a MemoryError.

struct.unpack comes to mind, but no luck thus far.

Any way to speed this up?

possible duplicate of http://stackoverflow.com/questions/3753589/packing-and-unpacking-variable-length-array-string-using-the-struct-module-in-py — santosh singh, Dec 22 '13 at 18:08
@immortal YES! Don't even need struct.unpack and the loop :S My code now a factor zillion higher. Thanks! btw: you're answer should be the accepted answer, how should I handle that? Thanks again. — Tim Knip, Dec 22 '13 at 19:02

score 5 · Accepted Answer · answered Dec 22 '13 at 19:45

5

Use binascii.unhexlify:

>>> import binascii
>>> binascii.unhexlify('AB02CF4AFF')
b'\xab\x02\xcfJ\xff'

(In Python 2 you can decode with the hex codec but this isn't portable to Python 3.)

answered Dec 22 '13 at 19:45

Gareth Rees

64,967
9
133
163

Perfect! Thanks, even better then @immortal 's answer because of portability to Python 3. – Tim Knip Dec 22 '13 at 19:50

score 4 · Answer 2 · answered Dec 22 '13 at 19:36

4

Give input.decode('hex') a try :)

Always a good idea to use built-in solutions

answered Dec 22 '13 at 19:36

immortal

3,118
20
38

score 1 · Answer 3 · answered Dec 22 '13 at 18:51

How about something like this?

def chrToInt(c):
    if c >= '0' and c <= '9':
        return int(ord(c) - ord('0'))
    elif c >= 'A' and c <= 'F':
        return int(ord(c) - ord('A')) + 10
    else:
        # invalid hex character, throw an exception or something here
        return None

def hexToBytes(input):
    bytes = []

    for i in range(0, len(input) - 1, 2):
        val = (chrToInt(input[i]) * 16) + chrToInt(input[i + 1])

        bytes.append(val)

    return bytes

print hexToBytes("AB02CF4AFF")

You could speed it up quite a bit by making chrToInt branchless by using binary operations, and you could also modify hexToBytes to say exactly how many characters it should read if you decide you want to use something bigger than bytes (so it returns it in groups of 4 for a short or 8 for an int).

The answer @immortal gave : `input.decode('hex')` is way faster for my usecase (Need the binary string as input for PIL Image.frombuffer). But thanks for answering, could come in handy for other cases. — Tim Knip, Dec 22 '13 at 19:11

python string every two chars to byte - do it fast

3 Answers3