Unpack format characters in Python

Question

I need the Python analog for this Perl string:

unpack("nNccH*", string_val)

I need the nNccH* - data format in Python format characters.

In Perl it unpack binary data to five variables:

16 bit value in "network" (big-endian)
32 bit value in "network" (big-endian)
Signed char (8-bit integer) value
Signed char (8-bit integer) value
Hexadecimal string, high nibble first

But I can't do it in Python

More:

bstring = ''
while DataByte = client[0].recv(1):
    bstring += DataByte
print len(bstring)
if len(bstring):
    a, b, c, d, e = unpack("nNccH*", bstring)

I never wrote in Perl or Python, but my current task is to write a multithreading Python server that was written in Perl...

I can find the equivalent of everything except for H*, for which I would assume you would play with p or s. — Senthil Kumaran, Feb 07 '12 at 12:52
You will need to calculate the string size, this answer could be helpful. http://stackoverflow.com/a/5849224/70350 — pojo, Feb 07 '12 at 12:54
`"while DataByte = client[0].recv(1):"` is not Python. This can never work. — S.Lott, Feb 07 '12 at 13:17
@SenthilKumaran: AFAIR `*` just means "as many elements as are left", so he can unpack everything before the `H*`, and then just grab the rest without unpack — Eli Bendersky, Feb 07 '12 at 13:31
By the way, Sir D, thanks for editing and clarifying the question. The last code snippet makes little sense though, as S.Lott noticed — Eli Bendersky, Feb 07 '12 at 13:45
See [**THIS**](https://stackoverflow.com/questions/7867714/auto-repeat-flag-in-a-pack-format-string/49927071) question for how to repeatedly apply a format string! — timgeb, Apr 19 '18 at 17:40

score 8 · Accepted Answer · edited Jun 18 '17 at 21:19

The Perl format "nNcc" is equivalent to the Python format "!HLbb". There is no direct equivalent in Python for Perl's "H*".

There are two problems.

Python's struct.unpack does not accept the wildcard character, *
Python's struct.unpack does not "hexlify" data strings

The first problem can be worked-around using a helper function like unpack.

The second problem can be solved using binascii.hexlify:

import struct
import binascii

def unpack(fmt, data):
    """
    Return struct.unpack(fmt, data) with the optional single * in fmt replaced with
    the appropriate number, given the length of data.
    """
    # http://stackoverflow.com/a/7867892/190597
    try:
        return struct.unpack(fmt, data)
    except struct.error:
        flen = struct.calcsize(fmt.replace('*', ''))
        alen = len(data)
        idx = fmt.find('*')
        before_char = fmt[idx-1]
        n = (alen-flen)//struct.calcsize(before_char)+1
        fmt = ''.join((fmt[:idx-1], str(n), before_char, fmt[idx+1:]))
        return struct.unpack(fmt, data)

data = open('data').read()
x = list(unpack("!HLbbs*", data))
# x[-1].encode('hex') works in Python 2, but not in Python 3
x[-1] = binascii.hexlify(x[-1])
print(x)

When tested on data produced by this Perl script:

$line = pack("nNccH*", 1, 2, 10, 4, '1fba');
print "$line";

The Python script yields

[1, 2, 10, 4, '1fba']

An alternative to `binascii.hexlify()` is `str.encode("hex")`. — Sven Marnach, Feb 07 '12 at 13:48
If you want Python 3 compatibility, you'll need `//` when calculating `n`, otherwise `str(n)` produces '16.0' and breaks the format string. — Mark Tolonen, Feb 07 '12 at 14:03
In Python 3.4 and newer, there is `struct.iter_unpack`. [Here](https://stackoverflow.com/a/49927071/3620003) is a demonstration. — timgeb, Apr 19 '18 at 17:46

score 7 · Answer 2 · answered Feb 07 '12 at 12:37

7

The equivalent Python function you're looking for is struct.unpack. Documentation of the format string is here: http://docs.python.org/library/struct.html

You will have a better chance of getting help if you actually explain what kind of unpacking you need. Not everyone knows Perl.

answered Feb 07 '12 at 12:37

Eli Bendersky

263,248
89
350
412

Thanks. I already read the perl and python unpack docs. But so far i don't understand some moments. – Sir D Feb 07 '12 at 12:40
@Eli -there could minor trouble in direct translation. For e.g how would one do H* in python? I guess, the user could have worded the question better. – Senthil Kumaran Feb 07 '12 at 12:46
@SenthilKumaran: note that the user has edited the question *after* my answer. Before the edit he didn't lay out the meanings of the format chars in Perl – Eli Bendersky Feb 07 '12 at 13:01
@SirD: "so far i don't understand some moments". Please be **specific** on what you do not understand. Please **update** the question to say what you do not understand. – S.Lott Feb 07 '12 at 13:16

Unpack format characters in Python

2 Answers2

Linked