1

So I have two simple ctypes struct

class S2 (ctypes.Structure):
    _fields_ = [
    ('A2',     ctypes.c_uint16*10),
    ('B2',     ctypes.c_uint32*10),
    ('C2',     ctypes.c_uint32*10) ]


class S1 (ctypes.Structure):
    _fields_ = [
    ('A',     ctypes.c_uint16),
    ('B',     ctypes.c_uint32),
    ('C',     S2) ]

Is it possible to do this same for example with namedtuple? How lists are handled in namedtuple?

Edit:

struc.pack usage

test_data = '0100000002000000' + 10*'01' + 10*'01' + 10*'01'

S2 = collections.namedtuple('S2', ['A2', 'B2', 'C2'])
S1 = collections.namedtuple('S1', ['A', 'B', 'REF_to_S2'])

Data2 = S2._make(struct.unpack('10p10p10p', binascii.unhexlify(test_data[16:])))
##this is not working, because there must be 3 args..
Data1 = S1._make(struct.unpack('ii', binascii.unhexlify(test_data[0:16])))

In the end i want to print data in readable format ( having key:value pairs visible). But now i cannot figure out how should i handle that unpack operation with two diffrent namedtuples...?

this unpack.struct operation will take care that value type issues, right ?

Juster
  • 385
  • 2
  • 3
  • 11
  • 7
    What are you planning on doing with these structures? Are you passing them to C functions or just using them in python? Also, do you need to guarantee the type of the fields? Finally, as far as I can tell, Structure objects are mutable whereas namedtuple are not. – mgilson Apr 26 '13 at 18:45
  • Just inside python but I would like to be sure that fields and received values are the same type. – Juster Apr 27 '13 at 13:19
  • To clarify: What you're ultimately asking is how to handle nested structures in `struct`, right? You can't do it directly, but the two most obvious ways to do it indirectly (embed the nested structures manually in the format strings, or string-manipulate the data manually) both work. See my updated answer for details. (And if that's not what you're asking, apologies.) – abarnert Apr 29 '13 at 18:11

1 Answers1

5

Is it possible to do this same for example with namedtuple?

It depends on what you mean by "same". You can easily create namedtuple types with the same fields:

S2 = collections.namedtuple('S2', ['A2', 'B2', 'C2'])
S1 = collections.namedtuple('S1', ['A', 'B', 'C'])

However, they're obviously not the same type, and won't have the same behavior.


First, these fields are normal Python attributes (and also normal tuple members), which means they don't have static types; they can hold values of any type.

So, you can do this:

s2 = S2([ctypes.c_uint16(i) for i in range(10)],
        [ctypes.c_uint32(i) for i in range(10)],
        [ctypes.c_uint32(i) for i in range(10)])
s1 = S1(ctypes.c_uint16(1), ctypes.c_uint32(2), s2)

But you can also do this:

s2 = S2('a', 'b', 'c')
s1 = S1('d', 'e', s2)

… or even:

s1 = S1('d', 'e', 'f')

And also, note that even the first example actually created lists of 10 ctypes values, not ctypes arrays. If you want that, you have to cast them explicitly.


Second, namedtuples are an extension of tuples, which means they're immutable, so you can't do this:

s1.C = s2

And most importantly, a namedtuple can't be used as a ctypes.Structure—you can't pass it to a C function, you have to write manual logic (e.g., around struct.pack) if you want to serialize it in some specific binary format, etc.


How lists are handled in namedtuple?

As mentioned above, the members of a namedtuple aren't statically typed, and can hold values of any type. So, they're handled the same way they are in a tuple, list, a normal class instance, a global variable, etc. Just stick a list in there and you've got a list.


yes that stuct.pack is what i need. But i cannot figure out how i should point that the last value in s1 is reference to s2 structure.

From the namedtuple side, you just use an S2 instance as the value for the S1.C, as in my examples above. Again, the items/attributes of a namedtuple are just like any other attributes/variables/etc. in Python, just names that hold references to objects. So, s1 = S1(1, 2, s2) will make the third item of s1 into another reference to the same object that s2 refers to.

As for how to use struct to serialize the data: The struct module doesn't have any way to directly delegate to an embedded object. But since the output of pack is just a bytes (or, in Python 2.x, str) object, you can do this with normal string manipulation:

# version 1
s2_struct = struct.Struct('!HII')
s1_header = struct.Struct('!HI')
def pack_s2(s2):
    return s2_struct.pack(s2.A2, s2.B2, s2.C2)
def unpack_s2(s2):
    return S2._make(s2_struct.unpack(s2))
def pack_s1(s1):
    return s1_header.pack(s1.A, s1.B) + pack_s2(s1.C)
def unpack_S1(s1):
    offset = len(s1_header)
    a, b = s1_header.unpack(s1[:offset])
    c = unpack_s2(s1[offset:])
    return S1._make(a, b, c)

(I personally would use S2(*struct.unpack rather than S2._make, but since the documentation does the latter repeatedly, I guess that has to be the intended way to do things…)

Alternatively, you can flatten the format strings manually:

s2_struct = struct.Struct('!HII')
s1_struct = struct.Struct('!HIHII')
def pack_s2(s2):
    return s2_struct.pack(s2.A2, s2.B2, s2.C2)
def pack_s1(s1):
    return s1_struct.pack(s1.A, s1.B, s1.C.A2, s1.C.B2, s1.C.C2)
def unpack_s2(s2):
    return S2._make(s2_struct.unpack(s2))
def unpack_S1(s1):
    a, b, a2, b2, c2 = s1_struct.unpack(s1)
    c = S2(a2, b2, c2)
    return S1(a, b, c)

I think the second version is easier to read, but it's also easier to get wrong, and requires you to think about composing objects at the binary level instead of the Python level, so… choose whichever you find the lesser of two evils.

abarnert
  • 354,177
  • 51
  • 601
  • 671
  • I don't know whether to consider this answer valiant or foolhardy. I mean, the question probably doesn't deserve an answer until the information requested by @mgilson's comment is provided. – John Y Apr 26 '13 at 20:06
  • Thanks, yes that stuct.pack is what i need. But i cannot figure out how i should point that the last value in s1 is reference to s2 structure. – Juster Apr 27 '13 at 13:45
  • @Juster: OK, I think I have a handle on what you're asking, so let me edit the answer. – abarnert Apr 29 '13 at 17:40
  • 1
    Just because `_make` is in usage examples for `namedtuple` doesn't mean you should feel pressure to use it instead of unpacking via `*`. The "official party line" regarding `_make` is that it's there to ["help avoid unpacking and repacking the arguments"](http://code.activestate.com/lists/python-list/636436/). So conceivably there is a tiny performance benefit to `_make` if the arguments are already bundled up. To me, this is not compelling enough to override your sense of taste, and I am quite sure plenty of people do prefer using `*`. – John Y Apr 29 '13 at 19:50
  • @JohnY: Thanks for the link! The question is why the `struct` docs use it. Obviously _someone_ thought it was a compelling reason. That doesn't necessarily prove anything, but… I think it's enough to leave both possibilities in there, along with the discussion? Or am I bending over backward too far here? – abarnert Apr 29 '13 at 20:17
  • I think the way you've handled it in this answer is fine. As for the `struct` docs: It's only in there once, in an example that populates a `namedtuple` from the `tuple` returned by `struct.unpack`. So it could have been a conscious decision to endorse `_make`, or it could have just been a case of "that's the one I happened to grab first". I wouldn't worry too much about it either way. – John Y Apr 29 '13 at 20:42