C sizeof(some_structure) returns different value in compare with python struct.calcsize(struct_string)

Question

I have a C structure.

typedef struct
{
    double cycle_time; 
    double cycle_duty; 
    double state; 
    double servo_mode; 
    double motion_mode; 
    double jcond; 
    
    struct
    {
        uint16_t buff_sz; 
        uint16_t buff_fill; 
        uint16_t cmd_cntr; 
        uint16_t res; 
    } wpi;
    double move_des_q[6]; 
    double move_des_qd[6]; 
    double move_des_x[6]; 
    double move_des_xd[6]; 
    double act_q[6]; 
    double act_qd[6]; 
    double act_x[6]; 
    double act_xd[6]; 
    double act_tq[6]; 
    double frict_tq[6]; 
    double ne_tq[6];
    double act_force_e[6];
    double act_force_0[6]; 
    double des_trq[6]; 
    double des_qd[6]; 
    double temp_m[6]; 
    double temp_e[6]; 
    double arm_current;
    double arm_voltage;
    double psu_voltage;
    struct
    {
        uint8_t dig_in_count; 
        uint8_t an_in_count;
        uint8_t dig_in[8];
        uint8_t an_in_curr_mode[4];
        double an_in_value[4];

        uint8_t dig_out_count; //number of bits
        uint8_t an_out_count;
        uint8_t dig_out[8];
        uint8_t an_out_curr_mode[4];
        double an_out_value[4];
    } io;

    struct
    {
        uint32_t jointState;
        float joint_volt;       
        float joint_amp;        
        uint8_t joint_window;
        float joint_des_iq;
                                 
        float joint_des_vel;
    } jointInfo[6];
} some_structure;

After running calculating code, i've found out the size (1136 bytes) of the C structure (at least i hope it's correct). Here is the code:

int main()
{
    printf("Size of struct ABC: %lu\n", sizeof(some_structure));
}

But, after checkout the python structure size i've found some difference (python size is 1114 bytes). Here is python code:

    struct_string = "<6d4H105d14B4d14B4dL2fB2fL2fB2fL2fB2fL2fB2fL2fB2fL2fB2f"
    struct_byte_size = struct.calcsize(struct_string)
    print(struct_byte_size)

What cause this size shift? How can i receive the data from socket and avoid this shift ? May i've a mistake while i was creating the struct_string?

UPD: Here is the struct-module rule-table and calcsize() description.

I don't know exactly what `struct.calcsize` actually does – but I assume it sums up the size of the individual members – recursively, if need be. `sizeof` in C, in contrast, measures the total amount of bytes a struct occupies in memory – and this can include *padding* bytes! You e.g. have placed for `uint8_t`, next free address is a multiple of four, but not 8 – on a 64-bit system the subsequent double requires alignment at a multiple of 8, though, thus four padding bytes are inserted. Analogously three padding bytes after the single `uint8_t` followed by a float. — Aconcagua, Jul 20 '23 at 14:23
Why should it be the same size? Different languages, possibly different alignments/padding etc... — Jabberwocky, Jul 20 '23 at 14:27
@Aconcagua, sorry, i'll clarify the struct.calcsize. It's a build in method. I'll add photo to the question — Mika, Jul 20 '23 at 14:29
If you receive data from socket, never use it as C struct directly. There may be different padding, different byte order. Use serialization/deserialization. — dimich, Jul 20 '23 at 14:29
@Mika Never mind, looked it up myself – and indeed: As you use `<` for formatting, indicating ['no alignment'](https://docs.python.org/3.8/library/struct.html#struct.calcsize), padding bytes are left out... Try `@` instead, using native padding, you might get the same result then. — Aconcagua, Jul 20 '23 at 14:36
You have padding bytes in the C-struct: for normal systems :2 bytes before `double an_in_value[4];`, 2 bytes before `double an_out_value[4];`, 3 bytes before `float joint_des_iq;` in an array of 6 elements, so 18 bytes in total. So 22 bytes for the whole struct. 1114 + 22 = 1136. Adds up. — mch, Jul 20 '23 at 14:36
@Aconcagua, i've tried. now python calculates up to 1180 bytes. :( — Mika, Jul 20 '23 at 14:39
@mch, looks interesting. should i insert these bytes somehow or how can i unpack the data? dimich idea with serialisation/deserialization looks interesting too, but i'm in process of finding the solution... — Mika, Jul 20 '23 at 14:42
Python doesn't seem to align the same way as C does then. You could in contrast try to [pack](https://stackoverflow.com/questions/21092415/force-c-structure-to-pack-tightly) your struct instead then – just for getting the same size! Don't use such structs for serialisation, that will bring on the long run more trouble than going the hard way of serialising manually where you don't have to cope with alignment and have direct control over byte order (if you do shift and mask the indvidual bytes). — Aconcagua, Jul 20 '23 at 14:44
@Mika You may be getting incorrect results because you have nested structures. I'd try Aconcagua's recommendation to see if you get consistent results by packing the structure - but that can be dangerous, especially if you're working with non-x86 architectures that are nowhere near as forgiving when accesses are misaligned. Python tends to play a bit loose with things - look at the struct rule-table you linked to - it's **wrong** about the size of `long` always being 4 bytes. "Works most of the time" is a pretty low, pathetic standard to aim for. — Andrew Henle, Jul 20 '23 at 14:48
You could serialise into a human readable format as well (XML, JSON, ...), though I personally admittedly prefer the binary formats... — Aconcagua, Jul 20 '23 at 14:49
@Aconcagua, it would be really beautiful, if i change the host structure. But, it's not possible, as long as programmers team will not rewrite the structure, which already used in other places, only bc of my app, which receives data from the host PC. That's a pity but team doesn't want to refactor not well written project :( — Mika, Jul 20 '23 at 14:54
Do I understand you right, the team uses this struct to serialise data, including the padding bytes, and you are trying to re-build this from within python? — Aconcagua, Jul 20 '23 at 14:56
@Aconcagua is it possible to calculate the current C struct size from python? May be ctypes or smth? Or there is no possibility to calculate this structure from python? — Mika, Jul 20 '23 at 14:58
@Aconcagua, sorry, didn't notice the comment. Team just wrote the C robot controller, which send data over the local network endlessly, while robot is working. So, now i can connect the controller via socket with ip and port and receive the byte stream. They send a C structure and i'm trying to decode it — Mika, Jul 20 '23 at 15:00
@Mika they can only send that struct as a byte stream if they have packed it and the reciever/tranciever have the same endianess. So I doubt that's what they are doing. Are you sure they don't already have a serialization function for the struct? — Fredrik, Jul 20 '23 at 15:02
Not really familiar with ctypes, you'd need to try yourself, maybe it works, maybe not. What probably would work, if other means fail, is writing an [extension](https://docs.python.org/3/extending/extending.html) where you'd have this struct in pure C and create an appropriate interface for accessing the members. — Aconcagua, Jul 20 '23 at 15:03
@Fredrik That would need to be verified, indeed, but in worst case this struct indeed is sent as is (`send(theSocket, &theStructInstance, sizeof(theStructInstance), 0);`) – even with unpacked struct – and no, it's not impossible to decode, even with mismatching endianness, but that will get a real pain (more the padding than the endianness) :( — Aconcagua, Jul 20 '23 at 15:12
@Fredrik i'm sorry. I didn't notice the message. As i known from the teamlead, somewhere they "serialize the structure and send it" — Mika, Jul 20 '23 at 15:18
@Mika well that is your answer then, look at how they are serializing/sending it and decode it the same way. It doesn't matter what python thinks the size of the structure is as you will decode and fill in the fields yourself with python code. — Fredrik, Jul 20 '23 at 15:22

Mark Tolonen · Accepted Answer · 2023-07-20T16:31:52.587

The difference is due to the C compiler adding padding bytes for alignment. Code is more efficient if variables are at addresses that are multiples of the size of the variable's type. When using Python's struct module with anything but native(@) alignment (the default), it doesn't use padding bytes.

As an example the Microsoft compiler indicates padding if all warnings are enabled via cl /Wall test.c:

test.c(43): warning C4820: '<unnamed-tag>': '2' bytes padding added after data member 'an_in_curr_mode'
test.c(49): warning C4820: '<unnamed-tag>': '2' bytes padding added after data member 'an_out_curr_mode'
test.c(56): warning C4820: '<unnamed-tag>': '3' bytes padding added after data member 'joint_window'

This padding can be disabled with #pragma pack. Example below:

test.c

#include <stdio.h>
#include <stdint.h>

//#pragma pack(push, 1)
typedef struct {
    double cycle_time;
    double cycle_duty;
    double state;
    double servo_mode;
    double motion_mode;
    double jcond;
    struct {
        uint16_t buff_sz;
        uint16_t buff_fill;
        uint16_t cmd_cntr;
        uint16_t res;
    } wpi;
    double move_des_q[6];
    double move_des_qd[6];
    double move_des_x[6];
    double move_des_xd[6];
    double act_q[6];
    double act_qd[6];
    double act_x[6];
    double act_xd[6];
    double act_tq[6];
    double frict_tq[6];
    double ne_tq[6];
    double act_force_e[6];
    double act_force_0[6];
    double des_trq[6];
    double des_qd[6];
    double temp_m[6];
    double temp_e[6];
    double arm_current;
    double arm_voltage;
    double psu_voltage;
    struct {
        uint8_t dig_in_count;
        uint8_t an_in_count;
        uint8_t dig_in[8];
        uint8_t an_in_curr_mode[4];  // 2 bytes padding after
        double an_in_value[4];

        uint8_t dig_out_count; //number of bits
        uint8_t an_out_count;
        uint8_t dig_out[8];
        uint8_t an_out_curr_mode[4];  // 2 bytes padding after
        double an_out_value[4];
    } io;
    struct {
        uint32_t jointState;
        float joint_volt;
        float joint_amp;
        uint8_t joint_window;  // 3 bytes padding after
        float joint_des_iq;

        float joint_des_vel;
    } jointInfo[6];  // 3 * 6 = 18 bytes total padding
} some_structure;  // 18 + 2 + 2 = 22 extra bytes due to padding
//#pragma pack(pop)

int main(void) {
    printf("Size of struct ABC: %zu\n", sizeof(some_structure));
}

Output with native padding:

Output with the #pragma pack statements uncommented:

Remove the < from the Python code to use native padding, or account for the padding:

import struct

struct_string = '6d4H105d14B4d14B4dL2fB2fL2fB2fL2fB2fL2fB2fL2fB2fL2fB2f'
print(struct.calcsize(struct_string)) # native alignment
print(struct.calcsize('<' + struct_string)) # no alignment

struct_string = '<6d4H105d14B2x4d14B2x4dL2fB3x2fL2fB3x2fL2fB3x2fL2fB3x2fL2fB3x2fL2fB3x2f'
print(struct.calcsize(struct_string)) # explicit padding

Output:

1136
1114
1136

Unpacking your structure returns a 187-tuple. It can be hard to figure out the correct offset for a particular value.

Consider using the ctypes module where you can specify named fields and nested structures to more naturally look up a particular value. Example:

import ctypes as ct

PACK = 1  # use 8 for native or remove the _pack_ lines

class wpi(ct.Structure):
    _pack_ = PACK
    _fields_ = (('buff_sz', ct.c_uint16),
                ('buff_fill', ct.c_uint16),
                ('cmd_cntr', ct.c_uint16),
                ('res', ct.c_uint16))

class io(ct.Structure):
    _pack_ = PACK
    _fields_ = (('dig_in_count', ct.c_uint8),
                ('an_in_count', ct.c_uint8),
                ('dig_in', ct.c_uint8 * 8),
                ('an_in_curr_mode', ct.c_uint8 * 4),
                ('an_in_value', ct.c_double * 4),
                ('dig_out_count', ct.c_uint8),
                ('an_out_count', ct.c_uint8),
                ('dig_out', ct.c_uint8 * 8),
                ('an_out_curr_mode', ct.c_uint8 * 4),
                ('an_out_value', ct.c_double * 4))

class jointInfo(ct.Structure):
    _pack_ = PACK
    _fields_ = (('jointState', ct.c_uint32),
                ('joint_volt', ct.c_float),
                ('joint_amp', ct.c_float),
                ('joint_window', ct.c_uint8),
                ('joint_des_iq', ct.c_float),
                ('joint_des_vel', ct.c_float))

class some_structure(ct.Structure):
    _pack_ = PACK
    _fields_ = (('cycle_time', ct.c_double),
                ('cycle_duty', ct.c_double),
                ('state', ct.c_double),
                ('servo_mode', ct.c_double),
                ('motion_mode', ct.c_double),
                ('jcond', ct.c_double),
                ('wpi', wpi),
                ('move_des_q', ct.c_double * 6),
                ('move_des_qd', ct.c_double * 6),
                ('move_des_x', ct.c_double * 6),
                ('move_des_xd', ct.c_double * 6),
                ('act_q', ct.c_double * 6),
                ('act_qd', ct.c_double * 6),
                ('act_x', ct.c_double * 6),
                ('act_xd', ct.c_double * 6),
                ('act_tq', ct.c_double * 6),
                ('frict_tq', ct.c_double * 6),
                ('ne_tq', ct.c_double * 6),
                ('act_force_e', ct.c_double * 6),
                ('act_force_0', ct.c_double * 6),
                ('des_trq', ct.c_double * 6),
                ('des_qd', ct.c_double * 6),
                ('temp_m', ct.c_double * 6),
                ('temp_e', ct.c_double * 6),
                ('arm_current', ct.c_double),
                ('arm_voltage', ct.c_double),
                ('psu_voltage', ct.c_double),
                ('io', io),
                ('jointInfo', jointInfo * 6))

print(ct.sizeof(some_structure))
# Construct the structure from some byte data
s = some_structure.from_buffer_copy(b'\x01' * ct.sizeof(some_structure))
print(s.jointInfo[2].joint_window)  # example to view a value

Output (PACK=1):

1114
1

Output (PACK=8):

1136
1

Thank you for your really well made answer. I'm trying to use it for now. But i have some issue here. In my Ubuntu 23.04 system, with 11th Gen Intel(R) Core(TM) i5-1135G7 2.40GHz i have this output (for your code section with comparing 3 variants of calculating size): `native alignment 1180`, `no alignment 1114` and `explicit padding 1136`. In talk about Ctypes, all output is the same, as yours. — Mika, Jul 21 '23 at 07:20
In addition, to clarify the topic (for me), where and why should we use padding? I'll really appreciate every trusted info-clue, which i can obtain about this topic. — Mika, Jul 21 '23 at 07:25
@Mika My second sentence answers that. “ Code is more efficient if variables are at addresses that are multiples of the size of the variable's type.” Reading a 4-byte integer from an odd address takes more CPU cycles that one at an address divisible by 4, so padding is used to align variables to appropriate addresses. — Mark Tolonen, Jul 21 '23 at 12:30
@Mika I don’t know why you’d get such a large discrepancy in native mode. See if your compiler has a padding warning like Microsoft’s that you can enable. — Mark Tolonen, Jul 21 '23 at 12:31

C sizeof(some_structure) returns different value in compare with python struct.calcsize(struct_string)

1 Answers1

Linked