nanopb oneof size requirements

Question

I came across nanopb and want to use it in my project. I'm writing code for an embedded device so memory limitations are a real concern.

My goal is to transfer data items from device to device, each data item has an 32bit identifier and an value. The value can be anything from 1 char to float to long string. I'm wondering what would be the most efficient way of declaring the messages for this type of problem.

I was thinking something like this:

message data_msg{
    message data_item{
        int32 id = 1;
        oneof value{
            int8  ival = 2;
            float fval = 3;
            string sval = 4;
        }
    }
  repeated data_item;
}

But as I understood this is converted in to C union, which is the size of the largest element. Say I limit the string to 50 chars, then the union is always 50 bytes long, even if I would need 4 bytes for a float.

Have I understood this correctly or is there some other way to accomplish this?

Thanks!

score 3 · Accepted Answer · answered May 22 '19 at 14:24

Your understanding is correct, the structure size in C will equal the size of the largest member of the oneof. However this is only the size in memory, the message size after serialization will be the minimum needed for the contents.

If the size in memory is an issue, there are several options available. The default of allocating maximum possibly needed size makes memory management easy. If you want to dynamically allocate only the needed amount of memory, you'll have to decide how you want to do it:

Using PB_ENABLE_MALLOC compilation option, you can use the FT_POINTER field type for the string and other large fields. The memory will then be allocated using malloc() from the system heap.
With FT_CALLBACK, instead of allocating any memory, you'll get a callback where you can read out the string and process or store it any way you want. For example, if you wanted to write the string to SD card, you could do so without ever storing it completely in memory.

In overall system design, static allocation for maximum size required is often the easiest to test for. If the data fits once, it will always fit. If you go for dynamic allocation, you'll need to analyze more carefully what is the maximum possible memory usage needed.

How about if I use just bunch of optional fields instead of the oneof? I assume it would not matter after serialization, but in memory there would be few unneccessary null pointers for each data_item taking space right? — Ou Tsei, May 23 '19 at 07:06
@OuTsei By default, optional fields are also directly allocated in the structure. If they are configured pointers (type = FT_POINTER), it implies the memory is allocated separately, as described in this answer. — jpa, May 23 '19 at 07:48
So maybe instead have separate data_item specification for all types of data(float,int string etc) and have a repeated structure for all of those separately? — Ou Tsei, May 23 '19 at 10:10
@OuTsei And where are you going to get the memory for that repeated structure, if its size is not constant? It will always end up in either statically allocating for the maximum size, or dynamically allocating. You can structure the dynamic allocation in many ways, but it won't change its basic properties, such as more difficult testing. — jpa, May 23 '19 at 17:36

nanopb oneof size requirements

1 Answers1