1

I implemented a small HttpClient for an IoT device and wanted to use Proto Buffer as a communication format. Because of the constraints of the platform I am using nanopb. That's the relevant code in C:

#include <pb_encode.h>
#include "device_data.pb.h"
#include <ESP8266HTTPClient.h>

[...]
pb_MEvent m_event = pb_MEvent_init_zero;
uint8_t m_buffer[21];
pb_ostream_t stream = pb_ostream_from_buffer(m_buffer, 21);
pb_encode(&stream, pb_MEvent_fields, &m_event);

int httpCode = httpClient.POST(m_buffer, stream.bytes_written);
[...]

I created a small flask server which exposes an endpoint. When I try to decode the message, I get the following error: google.protobuf.message.DecodeError: Error parsing message with type 'pb.MEvent'

That the code for it:

from flask import Flask, request

from device_data_pb2 import MEvent

app = Flask(__name__)

@app.route("/", methods = ['POST', "GET"])
def hello_world():
    m_event = MEvent()

    m_event.ParseFromString(request.data)
    print(m_event)

if __name__ == '__main__':
    app.run(host='0.0.0.0', port= 5000)

I tried to use io.ByteStream and read it then and I also tried to encode the bytes with ASCII and UTF-8 but neither of that approaches worked.

Could you please help me find out, what is causing the problem? The error message is not that helpful.

Update

This is the content fo the proto file:

syntax = "proto2";
package pb;

message MEvent {
    required float accelX = 1;
    required float accelY = 2;
    required float accelZ = 3;
    required float gyroX = 4;
    required float gyroY = 5;
    required float gyroZ = 6;
    required int64 msec = 7;
}

And here are some data points:

DATA:
0.16 -0.08 9.96 0.00 -0.00 0.02 0
HEX:
0d:98:d7:27:3e:15:bf:f7:ad:bd:1d:46:70:1f:41:25:b5:33:70:3b:2d

DATA:
0.16 -0.09 9.96 0.00 -0.00 0.02 0
HEX:
0d:cd:7d:20:3e:15:56:ab:bc:bd:1d:5f:6b:1f:41:25:79:22:a0:3b:2d


DATA:
0.15 -0.10 9.96 0.00 -0.00 0.02 0
HEX:
0d:89:0a:1e:3e:15:21:05:c4:bd:1d:de:52:1f:41:25:b5:33:70:3b:2d
ZenCoding
  • 105
  • 8
  • 1
    Please include the message definition for `MEvent` from .proto file and a hex dump of the raw data in `request.data`. – jpa Nov 30 '21 at 10:01
  • I added the float and int data and their corresponding hex representation. I printed it on the board and at the flask server, there is no difference. So the communication between board and server works, it seems like there is a problem with the data itself. – ZenCoding Nov 30 '21 at 12:05

1 Answers1

1

DATA: 0.16 -0.08 9.96 0.00 -0.00 0.02 0 HEX: 0d:98:d7:27:3e:15:bf:f7:ad:bd:1d:46:70:1f:41:25:b5:33:70:3b:2d

I'm using the nanopb/tests/raw_decode to analyze this. Marc Gravell's decode utility used to be great for this but for some reason it does not work very well for corrupted data anymore.

user@host:~/nanopb/tests$ echo 0d:98:d7:27:3e:15:bf:f7:ad:bd:1d:46:70:1f:41:25:b5:33:70:3b:2d \
| tr -d ':' | xxd -r -p | build/raw_decode/raw_decode

At 0: field tag 1, wire type 5 (32BIT), fixed32 value (4 bytes): 0x3e27d798
At 5: field tag 2, wire type 5 (32BIT), fixed32 value (4 bytes): 0xbdadf7bf
At 10: field tag 3, wire type 5 (32BIT), fixed32 value (4 bytes): 0x411f7046
At 15: field tag 4, wire type 5 (32BIT), fixed32 value (4 bytes): 0x3b7033b5
At 20: field tag 5, wire type 5 (32BIT)
ERROR: Failed to parse fixed32: io error
LATEST BYTES READ (21 to 21):

It appears your message is cut short. The amount of buffer space you reserve on this line is not large enough:

uint8_t m_buffer[21];

If you check the return value of pb_encode() it is probably also returning false and stream.errmsg indicates that the buffer is full.

jpa
  • 10,351
  • 1
  • 28
  • 45
  • Thanks for your answer! For some reason in the past, stream.bytes_written returned 21 and I took that as a constant. I reevaluated this and now it works. – ZenCoding Nov 30 '21 at 12:32
  • 1
    Yeah, protobuf is a variable length format. Nanopb generates a `#define MEvent_size xxx` in the header that you can use for the buffer size, it is the maximum encoded size that message type will have. – jpa Nov 30 '21 at 12:34