Junk values received from Event Hubs when using Azure EventHostProcessor Receiver

Question

I'm using Qpid-Proton-0.10(Python)- AMQP 1.0 based messaging library, to send data to Azure Event Hubs. To receive the data I use EventProcessorHost as per the following guide: https://azure.microsoft.com/en-in/documentation/articles/event-hubs-csharp-ephcs-getstarted/#receive-messages-with-eventprocessorhost. When received data through that, I see that two junk values getting attached before the data:

I don't see such characters while receiving the same using Qpid-Proton's recv.py. Is anyone facing the same problem?

P.S: I also used Azure Python SDK to send data to the Azure Event hubs and received it using both EventProcessorHost and recv.py of Qpid-Proton and faced no such problem. Only because Azure SDK doesn't contain any method to receive Events, I've to switch over to Qpid-Proton library.

Answer 1

I encountered the same issue using qpid-proton v0.29 for C++ to write a Microsoft Event Hub producer application and Microsoft.Azure.EventHubs v4.2 to write a Microsoft Event Hub consumer. The first "junk" byte at the beginning of the message body is the AMQP "embedded constructor," which specifies the format of the bytes that follow. In your screen shot, it looks like the body format is a string, e.g. "hello world". In this case, the "embedded constructor" is 0xA1, which indicates that the following bytes are ASCII and there is a variable number of them. The second "junk" byte is the number of bytes in the ASCII string. For "hello world", this would be 11. I'm not sure why the Microsoft library doesn't parse this for you. For more information on the AMQP data types and message format, see the following:

AMQP Data Types: http://docs.oasis-open.org/amqp/core/v1.0/os/amqp-core-types-v1.0-os.html#section-types

AMQP Message Format: http://docs.oasis-open.org/amqp/core/v1.0/os/amqp-core-messaging-v1.0-os.html#section-message-format

The AMQP specification states that the message body can be of type "AMQP Value", "Data", or "AMQP Sequence". I found that the Microsoft library doesn't like it if the message body is of type "AMQP Value". The newer Microsoft library Azure.Messaging.EventHubs v5.0.1 won't even parse the body data if it's an "AMQP value". It always shows 0 bytes in the message body. I was able to fix the issue by making sure the message body created by qpid-proton uses the binary "Data" type. I haven't tried it in Python, but here is what the C++ code looks like.

proton::message msg;
std::string bodyString = "Hello World!";
proton::binary bodyBinary(bodyString.begin(), bodyString.end());
msg.body(bodyBinary);

In Python, I would try making sure the message body has the "bytes" data type and see if that works.

Answer 2

I wanted to comment on Emily's answer, but I can't.

I had a similar problem and her answer helped a lot. The only thing that was missing is that you also need to set the inferred flag to true.

I am also working in C++:

proton::message msg;
std::string bodyString = "Hello World!";
msg.body(proton::binary(bodyString));
msg.inferred(true);

From the documentation:

The inferred flag for a message indicates how the message content is encoded into AMQP sections. If the inferred is true then binary and list values in the body of the message will be encoded as AMQP DATA and AMQP SEQUENCE sections, respectively. If inferred is false, then all values in the body of the message will be encoded as AMQP VALUE sections regardless of their type.

See the documentation here for C++ and here for Python.

Answer 3

I tried to reproduct your issus by using qpid-proton to send messages and using EventProcessorHost to receive messages, but I am failed. However, I think it seems to be caused by the Python Character Encoding, like the code using Unicode format in Python2 message.body = u"This is a text string". Please refer to https://msdn.microsoft.com/en-us/library/azure/jj841070.aspx for using qpid-proton in Python.

Per my experience, I suggest you to use the Service Bus/EventHubs APIs of the Azure SDK for Python or use EventHub REST API to send messages. It's a simple and stable way for EventHubs. About EventHub message sent REST API, please refer to https://msdn.microsoft.com/en-us/library/dn790664.aspx.

By now, using EventProcessorHost APIs in C# is the most effective way for receive messages from EventHubs, not need to be warried about the issues of Character Encoding.

Best Regards.