3

We have the need to send data between very different software components (from embedded devices to Web services). For various reasons (the fact that we got Mosquitto to compile on the most problematic platform not being the least of them), we decided to use the MQTT message brokering protocol. It's a rather lightweight protocol which provides an application-defined payload for messages to be published. We now need to find a way to specify the data in that payload.

The current idea is to use some existing IDL that brings the ability to access the data from at least C++ and Java, ideally also Python and others languages. If this would come with encoding/decoding APIs for these platforms which can be used to serialize the data into, and deserialize it out of, the MQTT protocol's payload, this would be ideal.

Is there anything like this available?

Note:

  • There's to be a lot of data. In fact, we will likely need to cut it down from the current specifications in order to avoid network congestion. An XML format might be too talkative for this.
  • The format should be easy to encode and decode, since on embedded platforms CPU shares are valuable. (That might be another reason to not so send XML over the wire.)
  • If any of the components needed for this (IDL compiler etc.) is not cross-platform, it should run on Windows.
  • I'm not a networking expert. If IDLs are a thing of the last decade, the technology has moved on, and there's better ways to move data between platforms over a network, go ahead and suggest it.
sbi
  • 219,715
  • 46
  • 258
  • 445
  • How about JSon ? it's plain text and quite easy to decode – r4phG Jul 21 '15 at 13:47
  • I'm not a big fan of the complexities that an IDL adds to a project. I use XML and JSON documents/objects across network links and use custom binary encapsulation of data and compression for efficiency in certain scenarios. It's easy to handle versioning. I tend to lean toward JSON mostly now, for ease of use in communication and presentation with web-based technologies. JSON can be slightly less "talkative" than XML but not significantly. Parser and encoder is easy to handle in object-oriented code with simple libraries. – mark Jul 21 '15 at 14:05
  • @r4phG: I dunno. I haven't had the need to use JSON, but from what I know it's probably usable from just about any language/platform. I am afraid, though, that textual protocols generate too many bytes and take too long to parse. What do you think? – sbi Jul 21 '15 at 14:06
  • Well, as @Mark mentionned, it's just a bit less talkative than XML, but still it's cross platform and easy to use. But I don't really the limit of "too many bytes" in your case :) – r4phG Jul 21 '15 at 14:09
  • @r4phG: Currently we acquire enough data that limiting factors of how much we can produce are the machines' ability to process it and to write the data to local flash memory fast enough. You also need consider that some of the networks we want to do this across operate in high-EMI environments some of which might prevent GB Ethernet from operating reliably. Plus, how much CPU percentage will parsing JSON consume on an embedded device with a RT OS that has to schedule high-priority tasks (acquiring data via industrial busses) with single-digit ms cycles? – sbi Jul 21 '15 at 14:24
  • @sbi Ok, I get it. But still, even if you find something smaller, if it's big with XML/JSon or whatever, I guess the lose of volume won't be that signifivative with another one. So your widthband issue would be hard to fix. For the parsing issue, instead of thinking of the amount of data you'd have to parse, maybe thinking of lightening the CPU usage by thinking of how to compute the data could help ? (Threading/cutting data in independant parts..). I'm not an expert, just trying to give some ideas here as the subject's interesting – r4phG Jul 21 '15 at 15:05

3 Answers3

2

You can use google protocol buffers (essentially this is kind of "binary" XML with a lot less overhead).

AFAIK there exist c++, java and python libraries.

For embedded c++ systems you can use the NanoPb implementation which is a light-weight google protocol buffer implementation in pure C code (we're currently using that library in our embedded system)

see https://developers.google.com/protocol-buffers/

Daniel
  • 1,041
  • 7
  • 13
1

You also can check Apache Thrift, Cap'n'proto, Msgpack

Sergei Nikulov
  • 5,029
  • 23
  • 36
  • Thanks. I have not yet looked sufficiently at Apache Thrift. Cap'n Proto looks very interesting, but in C++ it needs C++11 (which we don't have on all platforms). In MessagePack, as in other non-compiled data description languages (like JSON), all the data is dynamically typed. (Which makes it comparatively CPU-intense to work with, since it needs to be converted from strings. – sbi Jul 21 '15 at 17:13
0

I had search webservice philosophy with preferred code generation (definition)->server service code and (definition)->client of service (SOAP client generation from WSDL was tested previously). In this way I found Apache Thrift (btw Thrift IDL compiler is Win application). I had never practical contact with CORBA with IDL, only read few materials.

My solution is quite small and high speed of binary protocols wasn't too important, but IDL and many generated languages including ancient Delphi is very, very nice. Main languages at first are Java and C# - C++, Deplhi and dynamic languages is future option.

Jacek Cz
  • 1,872
  • 1
  • 15
  • 22