1

I am sending a message over ZeroMQ PUB/SUB archetype, using a tcp:// transport-class channel, after serializing a capnp message using capnp::messageToFlatArray. On the receiving side I receive the entire content in a zmq_msg_t message. But zmq_msg_data(&message) returns a memory location which is not capnp::word aligned. So FlatArrayMessageReader is throwing exception that the memory is not aligned.

Simplified code looks like this:

    zmq_msg_t message;
    zmq_msg_init(&message);

    zmq_msg_recv(&message, socket, flags);

    size_t size = zmq_msg_size(&message);
    auto data = zmq_msg_data(&message);
    auto pdata = kj::arrayPtr((const capnp::word*)data, size / sizeof(capnp::word));
    capnp::FlatArrayMessageReader msg = capnp::FlatArrayMessageReader(pdata);  // exception

What would be a good way to get the data aligned without copying the entire buffer? Or is there a way to receive the message in a word aligned memory without performance penalty - like disabling zero copy in zmq?

Trying on ubuntu 18.04 with capnp version 0.7.0, zeromq version 4.3.2 and gcc 7.4.0.

user3666197
  • 1
  • 6
  • 50
  • 92
Maruf Maniruzzaman
  • 441
  • 3
  • 13
  • Why are you sure that it is an alignment problem, the last time I had that was 25 years ago on SPARC processors? Also IMHO ditch zeromq - I wasted 3 years on it and it is a POS – Adrian Cornish Feb 02 '20 at 19:07
  • @AdrianCornish Appreciate your honest opinion about zmq :). And the exception itself says the memory is not aligned (ctor throws) and I can verify the message data buffer is not aligned as well using modulus. – Maruf Maniruzzaman Feb 03 '20 at 15:46
  • This is kind of funny to see my unpublished message being posted on stackoverflow. I have written the question but decided I do not want to post it. Or did I actually click on publish button? Who knows? – Maruf Maniruzzaman Feb 03 '20 at 15:51

1 Answers1

1

Q : What would be a good way to get the data aligned without copying the entire buffer?

Well, given the ZeroMQ (as-is) is based on an autonomously working Context()-instance engine, that has promised a Zero-Copy on its own right for the same, performance motivated reasons, the storage location of a message gets determined by the Context() internal policies, that do not "see", the less "obey" any of the capnp-preferences.

Unless one refactors not only the zmq_msg_init() per-se, but all the related performance-motivated internalities inside ZeroMQ, so as to externally "enforce" and "keep" some sort of ( here capnp-motivated, causing an exemplary "conflict-of-interests" ) higher-level indoctrinated memory-management ( aligned-allocation-, re-use- and release-policies ), there seems to be Zero-Chance for expecting such behaviour from the already mature, smart and slim (as-is) and right-sized, feature just-enough equipped tool, that was designed for providing fast, minimum-latency yet almost linear scalable messaging/signalling tool.

Q : is there a way to receive the message in a word aligned memory without performance penalty - like disabling zero copy in zmq?

AFAIK, never 've met such configuration "across" the published API(since v2.+ till v4.3 as of 2020/Q1).

May try to scan the source code with comments, where any such trick could have appeared to be put in-place.

user3666197
  • 1
  • 6
  • 50
  • 92
  • Yes, life is complicated. You can not make everybody happy. Thanks for taking time to explain. I had come to similar conclusion. For larger message the memory seems to be aligned perfectly - probably because allocation will not find anything or is an optimization effort by allocator itself for larger chunk of memory. Decided to copy message if its not aligned (which seems to be smaller most of the time anyway) for now and come back if it becomes an issue. BTW, disabling zerocpy may be an option - https://github.com/zeromq/libzmq/issues/3448 - but thats worse anyway. – Maruf Maniruzzaman Feb 03 '20 at 15:55