Cap’n Proto C++ serialize to char array (or any byte array)

Question

My goal is to send serialized data through MPI. I have done this with ProtoBuf but I would like to try to use a faster serialize method such as Cap’n Proto (I will try others as well but here I am stuck). With ProtoBuf I use the function SerializeToArray(void * data, int size) function which works just fine.

Now, I want to do the same thing but with Cap’n Proto but I cannot find anywhere how to do this (if you have a link please send it). Since Cap’n Proto claims to be a faster substitute for ProtoBuf I find this surprising. Maybe, I am going about this the completely wrong way.

So my question becomes:

How do I serialize to char array (or any byte array) with Cap’n Proto (if it is at all possible)? Or how do I serialize in a way which could easily be sent over MPI using C++?

I guess you can use Flat Array variants for this. Can you please check this [snippet](https://godbolt.org/z/f-Hj11) modified from the example from the [documentation](https://capnproto.org/cxx.html)? Look at the end of the second half of the function. Maybe, this is what you're looking for. Let me know. — Azeem, Apr 01 '20 at 18:24
Yes this got me on the right track and I have solved it now. Thank you! — Rickard Johansson, Apr 02 '20 at 15:13
That's awesome! :) You're welcome! It took me quite some time to figure everything out and then test it. Glad it helped! I'll post the code with some explanation as an answer shortly. — Azeem, Apr 02 '20 at 15:16

score 1 · Answer 1 · edited Apr 02 '20 at 16:07

I am not sure if this is the most efficient way to do this but it works.

messages.capnp file:

@0xf46c6bd8234dfab9;

struct Testmessage {
  string @0 :Text;
  float @1 :Float32;
  int @2 :Int32;
}

run_cap_n_proto.cpp file:

#include <iostream>
#include <capnp/serialize.h>
#include "messages.capnp.h"

int main()
{
    // Encode message
    ::capnp::MallocMessageBuilder message_builder;
    Testmessage::Builder message = message_builder.initRoot<Testmessage>();

    message.setString( "string" );
    message.setFloat( 3.14 );
    message.setInt( 1337 );

    auto encoded_array = capnp::messageToFlatArray(message_builder);
    auto encoded_array_ptr = encoded_array.asChars();
    auto encoded_char_array = encoded_array_ptr.begin();
    auto size = encoded_array_ptr.size();

    // Send message
    // Receive message

    // Decode message
    auto received_array = kj::ArrayPtr<capnp::word>(reinterpret_cast<capnp::word*>(encoded_char_array), size/sizeof(capnp::word));
    ::capnp::FlatArrayMessageReader message_receiver_builder(received_array);
    auto message_receiver = message_receiver_builder.getRoot<Testmessage>();
    auto s_r = message_receiver.getString().cStr();
    auto f_r = message_receiver.getFloat();
    auto i_r = message_receiver.getInt();

    std::cout << "received: " << s_r << ", " << f_r << ", " << i_r << std::endl;
}

To compile messages.capnp:

$ capnp compile -oc++ messages.capnp

To comile the main program:

$ g++ -o run_cap_n_proto run_cap_n_proto.cpp messages.capnp.c++ `pkg-config --cflags --libs capnp`

+1. I cleaned up the code a bit for better readability. Can you tell me why you're casting to `kj::ArrayPtr` on the receiver's end? If you could send the same from the sender's end, wouldn't that be a better alternative because `encoded_array.asPtr()` returns the `kj::ArrayPtr`? What do you think? — Azeem, Apr 02 '20 at 16:10
Awesome! That sounds great and i will definetly do that. The reason is that i am fairly new to c++ and very new to cap n proto. I find it hard to find good documentation on kj::array and kj::arrayptr and their differences. I have mainly looking att other examples because to a lot of this is still magic. I have been reading the array.h and serialize. h files but that is taking ages for a newbi like me so I am really happy you are pointing these things out! — Rickard Johansson, Apr 03 '20 at 12:57
Right. Keep going! I saw your `reinterpret_cast` and suggested you could do better by using the right type from the start. BTW, you could look at `std::array`, `std::unique_ptr`, and `std::make_unique` to get a better understanding of the equivalent stuff from `kj` namespace. Good luck! — Azeem, Apr 03 '20 at 13:04

score 1 · Accepted Answer · answered Apr 02 '20 at 15:52

For the message given in this AddrssBook example from documentation, you can do something like this:

// Documentation: https://capnproto.org/cxx.html
// AddressBook example

void sendMessage( const char* data, const std::size_t size ); 

void writeAddressBook()
{
    ::capnp::MallocMessageBuilder message;

    auto addressBook = message.initRoot<AddressBook>();
    auto people = addressBook.initPeople(1);

    auto alice = people[0];
    alice.setId(123);
    alice.setName("Alice");
    alice.setEmail("alice@example.com");

    auto alicePhones = alice.initPhones(1);
    alicePhones[0].setNumber("555-1212");
    alicePhones[0].setType(Person::PhoneNumber::Type::MOBILE);
    alice.getEmployment().setSchool("MIT");

    // get char array and send

    const auto m = capnp::messageToFlatArray( message );
    const auto c = m.asChars();
    std::cout << c.size() << '\n';

    sendMessage( c.begin(), c.size() ); // pass as char array
}

Cap’n Proto C++ serialize to char array (or any byte array)

2 Answers2