How to integrate Cap'n'Proto threads with non Cap'n'Proto threads?

Question

How do I properly integrate Cap'n'Proto client usage with surrounding multi-threaded code? The Cap'n'Proto docs say that each Cap'n'Proto interface is single-threaded with a dedicated event loop. Additionally they recommend using Cap'n'Proto to communicate between threads. However, the docs don't seem to describe how non-Cap'n'Proto threads (e.g. the UI loop) could integrate with that. Even if could integrate Cap'n'Proto event loops with the UI loop in some places, other models like thread pools (Android Binder, global libdispatch queues) seem more challenging.

I think the solution is to cache the thread executor for the client thread in a synchronized place that the non-capnp thread will access it.

I believe though that the calling thread always needs to be on its own event loop as well to marry them but I just want to make sure that's actually the case. My initial attempt to do that in a simple unit test is failing. I created a KjLooperEventPort class (following the structure for the node libuv adapter) to marry KJ & ALooper on Android.

Then my test code is:

TEST(KjLooper, CrossThreadPromise) {
  std::thread::id kjThreadId;
  ConditionVariable<const kj::Executor*> executorCv{nullptr};
  ConditionVariable<std::pair<bool, kj::Promise<void>>> looperThreadFinished{false, nullptr};

  std::thread looperThread([&] {
    auto looper = android::newLooper();
    android::KjLooperEventPort kjEventPort{looper};
    kj::WaitScope waitScope(kjEventPort.getKjLoop());

    auto finished = kj::newPromiseAndFulfiller<void>();
    looperThreadFinished.constructValueAndNotifyAll(true, kj::mv(finished.promise));

    executorCv.waitNotValue(nullptr);

    auto executor = executorCv.readCopy();
    kj::Promise<void> asyncPromise = executor->executeAsync([&] {
      ASSERT_EQ(std::this_thread::get_id(), kjThreadId);
    });
    asyncPromise = asyncPromise.then([tid = std::this_thread::get_id(), kjThreadId, &finished] {
      std::cerr << "Running promise completion on original thread\n";
      ASSERT_NE(tid, kjThreadId);
      ASSERT_EQ(std::this_thread::get_id(), tid);
      std::cerr << "Fulfilling\n";
      finished.fulfiller->fulfill();
      std::cerr << "Fulfilled\n";
    });
    asyncPromise.wait(waitScope);
  });

  std::thread kjThread([&] {
    kj::Promise<void> finished = kj::NEVER_DONE;
    looperThreadFinished.wait([&](auto& promise) {
      finished = kj::mv(promise.second);
      return promise.first;
    });

    auto ioContext = kj::setupAsyncIo();
    kjThreadId = std::this_thread::get_id();
    executorCv.setValueAndNotifyAll(&kj::getCurrentThreadExecutor());
    finished.wait(ioContext.waitScope);
  });

  looperThread.join();
  kjThread.join();
}

This crashes fulfilling the promise back to the kj thread.

terminating with uncaught exception of type kj::ExceptionImpl: kj/async.c++:1269: failed: expected threadLocalEventLoop == &loop || threadLocalEventLoop == nullptr; Event armed from different thread than it was created in.  You must use
 Executor to queue events cross-thread.

Kenton Varda · Answer 1 · 2020-07-05T17:17:18.790

Most Cap'n Proto RPC and KJ Promise-related objects can only be accessed in the thread that created them. Resolving a promise cross-thread, for example, will fail, as you saw.

Some ways you could solve this include:

You can use kj::Executor to schedule code to run on a different thread's event loop. The calling thread does NOT need to be a KJ event loop thread if you use executeSync() -- however, this function blocks until the other thread has had a chance to wake up and execute the function. I'm not sure how well this will perform in practice; if it's a problem, there is probably room to extend the Executor interface to handle this use case more efficiently.
You can communicate between threads by passing messages over pipes or socketpairs (but sending big messages this way would involve a lot of unnecessary copying to/from the socket buffer).
You could signal another thread's event loop to wake up using a pipe, signal, or (on Linux) eventfd, then have it look for messages in a mutex-protected queue. (But kj::Executor mostly obsoletes this technique.)
It's possible, though not easy, to adapt KJ's event loop to run on top of other event loops, so that both can run in the same thread. For example, node-capnp adapts KJ to run on top of libuv.

How to integrate Cap'n'Proto threads with non Cap'n'Proto threads?

1 Answers1