0

I have a Linux server which processes Cap'n Proto RPC requests. Some of those requests need to forward data in the request to another server running, in this case, a Kafka broker. The librdkafka and Cap'n Proto KJ libraries can both use poll() so i think the OS will ensure they can both run asynchronously but I'm not sure if further integration is required or beneficial. Does anyone have experience with this?

The question is a bit broader than the specifics i've listed.. there are other APIs which i may call in future from Cap'n Proto RPC so any broad guidelines would be appreciated.

James Fremen
  • 2,170
  • 2
  • 20
  • 29

1 Answers1

2

Unfortunately, this is not so simple. Yes, they both use poll(), but the problem is, only one library will be calling poll() at a time, and only that library will actually receive any events -- the other is stuck. This is a classic challenge with event loop libraries -- by default they cannot be used together.

One option is to try to use the libraries in separate threads. But, often event-driven libraries design around the assumption that you're doing everything in a single thread because otherwise why would you need an event loop?

But "The Right Thing" is to integrate the event loops. KJ's event loop is capable of being integrated with other event libraries. For example, I integrated it with libuv for node-capnp; see the first part of this file:

https://github.com/kentonv/node-capnp/blob/master/src/node-capnp/capnp.cc

(At some point I plan to separate out the libuv-related code here into a separate library shipped with Cap'n Proto.)

For another example, here's a pull request by Nathan Hourt to add integration with Qt's event loop -- but note that this one doesn't include I/O integration, I think because Nathan is using an implementation of AsyncIoStream into which he manually pushes data when it's available:

https://github.com/sandstorm-io/capnproto/pull/253

In any case, you'll need to do something similar with whatever Kafka uses. Hopefully you'll then contribute your code back to Cap'n Proto! :)

Kenton Varda
  • 41,353
  • 8
  • 121
  • 105
  • thanks Kenton. I looked at the node implementation but was a bit cautious because it adds libuv into the mix and it was considered slow and a bit kludgy. I'm tempted to move the Kafka bits into a new lib which links with KJ's interfaces. It would be nice if the RPC pipeline could access the new functionality with zero-copy, too, but i'm not sure how hard this is. Does this sound viable to you? I'm in an early-stage startup so we may not be able to open-source until we get some investors :). – James Fremen Oct 15 '15 at 05:34
  • librdkafka provides a non-blocking asynchronous interface, all IO operations are performed in background threads, so you should be fine simply relying on Cap'n's event dispatcher as your main loop. – Edenhill Oct 15 '15 at 10:59
  • @JamesFremen I pointed out the libuv implementation as an example of how to integrate KJ with an external event loop. I'm not aware of any performance issues or other major problems in the KJ-UV glue code. It's fully zero-copy. Unless Kafka's low-level event interface is very bad, you should be able to integrate with it in this way just fine. – Kenton Varda Oct 15 '15 at 21:58
  • @Edenhill Even if there is a background thread doing the I/O (which sounds surprising), it would presumably need some way to deliver events to the main thread, which requires some sort of event loop integration. – Kenton Varda Oct 15 '15 at 21:59
  • @Edenhill thanks for responding. I'm looking for a single-threaded implementation since i think it may provide finer-grained resource allocation with sufficient performance for my workload. – James Fremen Oct 16 '15 at 03:18
  • @KentVarda yes, i judged it based on my initial pass of the top-level project comments which state that it is slow and a short-term hack, words that i would never normally use :). I will probably have more questions as i climb the learning curve but so far so good. – James Fremen Oct 16 '15 at 03:28
  • @JamesFremen - Yeah, those comments are about the node capnp bindings as a whole, *not* the libuv glue specifically. (And they may be too harsh; the performance has not been any kind of problem for Sandstorm.) – Kenton Varda Oct 16 '15 at 05:42
  • @KentonVarda Producing Kafka messages is an async operation that returns immediately (rd_kafka_produce*()). Consuming messages is using a pull based mechanism (msg = rd_kafka_consume*(.., timeout_ms)) that you need to call from your application code. Additionally there is error reporting and delivery reports provided by rd_kafka_poll() that typically goes in your main loop. – Edenhill Oct 16 '15 at 08:14
  • @KentonVarda Internal IO threads are used for performance reasons and to provide a non-blocking interface to the application. – Edenhill Oct 16 '15 at 08:16