In async/await, what's the most idiomatic/efficient way to take an input buffer and, for each item, route it to an existing Future or a new one based on a condition? For example, you're running a server and each incoming packet might either be the start of a new interaction with a client or a continuation of an existing one. You can tell the difference by something in the packet, such as a UUID identifying the interaction.
I'm trying to do this and it's working pretty well for the most part, but every time a new client interaction starts, the latency between the initial request and response increases and I can't figure out why. I'm trying it using these two structs that I made:
pub struct InputBuffer<T> {
queue_send: SyncSender<(T, Instant)>,
queue_recv: Arc<Mutex<Receiver<(T, Instant)>>>,
waker_send: SyncSender<Waker>,
waker_recv: Arc<Mutex<Receiver<Waker>>>,
}
pub struct InputBufferNext<'a, T, F: Fn(&T) -> bool>(pub InputBuffer<T>, pub &'a F);
The relevant functions on InputBuffer are below. Every time we add a new item to the queue using push, we wake all the interested wakers. The choose function takes a Fn(&T) -> bool and returns the next item that satisfies this condition. The choose_and_then function repeatedly chooses items off the queue based on a given condition and runs a given function on them. This is what I'm using to start new client interactions. I've got an Arc<Mutex<HashSet<u64>>> storing UUIDs for active client interactions and this is shared with a closure that returns whether or not the UUID is new and this is the F parameter. If it's not already in the set, then the G parameter is a function that starts a new interaction. (In my use case, the key is more complicated than a u64, but this is adapted for simplicity).
impl<T> InputBuffer<T> {
pub fn push(&self, t:T) {
self.queue_send.send((t, Instant::now())).unwrap();
if let Ok(waker_recv) = self.waker_recv.lock() {
while let Ok(waker) = waker_recv.try_recv() {
waker.wake();
}
}
}
pub async fn choose<F: Fn(&T) -> bool>(self, choose: &F) -> T {
InputBufferNext::<T, F>(Clone::clone(&self), choose).await
}
pub async fn choose_and_then<F: Fn(&T) -> bool, G: Fn(T)>(self, choose: F, and_then: G) {
loop {
and_then(self.clone().choose(&choose).await);
}
}
}
The type that actually implements Future is InputBufferNext and it does it the way shown below. Each time we poll it, we receive items until we find one that matches our condition, then put back the ones that don't. There's a timeout implemented to make sure the buffer can't just accumulate unclaimed items indefinitely. If we don't find what we're looking for, we send our waker to the InputBuffer.
impl<'a, T, F: Fn(&T) -> bool> std::future::Future for InputBufferNext<'a, T, F> {
type Output = T;
fn poll(self: Pin<&mut Self>, cx: &mut Context) -> Poll<Self::Output> {
let mut put_back = vec![];
if let Ok(queue_recv) = self.0.queue_recv.lock() {
let mut accepted: Option<T> = None;
'inner: while let Ok((x, t0)) = queue_recv.try_recv() {
if (self.1)(&x) {
accepted = Some(x);
break 'inner;
} else if t0.elapsed() < INPUT_BUFFER_TIMEOUT {
put_back.push((x, t0));
}
}
for x_t0 in put_back {
self.0.queue_send.send(x_t0).unwrap();
}
if let Some(x) = accepted {
return Poll::Ready(x);
}
}
// We made it all the way through what's available and didn't find what we were looking for
// Let us know when we should try again
self.0.waker_send.send(cx.waker().clone()).unwrap();
Poll::Pending
}
}
Since InputBuffer can be cloned (I had to implement Clone by hand because it didn't like the derive macro, but it's made up of all cloneable types), I just clone it and give it to each async fn that gets spawned with each new interaction. When an interaction wants its next item, it calls choose with a function that picks out its UUID and gets back a Future.
Like I said, this implementation seems to be mostly working, but there's some kind of latency accumulating somewhere. It takes longer and longer for the server to respond to initial requests. I've checked the amount of time that each poll call takes along with the number of calls per second and those both seem to be stable, despite the growing latency. I've also checked the size of the queue of wakers and it seems steady at a low, reasonable number. The waker queue also gets emptied every time a new item is added. The timeout also ensures that the number of items in the queue stays limited.
I'm using a timeout around 100ms and seeing the latency grow to unacceptable levels after 80+ seconds. My use cases starts a new client interaction about 30-50 times per second, to give a rough idea. Each interaction also ends after a short time (<1 sec), so we're not just accumulating active interactions indefinitely. I'm using the ThreadPool executor from futures.
Any suggestions for how I could improve my implementation or abandon it completely in favor of a more highly-optimized library or different design pattern would be much appreciated. Thanks!
