How to use future::join_all with the multiplexed redis in the async tokio runtime

Question

I'm trying to use the Rust redis client in the asynchronous multiplexed mode, with tokio as the async runtime, and dynamic number of futures to join.

I had success using future::join3 on a constant number of futures, but I want to multiplex many more commands (the specific size should not have to be known in compile-time, but even that would be an improvement).

This is the working example when using future::join3; The example correctly prints Ok(Some("PONG")) Ok(Some("PONG")) Ok(Some("PONG"))

Cargo.toml

[package]
name = "redis_sample"
version = "0.1.0"
authors = ["---"]
edition = "2018"


[dependencies]
redis = { version = "0.17.0", features = ["aio", "tokio-comp", "tokio-rt-core"] }
tokio = { version = "0.2.23", features = ["full"] }
futures = "0.3.8"

src/main.rs

use futures::future;
use redis::RedisResult;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let redis_client = redis::Client::open("redis://127.0.0.1:6379")?;
    let mut redis_connection = redis_client.get_multiplexed_tokio_connection().await?;

    let results: (RedisResult<Option<String>>, RedisResult<Option<String>>, RedisResult<Option<String>>) = future::join3(
        redis::cmd("PING").query_async(&mut redis_connection.clone()),
        redis::cmd("PING").query_async(&mut redis_connection.clone()),
        redis::cmd("PING").query_async(&mut redis_connection),
    ).await;

    println!("{:?} {:?} {:?}", results.0, results.1, results.2);

    Ok(())
}

Now I want to do the same, but with n commands (let's say 10, but ideally I'd like to tune this to performance in production). This is as far as I got, but I'm unable to overcome the borrow rules; I tried storing some intermediaries (either the redis Cmd or the future itself) in a Vec to prolong their life, but that had other issues (with multiple mut references).

The Cargo.toml is the same; here's main.rs

use futures::{future, Future};
use std::pin::Pin;
use redis::RedisResult;

const BATCH_SIZE: usize = 10;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let redis_client = redis::Client::open("redis://127.0.0.1:6379")?;
    let redis_connection = redis_client.get_multiplexed_tokio_connection().await?;

    let mut commands: Vec<Pin<Box<dyn Future<Output = RedisResult<Option<String>>>>>> = vec![];
    for _ in 0..BATCH_SIZE {
        commands.push(Box::pin(redis::cmd("PING").query_async(& mut redis_connection.clone())));
    }
    let results = future::join_all(commands).await;

    println!("{:?}", results);

    Ok(())
}

I'm getting two compiler warnings (creates a temporary which is freed while still in use), and I don't know how to move forward with this code. I'm not 100% sold into using Pin, but I wasn't able to even store the futures without it.

Full compiler output:

   Compiling redis_sample v0.1.0 (/Users/gyfis/Documents/programming/rust/redis_sample)
error[E0716]: temporary value dropped while borrowed
  --> redis_sample/src/main.rs:14:32
   |
14 |         commands.push(Box::pin(redis::cmd("PING").query_async(& mut redis_connection.clone())));
   |                                ^^^^^^^^^^^^^^^^^^                                              - temporary value is freed at the end of this statement
   |                                |
   |                                creates a temporary which is freed while still in use
...
21 | }
   | - borrow might be used here, when `commands` is dropped and runs the `Drop` code for type `std::vec::Vec`
   |
   = note: consider using a `let` binding to create a longer lived value

error[E0716]: temporary value dropped while borrowed
  --> redis_sample/src/main.rs:14:69
   |
14 |         commands.push(Box::pin(redis::cmd("PING").query_async(& mut redis_connection.clone())));
   |                                                                     ^^^^^^^^^^^^^^^^^^^^^^^^   - temporary value is freed at the end of this statement
   |                                                                     |
   |                                                                     creates a temporary which is freed while still in use
...
21 | }
   | - borrow might be used here, when `commands` is dropped and runs the `Drop` code for type `std::vec::Vec`
   |
   = note: consider using a `let` binding to create a longer lived value

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0716`.
error: could not compile `redis_sample`.

Any help appreciated!

Sabrina Jewson · Accepted Answer · 2020-12-21T16:50:41.563

This should work, I just extended the lifetime of redis_connection.

use futures::{future, Future};
use std::pin::Pin;
use redis::RedisResult;

const BATCH_SIZE: usize = 10;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let redis_client = redis::Client::open("redis://127.0.0.1:6379")?;
    let redis_connection = redis_client.get_multiplexed_tokio_connection().await?;

    let mut commands: Vec<Pin<Box<dyn Future<Output = RedisResult<Option<String>>>>>> = vec![];
    for _ in 0..BATCH_SIZE {
        let mut redis_connection = redis_connection.clone();
        commands.push(Box::pin(async move {
            redis::cmd("PING").query_async(&mut redis_connection).await
        }));
    }
    let results = future::join_all(commands).await;

    println!("{:?}", results);

    Ok(())
}

Since you're inside a function body you don't even need to box the futures, type inference can do all the work:

use futures::future;
use redis::RedisResult;

const BATCH_SIZE: usize = 10;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let redis_client = redis::Client::open("redis://127.0.0.1:6379")?;
    let redis_connection = redis_client.get_multiplexed_tokio_connection().await?;

    let mut commands = vec![];
    for _ in 0..BATCH_SIZE {
        let mut redis_connection = redis_connection.clone();
        commands.push(async move {
            redis::cmd("PING").query_async::<_, Option<String>>(&mut redis_connection).await
        });
    }
    let results = future::join_all(commands).await;

    println!("{:?}", results);

    Ok(())
}

Thank you! The first example works well, that's wonderful. Do you mind describing a little bit more why you need to `async move {` and why you store the result of `.await`, or maybe link to the book/explanation somewhere? I'm probably looking at futures wrong, but I thought `.await` waits until the future finishes. The second example doesn't compile for me unfortunately, with two compiler errors on the `redis::cmd("PING").query_async` line - both are these: `error[E0698]: type inside \`async\` block must be known in this context` Thanks! — Gyfis, Dec 20 '20 at 20:36
I needed to async move to move the redis connection in to the future. Otherwise, the future would borrow the redis connection, but that would error as the future needs to outlive the redis connection - it needs to live longer than one iteration of the loop. Yes, you're right that await waits until the future is done, but an async block creates a new future and so doesn't run immediately. I'll fix the answer to make it compile. — Sabrina Jewson, Dec 21 '20 at 16:48

How to use future::join_all with the multiplexed redis in the async tokio runtime

1 Answers1