1

I want to process a vec of items concurrently. Basically, each item involves doing some I/O, but they are not dependant on one another. I could use futures::join_all (or in my case futures::try_join_all) to achieve that.

Since I don't care about processing results (except for the error in try_join_all case), I don't really want a vec of units (Vec<()>) in the end; It's just a useless allocation, where () (or Result<(), Error>) would be sufficient.

futures crate docs mentioned using FuturesUnordered directly, if needed, so I gave it a try (playground):

use futures::{
    stream::{FuturesUnordered, StreamExt},
    Future,
};

fn main() {
    tokio::spawn(async move {
        let foos = [Foo, Foo, Foo];
        join_all_discard(foos.iter().map(|foo| process_foo(foo))).await;
    });
}

async fn process_foo(foo: &Foo) {
    // do something async with foo
}

async fn join_all_discard<I>(iter: I) -> ()
where
    I: IntoIterator,
    I::Item: Future<Output = ()>,
{
    let mut stream: FuturesUnordered<_> = iter.into_iter().collect();
    while let Some(()) = stream.next().await {}
}

The error is

error: higher-ranked lifetime error
  --> src/lib.rs:9:5
   |
9  | /     tokio::spawn(async move {
10 | |         let foos = [Foo, Foo, Foo];
11 | |         join_all_discard(foos.iter().map(|foo| process_foo(foo))).await;
12 | |     });
   | |______^
   |
   = note: could not prove `[async block@src/lib.rs:9:18: 12:6]: std::marker::Send`

Compiler error manifests itself only with the call to tokio::spawn present (which kinda makes sense, since it possibly needs to send the future to a different thread)

Using join_all_discard(foos.iter().map(process_foo)).await (without the closure) eliminates the error, as well as using futures::join_all, yet my own implementation is flawed. I am lost. I suspect something has to do with the generic bounds on join_all_discard.

P.S. To solve the real problem I wrote try_join_all_discard, which exhibits the same error, and looks like this:

async fn try_join_all_discard<I, E>(iter: I) -> Result<(), E>
where
    I: IntoIterator,
    I::Item: Future<Output = Result<(), E>>,
{
    let mut stream: FuturesUnordered<_> = iter.into_iter().collect();
    loop {
        match stream.next().await {
            Some(Ok(())) => continue,
            Some(Err(e)) => break Err(e),
            None => break Ok(()),
        }
    }
}
Link0
  • 655
  • 5
  • 17
  • 1
    FWIW `Vec` where `ZST` is any zero sized type like for example `()` but others too, of any capacity [does not allocate](https://doc.rust-lang.org/std/vec/struct.Vec.html#method.with_capacity) – cafce25 Apr 14 '23 at 13:51
  • Also FWIW [`FuturesUnordered` _does_ allocate](https://docs.rs/futures-util/0.3.28/src/futures_util/stream/futures_unordered/mod.rs.html#166), and since it uses a linked list it's arguably worse than a `Vec`… – Jmb Apr 14 '23 at 13:56
  • The issue seems to be manipulating borrowed Foos, not entirely sure why but if you switch to `into_iter` and `process_foo` consuming the `Foo`, the issue is fixed. – Masklinn Apr 14 '23 at 13:57
  • @cafce25 huh, true. Didn't know that. Somehow I thought that for vec of ZSTs `vec[0] as * const () != vec[1] as * const ()`. – Link0 Apr 14 '23 at 13:57
  • Also I think `FuturesUnordered` is unnecessary, you shoudl be able to use `stream::iter` and `StreamExt::for_each_concurrent`. – Masklinn Apr 14 '23 at 13:59
  • And depending on the amount of setup code in the real case, you might get away with just returning `iter(foos).for_each_concurrent(None, process_foo)` from the sync block (that's a future, it doesn't need to be future-ized). Already the case with your `join_all_discard`. Though in both cases that assumes all the setup is sync *or* can be executed in the parent task if there's one. – Masklinn Apr 14 '23 at 14:03

1 Answers1

2

Change the trait bounds as such

async fn join_all_discard<I, F>(iter: I) -> ()
where
    I: IntoIterator<Item = F>,
    F: Future<Output = ()>
protwan
  • 139
  • 5
  • Yeah, It does work, but I'm hella confused as to why – Link0 Apr 15 '23 at 15:41
  • @Link0 I kinda understand it, but I am not sure so take this with a grain of salt. I guess it is something like `I::Item` implementing a trait isn't the same as it being a certain type `F` that implements the trait. I don't understand it well enough to put it all in words and give a proper explanation. – protwan Apr 15 '23 at 20:39
  • Maybe that `I::Item: Future` means items are only of `Future` trait, but `` means that `F` is specialized to a concrete type, which also may exhibit properties of `Send`. This concrete specialization may allow rustc to see through the async function, and determine, that in this specific case, it is `Send`. Compiler does "auto Send implementation" for futures of async fns, but only if it could prove that everything that fn deals with is `Send` itself. This looks like a limitation (bug?) in the current implementation. This is niche enough (and has a workaround) for it to not get fixed. – Link0 Apr 17 '23 at 06:37
  • @Link0 After some thinking and trying stuff, it seems that is not the case. If what you are saying (and what I was thinking) were true, then just adding `I::Item: Future + Send` should fix the problem, but it doesn't. Collecting the iterator before passing it fixes the problem. Also consuming the iterator `foos.into_iterator().map(|foo| async move { process_foo(&foo).await; })` works fine. But for some reason the generic `F` doesn't require any of that. In any case, I prefer other solutions, mentioned in the comments of the post. – protwan Apr 17 '23 at 12:43
  • Yeah, I've discarded current solution long time ago. This mostly became a puzzle, as to figure out what rustc is even doing here. Thanks for your time; appreciate it. – Link0 Apr 18 '23 at 10:49