How do you create an iterator using a closure in rust?

Question

I naively tried to do this:

struct Foo<'a, S: Send, T:Send> {
    next_:Box<Fn<(&'a mut S,), Option<T>> + Send>,
    state:S
}

impl<'a, S: Send, T: Send> Iterator<T> for Foo<'a, S, T> {
    fn next(&mut self) -> Option<T> {
        return self.next_.call((&mut self.state,));
    }
}

To create an iterator I could send to a task easily using a closure.

However, it generates the dreaded lifetime mismatch error:

<anon>:8:33: 8:48 error: cannot infer an appropriate lifetime for borrow expression due to conflicting requirements
<anon>:8         return self.next_.call((&mut self.state,));
                                         ^~~~~~~~~~~~~~~
<anon>:7:5: 9:6 help: consider using an explicit lifetime parameter as shown: fn next(&'a mut self) -> Option<T>
<anon>:7     fn next(&mut self) -> Option<T> {
<anon>:8         return self.next_.call((&mut self.state,));
<anon>:9     }
error: aborting due to previous error
playpen: application terminated with error code 101

I don't understand the error.

The closure should take an argument with the lifetime 'a, which is the lifetime of the struct.

The state is owned by the struct, so it has a lifetime of 'a.

Using next_.call((&mut self.state,)) doesn't invoke a task; it should only be for the duration of the call(), which as far as I can tell should be valid.

So the mismatch here is between the lifetime on self in next(), and the 'a in call... but I don't see why it would not be 'a.

What's the right way to fix the code above?

Is there a better approach to take to do this?

playpen: http://is.gd/hyNi0S

Answer 1

This requires higher-rank lifetimes, since the lifetime in the closure should not be part of the type signature: the closure just wants to take &'a mut S for any lifetime 'a (since it needs to call the function with data that is only guaranteed to last for the interior of the next method: nothing nameable externally), not the lifetime exposed (and somewhat controllable) externally in the type signature. This isn't possible, but I've seen Niko Matsakis talking about working on it on IRC, and there are preparatory pull requests like #18837 so this will hopefully appear soon.

To be clear: the code fails because next_ can only be called with a reference that lives for at least 'a, but &mut self.state only lives for as long as &mut self, which is not 'a unless it is declared as &'a mut self (which is why the compiler suggests it). Adding this lifetime is illegal since it does not satisfy the requirements of the trait declaration.

You can work around this using the old closures for now (this is essentially what a Fn trait object is anyway), and there's even a standard library type that does this for you: Unfold.

Answer 2

This is an unfortunate limitation of current Rust's trait system which will be lifted very soon. It is a lack of higher-kinded lifetimes. As far as I know, its implementation is currently being worked on.

Let's examine your struct definition closer (I've removed mut to allow further reasoning with 'static, this does not make it less general):

struct Foo<'a, S: Send, T:Send> {
    next_: Box<Fn<(&'a S,), Option<T>> + Send>,  // '
    state: S
}

'a lifetime parameter is an input parameter here. It means that it is provided by the user of the structure, not by its implementor.

(BTW, it is not the lifetime of the structure instance - you can't specify such lifetime with type parameters only; it must be a lifetime on self reference, which you also can't use because Iterator trait method does not have a lifetime parameter. This is just a side note, however, and it is unrelated to the actual problem)

This means that the user of your structure may choose 'a arbitrarily, including choosing some lifetime which is greater than the lifetime of your struct, for example, 'static. Now observe how such selection transforms the structure (just substituting 'static for 'a):

struct FooStatic<S: Send, T: Send> {
    next_: Box<Fn<(&'static S,), Option<T>> + Send>,  // '
    state: S
}

And suddenly, the closure only can accept 'static references, which is obviously not your case - lifetime of self in the next() method may be shorter, so you can't just pass it into the closure. This could only work if self lifetime did correspond to 'a (as suggested by the compiler):

fn next(&'a mut self) -> Option<T>

However, as I said earlier, you can't write this because it would violate trait contract.

With higher-kinded lifetimes it will be possible to specify lifetime parameter on the closure itself:

struct Foo<S: Send, T: Send> {
    next_: Box<for<'a> Fn<(&'a mut S,), Option<T>> + Send>,
    state: S
}

This way the lifetime parameter of the closure is chosen by the caller of the closure, in this case, it's the implementor of the Iterator trait (i.e. you :)), so it will be possible to call next_ with any reference, including a reference into Foo internals.

Answer 3

You can use iterate from itertools that do exactly what you need.

Otherwise if you want to enclose all the logic in a closure without use a state concept you can implement it by:

struct IterClosure<T, C>(C) where C: FnMut() -> Option<T>;

impl<T, C> Iterator for IterClosure<T, C> where C: FnMut() -> Option<T>
{
    type Item = T;

    fn next(&mut self) -> Option<Self::Item> {
        (self.0)()
    }
}

fn iter<T>(f: impl FnMut() -> Option<T>) -> impl Iterator<Item=T> {
    IterClosure(f)
}

fn main() {
    let mut it = (0..10).into_iter();
    let mut closure = || it.next();
    println!("{}", iter(closure).sum::<i32>());

    let mut it = (0..10).into_iter();
    iter(|| it.next()).for_each(
        |i| println!("{}", i)
    )
}