4

I'm writing a simple tokenizer in Rust but I'm having trouble. I've simplified the code a bit for the sake of this question:

use std::iter::Peekable;
use std::str::Chars;

struct Example<'a> {
    it: Peekable<Chars<'a>>,
}

impl<'a> Example<'a> {
    fn tokenize_string(&mut self) {
        loop {
            match self.it.peek() {
                None => break,
                Some(_x) => self.it.next(),
            };
        }
    }
}

The error I'm getting is:

error[E0499]: cannot borrow `self.it` as mutable more than once at a time
  --> src/main.rs:13:29
   |
11 |             match self.it.peek() {
   |                   ------- first mutable borrow occurs here
12 |                 None => break,
13 |                 Some(_x) => self.it.next(),
   |                             ^^^^^^^ second mutable borrow occurs here
14 |             };
   |             - first borrow ends here

I've been able to work around this by creating a copy of the iterator and calling peek() on that:

fn tokenize_string(&mut self) {
    loop {
        let mut iterator = self.it.clone();
        match iterator.peek() {
            None => break,
            Some(_x) => self.it.next(),
        };
    }
}

Is this the best way to do this? It seems a little hack-ish.

Shepmaster
  • 388,571
  • 95
  • 1,107
  • 1,366
James Hall
  • 300
  • 3
  • 10
  • 2
    This is a shortcoming in Rust right now: See https://github.com/rust-lang/rfcs/pull/396. Another workaround would be to `let m = self.it.peek(); match m {` – McPherrinM Nov 14 '14 at 04:19
  • @McPherrinM: I think this is worth an answer :) – Matthieu M. Nov 14 '14 at 07:22
  • @McPherrinM, I don't think `let m = self.it.peek()` works since the problem is `peek` returns a reference into `self.it`. – huon Nov 14 '14 at 07:36
  • I didn't put it as an answer because I didn't know if I was right. Not sure what the SO-ish thing to do here is. – McPherrinM Nov 14 '14 at 07:48
  • @dbaupp is right, I tried that and it still gave me the same error. If you want to try it out, the entire file is available [here](https://gist.github.com/JamesOwenHall/2ec6e2c24f8b37009ab0). You'll notice that the `tokenize_string` function is a little more complicated. – James Hall Nov 14 '14 at 12:49

1 Answers1

3

Since you're working with str::chars(), and char is Copy, you can dereference to get a char instead of &char. :

fn tokenize_string(&mut self) {
    loop {
        let r = self.it.peek().cloned();
        let n = match r {
            Some(_) => self.it.next(),
            None => break,
        };
        // whatever
    }
}

If you just want to check if the iterator has returned something, use is_some():

let r = self.it.peek().is_some();
if r { ... } else { ... }

In general, however, I'm not sure if it is possible exactly in this manner without non-lexical lifetimes. You will need to put the code which checks iterator state and the code which works with the iterator based on the state with scopes, something like this:

let r = {
    // work with self.it
};
if r { ... } else { ... }

Here any references into self must not escape the lexical block in r, so there is no direct match on a value which contains references into self. There's further examples of working around this in Rust borrow of a HashMap lasts beyond the scope it's in?.

Shepmaster
  • 388,571
  • 95
  • 1,107
  • 1,366
Vladimir Matveev
  • 120,085
  • 34
  • 287
  • 296
  • Actually, it seems like my code is working. All I'm doing with the copied iterator is calling `.peek()`. I'm not trying to advance the iterator at that step, so it shouldn't matter if they're independent. And no, I don't just want to check if the iterator returned something, this is just a simplified version of the function. You can see the whole file [here](https://gist.github.com/JamesOwenHall/2ec6e2c24f8b37009ab0). – James Hall Nov 14 '14 at 12:56
  • Ah yeah, sorry, somehow I missed that it was only peek. Anyway, I still think that duplicating iterators is not a good idea, and certainly not idiomatic. – Vladimir Matveev Nov 14 '14 at 14:04
  • I think you're right. I'm trying to figure out what the idiomatic way is. Any ideas? – James Hall Nov 14 '14 at 14:14
  • 1
    @JamesHall, I'm pretty sure that my answer qualifies for that :) – Vladimir Matveev Nov 14 '14 at 14:38